Oral finals and frequent low-stakes assessments are just a few alternatives to in-person exams that Jenny Quinn uses to ensure student learning and success.
While it may seem impossible to assess student learning without an in-person exam, Jenny Quinn is one math educator who has been proving that otherwise.
Jenny Quinn teaches her students via Zoom.
Quinn, who teaches undergraduate mathematics courses at UW Tacoma’s Interdisciplinary Arts & Sciences, has been experimenting with new approaches to testing. In spring 2020, she gave an oral final exam remotely in her Matrix Algebra course and used frequent, low-stakes mastery-based grading in her Precalculus class. In autumn, she’s organized her two Calculus II classes of 30 students each around a mixture of online and oral assessments, culminating in an individual oral final exam. With planning and TA support, these successful approaches could scale up to larger courses.
Spring 2020—Giving an oral final exam
Quinn was faced with a common problem: how to ensure her exam was testing student learning, rather than their ability to search online for solutions.
“It used to be you could ask students to show their work to be sure they understood the concepts. But now,” she explains, “there are computer algebra systems where you can enter any computational problem and it will be solved symbolically, giving all the intermediate steps.”
Instead, she asked students to meet with her briefly via Zoom to walk her through two of the questions on the test, “so they could show me they understood the concepts and not just procedure.” Even if students looked up the answer, which was allowed since they had the questions in advance, they had to demonstrate that they understood the solution.
Quinn’s inspiration for including an oral component was not solely to discourage cheating. A strong advocate of growth mindset, she has long been interested in mastery-based grading, in which learning competencies are shared up front and students can take assessments multiple times until they master each point.
“There’s no reason to cheat, but it’s more work for me to set up. That said, I’m not testing the right things if the exam has to be done in person.”
Quinn’s oral exams by the numbers:
9 questions total—Students get questions in advance so they can prepare for all nine. “I tell students, ‘You can use any resources. Internet search? Working in teams? That’s OK.’ There was one question in which I asked them to take a proof that we didn’t do in class and either create their own or find one online, cite and critique it. My favorite question was to write five true/false questions for this exam and explain their choices. They had to not only think about the substance of the questions but also how each reflected the learning in the course.”
20 minutes to answer 2 questions from the list of nine—Students choose one, the instructor the other. “If they picked a question from the first half of the quarter, I’d pick one from the second half. If they picked one with a lot of computation, then I’d give them a reflection/synthesis question. I tried to balance it out.”
50 points each—This is the maximum value for each question, based on a holistic grading rubric. “Students are afraid about how they say things (especially if English is not their first language), so I make it clear it’s not about the right words. It’s about the concepts they express. This kind of rubric is subject to bias, so you have to be aware of that. I ask myself if I would rate this answer the same if it were a different student—one I liked more or less—and I go back to the rubric to counteract by own biases. There’s a lot of thinking that has to happen.”
-4 points to “pass” on a question—Students can elect to pass on a question up to a maximum of two times but lose four points each time they do.
Community inspiration for new approaches to assessing math
Because UW exams are offered later than at many other universities, Quinn learned from her peers in the larger community of mathematicians who were giving remote finals earlier in spring.
She took particular insight from two items on a popular forum: Harvey Mudd’s Francis Su, who posted “7 exam questions for a pandemic (or any other time),” and Rick Cleary of Babson College, who based his final exam on Su’s approach.
“Su has written a wonderful book called ‘Math for Human Flourishing’ and was trying to align exam practices with what students need to flourish as humans. That was the first trigger,” says Quinn. “Then Rick Cleary shared that he gave an oral final exam based on Su, and I thought ‘I can do this!’”
Quinn next had to prepare her students for the new exam format so they wouldn’t be surprised.
She communicates what she’s doing and lessons learned with others via her blog Math in the Time of Corona.
“The UW pivoted to remote learning earlier than most academic institutions. As president-elect of the Mathematical Association of America, I felt it was important to share my hard-won knowledge with others making the same transition and to offer support by humanizing the experience.”
Her piece on alternatives to standard exams was the most popular in spring garnering over 1300 views from 70 different countries. Others have joined Quinn in sharing their experiences. She recommends Robert Talbert from Grand Valley State, for example, for his “beautiful series on how he was thinking about his teaching and doing inquiry-based learning in a hybrid setting.”
Autumn 2020—Scaffolding students’ ability to ‘speak math’
This quarter, Quinn is continuing to refine her practice. She will give oral final exams again but plans to develop students’ comfort and competence ‘speaking mathematics’ throughout the quarter. She introduced group oral assessments beginning in the second week to help students practice and build community early on. Her revised rubric rewards both individual competence and group accountability. There are two more oral quizzes in weeks five and eight. In each assessment, the group size gets smaller until, by the end of the quarter, students are presenting individually.
Students appreciate that classmates aren’t tempted or able to cheat
Quinn has been pleasantly surprised by her students’ reactions. “Students truly appreciate it. They worry about the academic integrity of their fellow students, as much as we do.” Students recommended better scaffolding for learning so they would be even more prepared and familiar with the format of the assessment. In response, this autumn Quinn introduced group oral assessments throughout the quarter. She continues to share via her blog as she updates and refines her practice.
How to scale this approach to larger classes
Quinn teaches two classes of 30 students and admits that this might prove challenging—but not impossible—in larger classes. The approach could work if teaching assistants were trained on the rubric and given practice using it in norming sessions prior to assessments to ensure inter-rater reliability across sections.
Quinn discusses in this short video how remote teaching prompted her to implement oral exams to encourage collaboration, leverage available resources, and maintains individual accountability:
For additional examples of UW faculty alternatives to timed exams, see Teaching Everywhere faculty blog posts:
Teaching Remotely: Assessment page includes information on utilizing alternative assessments, why high-stakes exams don’t transfer well to remote learning, academic integrity, and grading practices for online learning.
How to Talk to Your Students About Cheating video featuring Professor and Chair of Chemical Engineering Jim Pfaendtner. Learn how to talk to students about cheating — why they cheat and what to say on the first day of class to discourage cheating.
As the UW seeks to support data management and analysis more broadly across disciplines, the answer may just lie with Jupyter Notebooks.
David Shean, Assistant Professor of Civil & Environmental Engineering
By Ignacio Lobos
When David Shean started searching for a more effective and interactive tool to teach students how to analyze complex geospatial data sets in his mixed graduate and undergraduate class, he zeroed in on Jupyter Notebooks.
The Jupyter Notebook is not a laptop — it’s a powerful open-source web application that offers students the opportunity to work with 100 different coding languages. They can combine live code with text and narrative to support their data analysis work, and graphs, equations, videos, animations, maps and other visualizations to explain their results more fully — whether they’re working in the humanities or STEM fields.
“The notebooks are an amazing teaching and learning tool,” said Shean, an assistant professor in the UW Department of Civil & Environmental Engineering who carefully built his most recent geospatial data analysis classes around their use.
“One of the things that makes the notebooks great is that students write code, and immediately see results,” Shean said. “They can interactively tweak, interpret, and repeat until they are satisfied, which enables data-driven discovery.”
Using their notebooks, students in Shean’s class learned how to analyze geospatial data to answer questions such as, “How much of Whidbey Island would be flooded due to sea level rise if the Antarctic and Greenland ice sheets melted?” “What are the most hazardous roads in Washington state?” “How much snow was on Mt. Rainier yesterday?” “Is the Arctic warming faster than the rest of the planet?”
In his class, students worked with sophisticated open-source software packages to visualize a given problem with a geographic component. “The notebooks can be run so that all of the plots are interactive, such as zooming or panning around high-resolution satellite images or clusters of data points on a chart, which is really powerful,” he said.
Shean partners with UW-IT to launch Jupyter Notebooks project at UW
Shean has used the notebooks in two pilot projects for UW courses in the past year, most recently in a partnership with UW Information Technology’s Academic Experience Design & Delivery (AXDD) unit.
“With David’s class and other classes that participated in the pilot, we’re carefully creating an enterprise service to support Jupyter notebooks for teaching and learning,” said Tom Lewis, AXDD director. “We are collaborating with the eScience Institute, UC Berkeley and others for knowledge transfer and system design. The goal is to provide easy-to-use research tools for use in courses.”
Why Jupyter Notebooks are so powerful
At the UW, where the Office of the Provost is working to support data management and data analysis more broadly across disciplines, the notebooks fit the bill. Among their advantages:
Students in multiple areas of study can use the notebooks to explore and work on a multitude of real-world applications, such as helping a city relieve traffic gridlock, a health department track a viral infection, or a utility reduce operating costs.
The notebooks are ubiquitous worldwide, a tool used by professionals to collaborate across multiple fields.
Students access their work and share it from anywhere — which is proving valuable during the COVID-19 pandemic, with classes being held remotely. All students need is a web browser and an internet connection, and they’re ready to work.
The notebooks lower the entry barrier, bringing powerful tools within the reach of more students.
Because students use their own laptops and other devices, the notebooks decrease or eliminate the need for expensive computer labs with limited hours of operation.
Faculty can provide a state-of-the-art computing environment to their students with none of the hassles of maintaining it.
The notebooks are hosted in the cloud, with powerful servers that can handle the computational work and vast amounts of data.
With easier and cheaper access to cloud computing, the notebooks are becoming a more viable and powerful tool in pedagogy.
And, most importantly, students love them.
“It is the future,” said Friedrich Knuth, a Ph.D. candidate who participated in Shean’s winter 2019 class.
“The software is free and open-source, so as a student and young professional, I can take the methods that I learned to develop with me anywhere I go,” Knuth said. “I don’t need to own a powerful machine myself, as all computation is done on the servers hosting the notebook. This lowers the barrier of entry for scientific data analysis and exploration.”
How Shean built a class using Jupyter Notebooks and other tools
When Shean made the decision to use the notebooks, he worked with AXDD to carefully build a computing environment to enhance the teaching and learning experience in his classroom. After lengthy discussions about Shean’s needs, AXDD set up the system, maintained it and covered the costs of operations, including buying cloud server time, during the pilot.
The notebooks are one of several key components that make up Project Jupyter, a non-profit, open-source organization that supports interactive data science and scientific computing with robust online tools.
“We’re using a JupyterHub for cloud infrastructure, a JupyterLab environment for interactive computing, and the Jupyter Notebooks for teaching and development,” Shean said.
“The UW-IT support freed me up to focus on the course content, and to develop the notebooks each week,” Shean said. “I developed and tested all course material on the JupyterHub, and the students did all of their lab exercises, homework, and final projects using the same resources. There was never a question about compatibility.”
Other faculty, he pointed out, “might want to use notebooks for a few labs, or encourage students to use the notebooks as a substitute for MATLAB or Excel. Or, as with my course, the notebooks can provide the foundation.”
“I perform interactive tutorial/demos with notebooks. All of the lab/homework instructions and questions are embedded in the notebooks. And I designed the notebooks to build in complexity, starting with simple examples to illustrate basic concepts, then adding new datasets and challenges with fewer instructions, so students apply what they did earlier to solve more complex problems with real-world implications,” he said.
“The Jupyter resources transformed the way I taught the class,” said Shean, a young faculty member who has been in front of the classroom for only a couple of years.
“Students are very comfortable in the modern tech environment. They get it, and more and more, they’re demanding tools that will allow them to explore and solve large data problems.”
How to get started with Jupyter Notebook
In the past year, AXDD has worked with 15 faculty members from different areas of study to understand their teaching needs. It also built and tested a set of JupyterHub/Notebook environments that can be quickly deployed by instructors. The pilot was so successful that UW-IT will be offering the service to faculty starting this fall. Currently, there’s no cost for this service. Visit IT Connect to learn how you can bring the notebooks into your classroom.
“Students love the notebooks,” Shean said. “They see value in them, and are learning from them. It’s really rewarding to hear.”
Philosophy lecturer Ian Schnee‘s e-book about logic has proved to be irresistible to tech-savvy students. And there’s no reason other faculty can’t accomplish the same with their texts.
Ian Schnee, lecturer, Philosophy
By using a new e-book in his Introduction to Logic classes, Ian Schnee wanted to engage his students more fully with the material and save them money — as much as $55,000 in just three quarters.
To make it happen, Schnee wrote and unveiled his own e-book in winter 2019. His Logic Course Adventure, an active learning textbook for formal logic, was his latest effort to modernize the classroom experience with evidence-based teaching and learning approaches with a slight twist: he wanted to take his teaching into students’ natural spaces — think cell phones and other devices — and show them that traditional courses in the social sciences can be exciting and fun.
“I want to make philosophy fun and engaging for my students — because philosophy is all of these things — but I can’t ask them to read 10-20 pages of dry prose from a book that hasn’t been updated in decades and expect them to come prepared to discuss logic during my lectures,” Schnee said. “The traditional textbooks that are in use today for teaching logic are dense, hard to read, and frankly, boring.”
The evidence shows that forcing students to read traditional textbooks is not the best way to learn. But if you couple reading to engaging quizzes as they read, they’re more likely to absorb the material, Schnee said. And so far, his highly interactive e-book has accomplished that and more, his students agree.
“I was at first a bit wary of this e-book because I believe my class was one of the first to use it,” student Kelsey Kinoshita said. “But it really contributed to my learning, and the class would not have been the same without it.”
Student interacting with the textbook on their phone. Photo: Elizabeth Lowry
Alex Saveau read the e-book on the go on his phone, sometimes on the way to a part-time job – and found the seemingly endless number of problems captivating.
“What really makes a difference is that you can engage with the textbook,” Saveau said. “For other classes, I find myself fighting to not drift off into a daydream while reading the textbook.”
But Schnee’s e-book challenged him constantly with quick short quizzes, even after just reading a few sentences. The constant problems that require an answer before moving on to another section “keep you focused and accountable,” Saveau said.
E-book becomes latest piece in modern teaching environment
During the past decade, Schnee has carefully deployed new technologies and teaching approaches inside and outside the classroom to increase learning and engagement.
He “flipped” his classrooms when appropriate, and when Poll Everywhere, the UW’s preferred classroom response system, was introduced at the University in 2016, he became one of its earliest adopters.
During lectures, Schnee uses Poll Everywhere and other evidence-based teaching strategies such as collaborative group work, think-pair-share activities — where students work together to come up with answers — and the judicious use of random calling, which increases participation while decreasing instructor bias. With the e-book, he wanted to get the same engagement — only this time outside the classroom.
A very active e-book
Everyone learns better by actively engaging with the material, so Schnee knew his e-book would have to do a lot more heavy lifting than a regular book, even a typical e-book. E-books have been around for a while, but most are passive like their paper-based counterparts.
“I wanted to revolutionize the paper book, and create an e-book that was fully interactive, with intuitive software and no learning curve for my students,” Schnee said.
He had been thinking about it for 10 years, but it was only until recently that tech caught up to his vision. In 2018, he spent nine months writing his book and building his own website to host it. The latter was more by necessity than design. There are plenty of e-book platforms on the web, but many are quite costly — as much as $1,000 per year — or unwieldy for creators and users alike. So, Schnee learned programming to build his own.
“I called it an adventure, because I wanted the book to be fun,” Schnee said. “Reading separate sections of the e-book doesn’t take a lot of time, and I discovered that many students did all of the problems. And they kept telling me they wanted more and more problems. It was quite a pleasant surprise.”
A file folder lands on your desk with a thud. You’re a rookie police detective, so the Sergeant puts you on the case no one else wants. A bank robbery. Inside the file you see pictures of three suspects: Pia, Quinn, and Raquel.
“We’ve got all three of them in custody,” the Sergeant says. “At least one of them is involved, but I can’t keep them past morning. You need to figure out who did it by 8 a.m.”
Now you know why no one else wanted this case. It takes you all night but you finally get through the file. By 7 a.m., here’s what you’ve figured out:
1. One or more of these people is guilty: Pia, Quinn and Raquel.
2. No one else was involved.
3. Pia never works without Quinn. So if Pia is guilty, so is Quinn.
4. Raquel never works alone.
What do you tell the Sergeant?
Former student Andrew Stahl was happy to put on his “rookie cop” thinking cap to come up with the answer.
“The e-book enabled us to learn the fundamentals of logic on our own so class time could be spent working on more complex problems and answering student questions,” he said. “I certainly found it more engaging than a regular book, as the embedded problems keep you from moving forward without first understanding what you have already read.”
So, you want to create an e-book
If you’re interested in creating an e-book, Schnee has some advice.
Don’t go it alone: Schnee tells colleagues there’s no reason they have to go at it alone. So, if you are ready to get started with an e-book, he will be happy to chat with any faculty member about his own experiences and process in putting together the logic adventure, and point them to the appropriate resources, including UW Libraries and other faculty who are involved in evidence-based teaching.
Schnee created his e-book from a combination of open-source software and his own code, and he’s hoping that instructors at other schools will one day use it as well.
Start small: There’s no need to write an entire textbook at once. Schnee’s first draft only covered 75 percent of his course material, but instructors could start with just one chapter or assignment.
Use ready-made software: For instructors who don’t want to make their own website, they can start with Pages in Canvas, which have much of the same functionality.
Think accessibility: Schnee is committed to making his e-book highly accessible, building in captioning and alt-text.
Don’t crash: He learned a quick lesson about managing high internet traffic (his website crashed on the first week of class, for four straight days, before he fixed the problem).
Make it interactive: For now, his e-book is not being used for grading. The problems are for practice only, but students really want them, so he is planning on doubling them in an upgraded version.
Iterate: An e-book is a living book. Feedback from students continue to make it better, and Schnee adds more problems as necessary.
Not just for philosophy
Schnee remarks that e-books like his are not just for philosophy and logic. His students agree:
“Since this textbook was for logic, which has problem sets, I think it may work well for other science and math-based classes,’’ Kinoshita said. “The strengths of Ian’s textbook — easy to understand, concise, entertaining examples, straightforward, engaging — are what would transfer best in creating textbooks/e-books for other classes.”
When the snow began to fall in Winter 2019, instructors across the UW had to think quickly about how to keep their courses on track.
Instructors such as Haideh Salehi-Esfahani and Scott Spaulding, who were already using technology tools in their teaching, were well-equipped to ride out the storm. Others, like Riki Thompson, adapted quickly.
These three instructors tell their stories and share tips about how they ensured instruction continued despite the snowfall. Their best collective advice for people who don’t already use Zoom or Panopto in their teaching is to think about starting now, regardless of the weather.
Whether it’s snow, road closures, a broken ankle, or a conference presentation, faculty are making good use of teaching technologies to keep students on track when missing class is unavoidable.
Scott Spaulding, senior lecturer, education
When familiar, Zoom conference calling is an easy back-up
During Winter 2019, Scott Spaulding was teaching a graduate course in which Zoom conference call technology is a regular feature.
When class was cancelled due to snow, Spaulding wasn’t sure if he was allowed to hold class remotely when campus was closed. “I wanted to have class and the students did, too, but I didn’t know if we could,” until a provost message clarified that classes could meet. “That was good to have that guidance from the provost’s office.”
Many of the students in Spaulding’s program are teachers who join the class remotely from work. As a result, the program sets guidelines for how students use Zoom in the course.
“We ask them to connect from a place that’s distraction-free. What’s behind you? What’s your connection speed like? We emphasize this for our program at orientation, before they come to any class. Students might be running late and want to connect via phone while they’re driving. We say don’t do it.”
When the snow hit, students were familiar with the tool, although they had to figure out how to set themselves up at home. Spaulding suggests patience with the unexpected, “You can’t prevent everything from happening—a kid walks on camera, a cat—one day we had six cats wandering around on screen.”
When Zoom is not an option, try Panopto or Canvas modules
Spaulding was prepared in 2019 for the snow, but knows what it’s like to be caught off-guard.
A different quarter, he had to miss the second day of class to attend a conference. He says, “I thought we could meet via Zoom.” But since it was early in the quarter, students didn’t know how.
Instead, Spaulding created a Canvas module. “It was super-explicit with discussions, readings, and the need for them to comment.” He made a Panopto video from his hotel room of slides and uploaded the video to the Canvas page. “I told students to engage in discussion at the same time we would have met in class. I don’t normally use modules but it worked great this time.”
Tips:
Be familiar with how to use tools yourself before asking students to use them.
Make sure your students know what to expect. Tell them, for example, ‘If we have inclement weather, we’ll use Zoom.’
Set expectations for access to a computer with a strong connection and a camera—and a quiet place. Tell students, ‘Don’t call from a coffee shop.’
Riki Thompson, associate professor, Rhetoric & Writing Studies, UW Tacoma
Seminar students led discussions via Zoom conference calling
When the snow started to fall in Winter quarter 2019, Associate Professor Riki Thompson was leading a group of graduate students doing independent study. She quickly looked for alternate ways for the group to hold their seminar discussion.
Students had prepared to discuss the reading and facilitate a conversation—a conversation that if done remotely would require conference call technology.
Thompson recalls, “We thought about Skype but didn’t know if it had multiple screens. We talked about doing Google Hangouts, but one of the students had trouble with it. Just trying to identify the tech we could use took some time.”
Thompson had a free, personal Zoom account. It seemed the best option with one important exception: sessions time out after 40 minutes. But Thompson found a creative workaround. She told students, “We’ll talk for 40 minutes and then take a five-minute break—like in face-to-face classes—and then return.” Building in a five-minute break gave her time to start a new Zoom session, while avoiding feeling rushed.
The students set to present during the session emailed handouts, and the class took advantage of Zoom’s chat function during the discussion. “If someone had a burning question, they could add it to the chat feature, so we could return to it,” says Thompson.
While the group didn’t use the shared screen because everyone already had the handouts, Thompson reports that next time she would use that feature to highlight elements of the document.
Tips:
Check to see if you have access to a Zoom account…before the snow starts falling
If you don’t have access, encourage your department to get Zoom or set up a personal account
Tailor your use of tools such as chat function and screen sharing to the discussion
Haideh Salehi-Esfahani, principal lecturer, economics
Posting a recorded lecture kept a large class on track
When a winter storm shut down UW’s Seattle campus in 2019, Haideh Salehi-Esfahani had not prepared but regardless was ready. Her ECON 200 Principles of Micro-Economics students—all 450 of them—watched a video recording she posted of the lecture she had given on the same day the year before.
Salehi-Esfahani, a Principle Lecturer of Economics, uses the lecture-capture tool Panopto in her larger classes, whatever the weather, as a matter of course. Every lecture is recorded and then posted on the course web site. The video shows both the instructor and everything she projects on the screen during the class.
According to Salehi-Esfahani, posting lectures regularly supports her students’ learning and “gives students a chance to review the lecture, even in a small class.”
Posting lectures has the added benefit of maintaining continuity when a class is unable to meet. “What’s really nice about this,” says Salehi-Esfahani, “if you have consistent lectures, students don’t miss anything.”
For those who don’t have a handy archive of last years’ lecture videos or are teaching a new course, it’s possible to use the same technology to record a lecture from home. Faculty can even learn how to use Panopto remotely, as well.
That said, Salehi-Esfahani suggests taking advantage of available training—before you need it. “It’s worth spending a few minutes with the guys at Learning Technologies to learn how to do this.”
Snow can close a campus but is not the only reason a class can’t meet. “You may go to a conference or you may get sick,” says Haideh, “this is not just for winter weather.”
Tips:
Plan ahead for how you’ll teach when your class can’t meet
Practice with the tool before you need it
Attend a Panopto training by Learning Technologies
In any major, UW students learn to evaluate sources, use texts responsibly and understand the impacts of information. The rise in fake news and misinformation creates an even greater need for these skills, in and beyond the classroom.
At the same time, the topic itself can be an effective and timely way to engage students. Some instructors have been incorporating fake news and misinformation into their courses so students develop critical thinking skills and, in some cases, come up with concrete solutions to the problem.
“Calling Bullshit” arms students at the UW and beyond with tools to spot BS, wherever it appears
Jevin West, assistant professor in the Information School and Carl Bergstrom, professor in Biology. Photo credit: Quinn Russell Brown.
“Our world is saturated in bullshit,” begins the syllabus for INFO 198/BIOL 106B, the cross-listed “Calling Bullshit: Data Reasoning in a Digital World,” designed and taught by Carl Bergstrom, professor in Biology and Jevin West, assistant professor in the Information School. The course’s objective? To help students “learn to detect and defuse it.”
Bergstrom and West have been collaborators for years, and had long talked about a problem they both noticed in STEM higher education: in Bergstrom’s terms, that “We do a really good job teaching the mechanics of subjects. But we’re not teaching students to engage with uncertainty and weigh different arguments against each other. That’s a really big problem because it makes us particularly vulnerable where numbers are concerned.”
So in winter 2017, they decided to create a course that would teach habits of mind more commonly connected with Humanities fields — engaging uncertainty, questioning evidence — to STEM students. But almost as soon as the course launched that spring, it became something much bigger: a course about approaching information critically, applicable to any discipline.
Meeting a need for BS detection skills — and going viral
“Calling Bullshit,” first offered as a one-credit course, attracted students from a wide range of disciplines. When Bergstrom and West taught it a second time, in fall 2017 as a full-fledged three-credit course, enrolled students represented over 40 different majors.
The “Calling Bullshit” website makes course lectures available for public viewing. Each video is under ten minutes, making them easy to watch and share. Photo credit: Shantelle Liu.
And its impact has extended well beyond their classroom — far beyond the UW. Before launching the course, Bergstrom and West created a robust website with all course materials accessible to other institutions and to the general public. The course has gone viral, with new courses being explicitly modeled on “Calling Bullshit” at over 70 universities and high schools around the nation, and across the disciplinary landscape. West says they’ve been surprised and excited to hear from so many people —students, teachers, retirees — who have watched their lectures, read course texts and found the course interesting and valuable. In under two years, the course’s Twitter account has garnered over 8,000 followers — and the course website has been viewed 1.5 million times.
Empowering students to detect BS, in any field
The course’s learning goals make it easy to see why the course is filling a need for so many students at so many institutions; these include, “Remain vigilant for bullshit contaminating your information diet,” and “Figure out for yourself precisely why a particular bit of bullshit is bullshit.”
So, what exactly is the “bullshit” that students most need tools for detecting? Bergstrom and West say it’s that which cloaks itself in what we’re often inclined to treat as “truth”: namely, “statistical figures, data graphics, and other forms of presentation intended to persuade by impressing and overwhelming a reader or listener.”
As their students discover, detecting bullshit is no easy task. “We give them a set of rules for calling BS,” says West, “like ‘if it seems too good to be true, it probably is.’ Students feel empowered when they can spot this stuff.” Students are asked to bring in examples of bullshit when they encounter it in daily life. For Bergstrom and West, the goal is for everyone to become effective bullshit detectors.
And these skills have enormous value for students, including in their future careers. “When we talk to big companies, they want people who have these skills to look at a whole situation, take a proposal that’s on the table and evaluate and challenge that, in a quantitative domain,” says Bergstrom. “It’s an absolutely crucial part of STEM education — and really, of all education.”
Jackson School Task Force tackles the international problem of fake news and misinformation — and offers solutions
In the Jackson School for International Studies, students have a unique opportunity to enact real change on a pressing social issue: participation in a Jackson School Task Force. In winter 2018, a task force tackled fake news and misinformation — producing a 100-page report, now accessible online and contributing to the academic conversation on the issue.
Scott Radnitz, associate professor of International Studies and adjunct associate professor of Political Science and Sociology. Photo credit: Shantelle Liu.
“The New State of the News: Confronting Misinformation in the Digital Age” was designed and taught by Scott Radnitz, associate professor of International Studies and adjunct associate professor of Political Science and Sociology. Task forces, offered every winter quarter to seniors in the International Studies major, are themed around real world problems. Students collaboratively research and write a detailed report directed toward policy makers, and at the end of the quarter, gain professional experience through defending their findings to a visiting subject matter expert.
Engaging students around an urgent and timely topic: U.S. and global disinformation
Radnitz, a political scientist, studies the post-Soviet region; three years ago he started working on a book about the central role of conspiracy theories in post-Soviet states’ politics. “It’s a different way of dealing with media in that region,” he says, “when you don’t know what you can trust.” But something has recently changed, he says: conspiracy theories and fake news have also become more prevalent in the U.S. He decided the issue — urgent, currently unfolding, and international in scope — would be prime task force material.
Radnitz suspected the topic would be of special interest to students as well — and he was right. “Students are engaged in this issue personally because they’re avid social media users,” he says, “but also because everyone was so immersed in the 2016 presidential election, following the revelations about fake news as they came out. In some ways my students aren’t that far behind the experts, because everyone’s trying to catch up and figure it out.”
The Jackson School’s winter 2018 Task Force: “The New State of the News: Confronting Misinformation in the Digital Age,” allowed students to wrestle with this complex issue while developing research and professional skills. Photo Credit: Franceska Rojas.
For students, the task force’s timeliness imbued it with a sense of high stakes — and also helped develop their research skills. “Considering the immediate nature of fake news and misinformation, we had to keep track of the new material and developments that were emerging every day and basically work in real time,” says participant Oleksandra Makushenko, class of 2018. “It provided experience in teamwork, crisis management and working under a strict deadline.”
A multi-faceted problem requires a multi-faceted approach
In task force courses, students research an aspect of the larger issue, and then write a section of what becomes a comprehensive report. “Fake news is a pervasive problem and needs to be tackled from different angles simultaneously,” says Radnitz, so the class decided on three angles early on:
Individual and Collective Psychology
Business and Technology
International Case Studies and Governance
For each topic, students outlined the problem — for example, for topic A, the individual and social forces that make people susceptible to misinformation. For each topic students also make practical recommendations — for example, for topic B, on how pressures might be placed on businesses to limit the spread of misinformation.
Some students took international angles to their research. The report includes a comparative case study on public trust of the media in three different countries, and a section on how the Ukrainian government has responded to Russian disinformation. Radnitz says that early on, the class decided that their audience should be policy makers, academics and the U.S. public — in order to help us “learn lessons for how we can confront this problem at home.”
Developing student expertise — and contributing real results
At the end of any task force, an expert in the field visits the class for an evaluation and defense of their findings — in Radnitz’s course, former CNN Moscow bureau chief Jill Dougherty. After thorough preparation, students presented their work to Dougherty, and then responded to her questions for over an hour. The benefit of an expert’s on-the-ground perspective, and the experience of withstanding scrutiny on their research, are part of the unique professional development that task forces offer to Jackson School majors.
“Probably the most important thing I learned,” says Makushenko, “is that there is no easy fix for complex problems. It takes time, committed people, substantial resources to address something that is broken in a holistic manner.” Fortunately, some of these committed people are at the UW. Radnitz says he will likely teach a version of the “The New State of the News” for his next task force. “The issue isn’t going anywhere,” he says — and UW Jackson School students have expertise that can make a difference.
In required writing courses, students learn to critically evaluate information
All UW students, no matter their major, take a course in English composition. In the Expository Writing Program, where most students take a 100-level English course, students learn that writing, reading and research have everything to do with critical thinking: about the information that surrounds us, how arguments are composed, how evidence is used and how context affects meaning.
“A big part of the work we do in the EWP is teaching students how to effectively collect, evaluate and interpret sources in order to support their writing,” says Denise Grollmus, former EWP assistant director. “This has become increasingly important in light of the prevalence of fake news. We focus not only on how students can learn to evaluate information, but also how we can use the issue of fake news to teach these evaluative techniques.”
While EWP instructors — mostly graduate students in the English department — design their own curricula, all EWP courses share common learning outcomes. These focus on writing but also on habits of mind, including “engaging in analysis — the close scrutiny and examination of evidence, claims and assumptions — to explore and support a line of inquiry.”
Some EWP instructors use fake news to teach exactly why these skills are so very important. Matthew Hitchman, EWP instructor and assistant director, introduces students to the research process with fake news. “I try to move beyond the dichotomy of reliable/unreliable sources, because ‘reliability’ often reads as ‘academic,’” he says, and students should get comfortable using different types of sources. This means that inevitably, students will encounter bad information — so Hitchman begins his research sequence by giving students a range of real, faked and satirical sources to evaluate. They then analyze the sources’ context, credibility, purpose — and finally, their biases. Hitchman works with students to carefully define terms such as ‘fake’ and ‘misleading,’ “which are quite distinct from ‘bias,’” he says. “It’s important to learn that while sources should be scrutinized for forms of bias, there are some types of information that shouldn’t be entertained
at all.”
In addition, says Grollmus, “because even the most reputable sources can still get it wrong, EWP instructors ask students to consider the credibility of a source’s citations. We ask them to consider whether an article includes a diverse set of sources, for example, and what the exclusion of certain voices might mean. Our hope is that by training students in these evaluative techniques, we’ll help them become more critical readers and thus better informed citizens.”
To ensure that these values stay central to all EWP courses, staff are currently drafting new program mission statements that address pressing social issues, including fake news and misinformation. “It’s an issue that asks all of us to think about sources, be discerning readers and see rhetoric as something with impact and the potential to do harm,” says Emily George, EWP assistant director.
Most students take a composition course through the EWP in their first year; as they continue on as readers and writers in various fields, they carry these skills with them. At the UW, “critical reading, writing and thinking skills aren’t limited to first year composition, or even to the classroom,” says George. “They’re part of every student’s general education.”
The Evidence-Based Teaching (EBT) program supports faculty to try new, research-based teaching strategies in classrooms across disciplines.
Across the UW, faculty are committed to using the best possible teaching strategies to enhance student learning — whether that learning takes place in large lecture classes or small seminars, in Philosophy or in Oceanography.
What are the benefits of being an EBT member?
Improved teaching and student learning outcomes
Expedited support from the Center for Teaching & Learning and Learning Technologies
Mentorship from faculty across campus who are implementing best practices into their own teaching
A community of peers invested in improving their own teaching and supporting one another
Recognition from the provost
Opportunities to advance as a leader in teaching and learning at the UW
What’s expected of EBT members?
Attend biweekly meetings
Conduct peer observations
Participate in online discussion boards
Join EBT
To join the 2018-19 cohort or to learn more about the program, contact UWebt@uw.edu
At the same time, few faculty have time or support to continuously evolve their teaching — or necessarily know where to begin. Frequently, instructors learn to teach on the job without formal training, and community around teaching and learning can be hard to come by.
Enter the Evidence-Based Teaching Program (EBT): a program offering mentorship, a community of peers, opportunities for leadership, and expedited support from the Center for Teaching and Learning and Learning Technologies to faculty in all disciplines. The EBT program also offers opportunities to innovate: to use research on pedagogy, and the support of teaching colleagues, to implement new techniques and tools to become better teachers, and help students become better learners.
“From the first meeting, I knew that this is what I’d been looking for: a program that wants to prioritize empirical work in teaching, and support faculty to continuously improve their teaching,” says Lecturer in Philosophy Ian Schnee, who joined EBT in 2016. Over the past two years, participation in EBT has “transformed my teaching,” Schnee says.
A program built for growth
EBT began in 2015 as part of the Office of the Provost’s Teaching & Learning initiative, in response to a meta-analysis, co-authored by UW researchers including Principal Lecturers in Biology Scott Freeman and Mary Pat Wenderoth, that showed dramatic gains in student learning when instructors use active learning techniques. A small team in Academic & Student Affairs, led by Senior Director Marisa Nickle, partnered with Wenderoth to design the pilot.
The program takes a three-phase approach: Exploration, Implementation and Research. Faculty can participate for just the Exploration quarter, in which they read literature around teaching and learning (as extensively as they’d like) and develop goals for their own courses. Those who stay on for an Implementation quarter then implement those goals with support and mentorship from colleagues who have more experience with EBT. Some faculty are now even conducting their own research. Members can progress through the program’s mentoring ladder, from early-stage participants to group coaches or “leads” — allowing professional development opportunities for tenure-track faculty and lecturers alike.
EBT has seen enormous growth over the past three years, largely through word of mouth, as participants share their experiences with colleagues. It continues to attract instructors interested in learning about education research and implementing what they learn in real time, with support from an interdisciplinary community of peers.
The program now has three “leads” who have moved up the mentoring ladder: Schnee, Mikelle Nuwer, senior lecturer in Oceanography, and Kimberlee Gillis-Bridges, principal lecturer in English. Faculty from over 50 departments have engaged with the program, from all three campuses; over 13,000 students have taken EBT-informed courses; and the new 2018-2019 cohort is predicted to include over one hundred faculty participants.
Exploring the research — and implementing new, innovative teaching practices
In the first two phases of the EBT program, members explore existing research on teaching that applies to their own unique contexts. They meet regularly to discuss what they’ve read, as well as their own goals and potential challenges, and they observe at least two other participants’ courses. Those who stay on then implement their findings in their own classrooms, while continuing to meet in small groups to share and troubleshoot.
Mikelle Nuwer tends to teach large lecture courses in Oceanography — and like many faculty, she used to rely in large part on a traditional lecture format. But reading and discussing the evidence on active learning, and receiving support from EBT peers as she implemented new strategies, has fundamentally changed her teaching, she says.
“EBT has been really important in showing me again both why I love to teach and how I can do it better,” says Mikelle Nuwer, senior lecturer in Oceanography. Photo credit: Shantelle Liu.
Now, she relies on active learning techniques that have been shown to be more effective at engaging students. These include assigning more collaborative group work and think-pair-share activities, and using clickers to increase engagement. Nuwer has seen the results for herself, she says. Even in large lectures, more students participate more actively, more often.
“Because of the diversity of our students, having a toolbox — tools to engage different learning styles — is really important,” she says.
And faculty are using evidence to do more than just restructure class time. Research is continuously being published on everything from online and hybrid teaching, to assessment strategies, to creating inclusive classrooms.
“On any aspect of teaching, there’s evidence out there,” says Kristi Straus, who recently stepped down as EBT lead to serve as acting director of the Environmental Studies program. “In EBT, it’s the job of lead faculty to help other faculty find that literature, and then help them apply it to their own contexts.”
Opportunities for original research and leadership
While most EBT participants join an Exploration or Implementation group, some connect to the program in a Research group. These participants focus on conducting their own classroom research to close gaps between teaching and learning . “Faculty may not have the critical mass or the expertise in their own departments to do research on teaching,” says Colleen Craig, senior lecturer in Chemistry and Academic & Student Affairs Teaching Fellow. “A crucial thing that EBT offers is an outlet for high-quality, rigorous education research in faculty’s own courses.”
The program also offers unique opportunities for leadership. Participants receive mentorship and collaborate with peers throughout each phase of EBT, but they can also move up the mentoring ladder to become coaches, and continue to learn through leading others.
Building community around teaching, across disciplines
Ian Schnee, lecturer in Philosophy, makes use of polling tools such as Poll Everywhere in his large lecture classes – an evidence-based technique to increase student engagement. Photo credit: Shantelle Liu.
Ian Schnee says that finding a community of faculty invested in teaching has been one of the most valuable aspects of EBT. “It’s great to meet all these other instructors whose fundamental mission is to teach well,” he says. Nuwer agrees. “The support we’ve gotten to create plans and implement new tools has been incredible,” she says, “but so are the things we learn from each other, and the ability to discuss our challenges and successes.”
The interdisciplinary peer group can have an enormous, even surprising impact. Approaches more common to teaching in the humanities can help STEM instructors think differently about their own habits, and vice versa. Meanwhile, says Schnee, “it becomes apparent that we do so many things in common. Everyone teaches critical reasoning; everyone strives to do more than teach facts, but to teach students how to apply facts and problem-solve,” he says.
Nuwer, for one, developed new assessment techniques for her Oceanography courses after observing how EBT colleague Gillis-Bridges evaluates student work in her English courses. “Kimberlee’s assessment tools were so different,” Nuwer says. “To see what she expects from her students inspired me.” Now, Nuwer uses a combination of assessments in addition to exams, including having students create videos or posters — techniques borrowed from Gillis-Bridges and other colleagues in other fields.
Ultimately, these interactions across disciplines benefit students. “Because I have more diversity of assessment, I see all my students do better,” says Nuwer. “It allows more students to show different kinds of strengths.”
Looking ahead
After three years as a pilot program, EBT has found a new home in the Center for Teaching & Learning. “It’s a natural fit with the expertise of CTL consultants,” says Colleen Craig, who now works with the CTL to support the transition and the program’s continued growth. “We’re thrilled to take on the EBT program,” says CTL Director Beth Kalikoff, “because it runs on the creative and scholarly energies of UW faculty.”
“In addition to building community among teachers at UW, EBT is a way to contribute to the broader community of education research. It all has downstream benefits.”
– Colleen Craig, senior lecturer in Chemistry
The program has the potential to grow in a number of ways: by bringing in more faculty from more departments; by encouraging new faculty to participate in their first few years teaching; and by supporting original research from EBT participants on the scholarship of teaching and learning — research they can publish and add to the growing body of evidence.
And EBT’s impact can extend to the departmental level, as departments such as Public Health and Philosophy have discovered. The Philosophy department has launched a new committee, led by Schnee, to review findings from EBT groups and potentially implement EBT-based techniques across Philosophy curricula. “The EBT program is helping the Philosophy Department to bring best practices to the entire department, helping us to think outside the box to make our courses more engaging and effective,” says Andrea Woody, chair of Philosophy.
Katie Kirkland, project manager in the Office of the Provost, managed the pilot and worked closely with EBT leads to develop the program in its first two years of growth. “EBT not only helps the university fulfill its mission to be a superb teaching institution — to be not just an R1 but a ‘T1’,” says Kirkland. “It has also led to greater collaboration among departments across campus that support teaching faculty.”
Kalikoff agrees. “EBT is changing cultures of teaching at UW,” she says, “because the faculty participants, coaches and leads care so deeply about sharing their passion for discovery with students and with each other.”
A geography workshop paired its students with nonprofit organizations to help solve critical community issues, offering a model of how to bring service learning into the classroom.
Sarah Elwood, Professor of Geography. Photo credit: Lisa Faustino.
Two fundamental questions help shape Professor Sarah Elwood’s approach to teaching a capstone course in the Department of Geography.
“What do I want my students to learn? And what does the world around us need?”
These are important issues for Elwood and the students who participate in the Geographic Information Systems (GIS) Workshop, an upper-level course that uses emerging technologies and the power of spatial data to tackle the critical needs of community organizations serving underrepresented populations.
“One of the many things we do in the geography department is to share with students our long-term commitment to public service and scholarship,” says Elwood, who has been widely recognized for her work inside and outside the classroom.
“We have a deep commitment to giving students meaningful assignments that engage them in community service and prepare them with the skills they need once they leave the University,” she says.
An example of that commitment was the most recent GIS workshop that Elwood taught, with about 40 students and two TAs working alongside organizations such as the South King County Food Coalition, Northwest Justice Project, Statewide Poverty Action Network, and Real Change News, a newspaper sold by the homeless in the Seattle area.
“All of the groups have pressing needs, and the urgency of the work makes the classroom experience real for the students,’’ Elwood says.
Linking class work to real world experiences is a valuable teaching and learning tool. And the UW geography department offers a model for other departments to emulate those experiences in their own workshops or classes.
It is certainly an approach that has worked well for the students, offering a tough challenge with a great payoff, said Jacqueline Ines, a former student who worked with the South King County Food Coalition, a group of 12 food banks collectively serving more than 35,000 families each year. The coalition partnered with the workshop to help find unserved potential clients and develop strategies to connect people in need to food bank services in a decentralized geographic area.
“This was one of my favorite projects to work on because it was equally fun and stressful to have a real-life audience to present our work to who would be utilizing the findings we ended up producing,” Ines says.
Mapping data to help communities solve problems
Geography students learn to use GIS — at its most basic, a software for gathering, managing and analyzing geographic data. And because it is deeply rooted in geography, that data is integrated into maps — where information is more easily brought to life.
Practically every field of study uses GIS to make maps that help them better understand an issue and solve problems. Whether someone is studying loss of habitat in the swamps of Florida, the effects of climate change on economically undeveloped island nations, or the opioid epidemic in the Midwest, you can bet that GIS tools are being used to map it all in order to understand the root causes and ultimately offer solutions.
These are some of the reasons Professor of Geography, Timothy Nyerges founded the workshop nearly two decades ago. He wanted students to use their GIS skills to gain a better understanding of environmental and sustainability issues. Elwood started teaching the workshop shortly after she came to the University in 2006, shifting the focus of her course to social justice and poverty issues. Today, professors take turns teaching the workshop, with topics switching back and forth. Professor Suzanne Withers will be teaching the workshop in spring 2019.
But no matter the field under study, Elwood says acquiring and working with data is difficult — and often quite expensive. That makes it inaccessible to community organizations, including food banks and environmental nonprofits, seeking to improve their decision-making. That’s where the partnership with the UW students comes in.
GIS has migrated to the web and cloud computing, and accessing, analyzing and applying the needed data is a complex undertaking. Putting a map together for use by a client such as a food bank requires specialized skills and teamwork, and GIS students are versed in a number of the technologies needed.
As geographers, says Elwood, “we bring the technical expertise, but our students quickly learn that our partners bring rich community expertise.”
How one student project showed food banks where to focus efforts
Students work closely with their South King County Food Coalition community project partners during the workshop. Courtesy of Sarah Elwood.
Consider the work of Ines and her fellow students with the food coalition, Elwood says.
“We use Census data in a lot of our GIS work,” Elwood said. “But the Census doesn’t ask, ‘Are you hungry?’ And that’s a fundamental question that needs to be asked if we are trying to analyze gaps in services in a particular community.”
The South King County Food Coalition was seeking to answer that fundamental question, as well as several others, when it started working with the geography department. The coalition has been considering whether to start a mobile van service to deliver food to some of their most vulnerable clients, including elderly residents living in isolated areas with no public transportation and single parents who have little time to travel to a food bank. But it needed to know where those clients were.
Ines and her team set to work. Using GIS, they identified and mapped regions in South King County that were underserved by the food coalition, explored the prospect of introducing a mobile delivery service to the Des Moines Area Food Bank’s operations, and compiled an analysis of data to locate potential long-term volunteers to help grow the coalition’s fresh produce at Elk Run Farm.
“While we all collectively worked on the end-product, we were able to split parts of the projects out to those who were comfortable extracting and analyzing the types of data we had to investigate for each individual part,” Ines said. “Two of my partners focused more on the bivariate choropleth map (a map that uses color to show quantities within a geographical area) because they did the most research on the data that was used to create that map, while another partner and I were primarily responsible for the hot spot map (a map that uses statistical analysis to define areas of high occurrence versus areas of low occurrence).
I think we created a helpful map for them,” she said.
And their work is likely to pay off in big ways, Elwood said. “They built a realistic model that showed a need in several areas.”
Ultimately, the workshop is not just about accurately mapping data. “We want our students to bring together what they have learned here over four years about technology, race, class and poverty,” Elwood said, “and integrate these lessons so they can effectively support communities that are often marginalized and underrepresented.”
Elwood’s top tips for successful community partnerships:
Look for opportunities to bring real world experience into the classroom
Elwood, and Nyerges before her, identified real world applications for what they’re teaching their students. “Our workshop is not just about geography,” Elwood said. “My students learn about social justice issues, public service and collaboration.” They suggest looking for real world applications in your field, such as what the Henry M. Jackson School of International Studies did for its capstone projects.
Define the relationship
“I don’t use the language of ‘client.’ They are colleagues, project partners,” Elwood says about community organizations that agree to work with workshop participants. “We are working together to find solutions, so we strive to teach humility, and learn humility.” She also makes sure that any organization is committed to a long-term partnership with the UW. She suggests defining what the workshop needs as well as what partners need, and ask organizations to “Be a good partner to our students.”
Balance your teams
Students bring all kinds of experiences, and that’s particularly important in a workshop that relies heavily on tech tools. So make sure each student team is balanced, such as Ines’ team. And don’t overlook the intangibles. UW students come from all social and class backgrounds, and they have much to learn from one another. Help them recognize how these differences can make for a better team.
A new course add-on option adds short-term travel to international, online collaboration — helping more students to have rich global learning experiences, at home and abroad.
Teaching sustainability through international partnership
Kristi Straus, lecturer, Program on the Enivronment
Kristi Straus, lecturer in the College of the Environment, knew that her students could learn an enormous amount about sustainability issues if they could place them in more global context. But traditional study abroad programs aren’t always feasible, or accessible, for many students.
So when Straus was approached by the Academic & Student Affairs in Fall 2017 to design and pilot a new “global flip” course model, she jumped at the chance to add a short-term study abroad option to her introductory course on sustainability. Straus partnered with professors at Tsinghua University in Beijing to design an international — and internationally collaborative — component to her ENVIR 239 Sustainability: Personal Choices, Broad Impacts course. The 15 students who enrolled in the “global flip” option (of 80 total students in the course) worked alongside their peers in China to tackle real environmental issues in both countries — first online, then in-person — without the time or cost of traditional study abroad.
Using technology to “flip the classroom” on a global scale
Straus had long practiced the “flipped classroom”: in which students study course content outside of class via recorded lecture or course texts, and do more active learning during class time. The idea — increasingly popular nationwide — is that “flipping” makes best use of learning time, as instructors can support students through the more applied learning activities in the classroom.
A “global flip” course combines a flipped classroom with Collaborative Online International Learning (COIL). Sometimes called “virtual exchange,” COIL refers to any method that uses technology to bring global experiences into classrooms or curricula. In COIL courses, faculty in different nations loosely sync their curricula so that students collaborate on projects in international groups.
In Straus’ course, “flipping” a COIL course meant that students read materials and watched lectures outside of class; during class, they worked collaboratively amongst themselves and with their Tsinghua peers. Throughout the course, students on both campuses connected online — via WeChat, online discussion boards and video conferencing — to think critically about how personal choices affect social, economic and environmental sustainability. They shared their experiences with assignments and activities, and compared sustainability issues, policies and cultural norms in the U.S. and China.
“It is certainly innovative,” says Straus of combining COIL with a “flipped” classroom. “It helps us to learn together across boundaries, to use technology and the skills of our students, while also teaching our students multicultural, multi-disciplinary collaboration and problem-solving, virtually. I can’t imagine more important skills for this generation.”
Maximizing the short-term study abroad
UW Students at the Gaobeidian Waste Water Treatment Plant learned about how waste water is managed in Beijing. Photo credit: Shunxi Liu.
In addition, the “global flip” includes short-term study abroad — adding travel and on-the-ground applied learning to the experience.
At the end of the term, during exams week, the UW students traveled to Beijing for 10 days. Together with their Tsinghua peers, UW students attended lectures, went on field trips and collaborated on course projects. They saw Chinese sustainability efforts at work through field trips to industry and government labs, and vast solar and wind farms. They also explored Beijing, including the Great Wall and the Forbidden City, to learn about Chinese culture and history, and to think about how cultural and political norms influence sustainability.
A short-term study abroad allows more students to access immersive international experiences. The cost of going abroad is much lower than a full-quarter program, and students with tight timelines to graduation or restrictive course requirements in their majors can more easily manage the travel schedule.
Placing learning in global context
For UW students, says Straus, there is enormous value in having cross-cultural conversations around pressing global challenges, including sustainability — whether those conversations take place online or in person. Students not only learn about another nation’s sustainability issues and policies, but they learn that sustainability has everything to do with cultural norms and ways of thinking about the environment.
Straus emphasizes that the diversity among both UW students and Tsinghua students enhanced everyone’s learning experience as well. About half of her 15 UW students were international students from China — and many of the Tsinghua students were international students as well, from Spain, Brazil, Hong Kong and elsewhere. For the UW Chinese international students, the course offered an opportunity to work with scholars at the prestigious Tsinghua, and to learn about environmental sectors in China where they might return to work after college. For all students at both universities, international and domestic students alike offered distinct insights around relationships between sustainability and culture.
A model with far-reaching impact
“This program is not just about environment and sustainability. It’s about comparing two countries and building a community of global citizens.”
— Mike Liu, class of 2018, Environmental Studies major
The “global flip” builds potentially long-lasting connections across institutions, faculty and even nations. “I see this program as planting the seed for future environmental collaboration between the U.S. and China,” says UW student Shunxi Liu, “and it was exciting to be part of it.”
This highly transferable model is inspiring others, across disciplines. UW instructors in departments ranging from English and Philosophy to Oceanography are currently seeking partners in countries around the world to teach global flip classes of their own. Proposed courses include titles such as PHILOSOPHY 149: Existentialism and Film and ENVIRONMENT 300: Diversity and Ecology of Coral Reefs.
“Global flip” tips
Sync schedules carefully. Syncing courses across nations means accounting for different scheduling challenges, such as time differences. But it also means making sure that academic calendars can be coordinated. In addition, advance planning is needed to make sure that assignments follow similar sequences, and that topics are introduced at a similar pace.
Set clear expectations for all partners. From the start, communicate with partner instructors to make sure expectations are clear for all sides (for example, what will collaboration look like? For students? For instructors?). Keep lines of communication open, and check in with partner instructors regularly.
Recruit early (and everywhere). A mix of student disciplinary backgrounds, and of international and domestic students, brings diverse perspectives and adds to everyone’s experience, says Straus. Make use of advisors and reach out to specific schools and colleges to recruit a wide range of students. Students can also be encouraged to apply for scholarships through the Office of Global Affairs — which can often cover the full cost of short-term study abroad.
Choose tools strategically. Decide what COIL platforms will be easy and accessible for students on all campuses to use. Also, get help from UW Academic Technologies, who Straus says were an enormous help in navigating the complex challenges of the Chinese firewall.
UW Study Abroad partners with faculty to develop, plan and run programs on programmatic, logistical, and health and safety fronts.
UW Bothell COIL Initiative: A fellowship program to support faculty and staff across UW’s three campuses to establish online international collaborations.
UW Tacoma COIL Fellows Program: A fellowship program that links UW courses with courses in different countries through real-time and asynchronous technologies.
Global Innovation Fund: A fund to provide initial support for initiatives and programs that enhance the UW’s global engagement and reach.
UW librarians are our community’s resident experts in research and information. On all three campuses, librarians are hard at work creating new resources to help students — and instructors — evaluate the information (good and bad) that surrounds us every day.
Amanda Hornby, head of Teaching & Learning at UW libraries.
“Libraries have a long history of teaching literacies that are needed not only for research but for everyday citizenship,” says Amanda Hornby, head of Teaching & Learning at the UW Seattle campus libraries. “It’s something the UW libraries care deeply about, and have been thinking about for years.”
Lately, Hornby says, UW librarians are getting more requests from instructors to support curriculum design — specifically around information literacy and source evaluation. Across our community, librarians are creating new ways to help instructors and students develop these critical skills.
Navigating the new information landscape
Nia Lam, media studies librarian at UW Bothell, says that working with sources today means a fundamental shift in thinking about information. When information can be created and disseminated almost instantly online, she says, we need new ways to evaluate credibility, and new ways to put information in context.
“As UW librarians, we’re on the front lines in terms of thinking about and teaching critical inquiry, information literacy and research skills, and we collaborate with educators across disciplines to foster these lifelong skills in students.”
— Nia Lam, UW Bothell media studies librarian
“We try to talk more about the whole process that goes into creating information,” Lam says, including how long it takes for different types of sources to be produced. Sources that can be produced quickly (such as social media posts) may offer more immediate perspective, but less depth of knowledge, than sources that rely on more involved processes to produce (such as journal articles). To provide an accessible resource on the topic, UW Bothell librarians Chelsea Nesvig and Chloe Horning created a short video that walks through the “information timeline” — through the story of a baby elephant. An event — such as the premature birth of a baby elephant at a local zoo, of interest to scientists and the general public — will be covered by social media, then newspapers, then magazines, all before scholarly articles or books, which take longer to create.
In general, says Lam, “source evaluation is a big part of information literacy — and this has changed over time.” When articles are read in the context of a larger, print publication, it’s easier to use context cues to gauge credibility (for example, “Butter Can Kill You” carries different weight if read on the front page of the National Enquirer or the New England Journal of Medicine). But when everything is on a screen, Lam says, it can be harder to tell the difference between a magazine, newspaper or scholarly journal article, or to tell if the publication is legitimate. On all three campuses, UW librarians consult with faculty in creating research assignments to make sure students get practice working with different types of sources (not just scholarly journal articles). “Students need to be able to assess credibility in every text they read,” Lam says, whether a tweet or a tome.
For UW Tacoma instructional design librarian Marisa Petrich, the new landscape requires other shifts in thinking about information as well. It’s important to teach students how to spot unreliable information, Petrich says. But it’s just as important to teach what good information looks like, so that they know how to identify it — and, how to use or write it themselves. “We need to invite students to think of themselves not only as people who consume information, but as people who produce information,” she says.
Supporting instructors in any field, before courses begin
UW librarians are invaluable resources for instructors — not just while courses are in session, but from the beginning of the course design process. More and more instructors have been consulting with librarians on best practices for teaching information literacy and source evaluation before courses begin. “We’re seeing instructors in a variety of disciplines wanting to revise their curricula and learning outcomes,” says Hornby, “to make sure these are key areas they hit with students.”
“It’s always been true that the information we use matters. Its context matters. Now, we’re in a moment when we can show people how it matters, in ways that really hit home.”
When Hornby consults with instructors in any discipline, she asks a series of questions to prompt reflection around how these skills show up in curricula, including: “How do you build in time for critical thinking and evaluation of sources into your course or assignments?” For Hornby, it’s key that students encounter a diverse range of texts and voices in course readings — in general, but also to support source evaluation skills.
Instructors on any campus can also upload to their courses three Canvas Commons modules, created by Petrich, on identifying good sources and detecting bias. Susan Johnson, assistant professor of nursing and healthcare leadership at UW Tacoma, says that after using the modules, her students report being more critical about what they share on social media and “more confident in their ability to spot disreputable sources.” The modules are an invaluable resource, Johnson says, because “it’s something that I didn’t have the expertise to do.”
Subject librarians support discipline-specific information skills in class and beyond
Nearly every UW department has a subject librarian with special expertise in the field. Subject librarians support students at any step in the research process, and increasingly, visit classrooms to teach research skills upon instructor request.
Kian Flynn, Geography & Global Studies subject librarian.
Kian Flynn, Geography & Global Studies subject librarian, says his field is one where students need a special set of information literacy skills in the digital age. Students often need to find and use data, he says — for example, census data to build a map — but sometimes, students are overwhelmed by the process of finding authoritative data sources. So Flynn began supporting instructors to bring more data and statistical literacy into curricula. In winter 2018, Flynn worked with a Geography course on using census data to create maps — holding a classroom session, creating a follow-up quiz to prompt further reflection, and working with individual students throughout the term.
“In and beyond the classroom,” Flynn says, “unpacking data visualizations is a skill students need.” In popular media, he points out, we see complex data and map visualizations all the time — and map-making software will continue to develop and be more accessible to the general public. “The power of maps and graphs is that they connect with us,” Flynn says; they can make sense of data in a visual way, but can be particularly dangerous when they misrepresent data. “We need the skills to make good decisions,” he says, for reading maps and graphs, but also for creating them.
For Jessica Albano, communication studies librarian, visiting communications classrooms to talk about source evaluation is a big part of the work she does to support students in the field. In addition to individual sessions, she says, she hopes instructors will continue talking more about these skills throughout their courses. Hornby agrees. “It does take time,” she says, “but I feel very strongly that slowing down and giving students time to unpack sources and arguments gives them a different ownership over the media they consume every day.”
Workshops and events on fake news and misinformation
The event “Why Journalism Matters: News Literacy in a Democracy” included a talk by Thanh Tan, current multimedia editorial writer for The Seattle Times. Photo credit: Cathy You of the UW Daily.
From workshops and events to interactive displays, UW librarians engage the community around information literacy in a variety of ways.
A News Literacy workshop series at UW Tacoma in spring 2017, designed by librarian Marisa Petrich, discussed fake news and misinformation, the reasons for its prevalence and how to evaluate it. The three-part series was designed for students, Petrich says, but was also well-attended by faculty. “The series was invaluable to me as an instructor,” says Ellen Moore, senior lecturer in the School of Interdisciplinary Arts and Sciences at UW Tacoma. “It provided a key resource for our campus, where building a critical perspective on the media as a whole provides the foundation for student learning.” A media and news literacy conference designed by librarian Jessica Albano with the Department of Communication — “Why Journalism Matters: News Literacy in a Democracy” — provided a forum for students, instructors and staff to discuss journalism, media responsibility and ethics, and related topics in April 2017. The half-day event kicked off with an exclusive interview with Washington State Attorney General Bob Ferguson, focusing on the First Amendment and the role of a free press in a democracy.
A “Making Sense of the News” display in Suzzalo Library includes posters, example sources and evaluation checks, and interactive pieces: for example, a “credibility scale” on which visitors place magnets featuring logos of various news organizations, and a “Where do you get your news?” poster to which community members have been continuously adding. Albano says they’ve received several requests from high school librarians wanting to adapt the display for their own students.
As part of the interactive “Making Sense of the News” display in Suzzalo, visitors could place various news outlets along a reliability and standards scale. Photo credit: Cathy You of the UW Daily.
Resources for students and instructors
At any time, in any course, students and instructors can access these librarian-created resources on finding, evaluating and using information — including, on fake news and misinformation.
An Instructor Toolkit in the teaching support section of the UW libraries page connects instructors to specific types of librarian support. “When instructors in a given field type in ‘fake news’ or ‘source evaluation,’ for example, we can connect them with the right librarian — to support them on assignment design, in the classroom or on Canvas,” says librarian Amanda Hornby.
Savvy Info Consumers: Evaluating Information provides checklists for evaluating information (the SMART Check, the CRAPP Test, the 5 W Questions) and applies them to different types of information.
Savvy Info Consumers: Detecting Bias in the News offers many common examples of bias to evaluate, including “Bias by headline,” “Bias by photos, captions and camera angles,” and “Bias by selection and omission.”
Savvy Info Consumers: Fake News outlines definitions of “misleading,” “highly partisan,” “satire,” “clickbait” and “fake news” along with several examples of each, and prompting questions to distinguish the difference.
A news research guide links to several external sources on fake news, as well as fact-checking sites.
An “Information Timeline” video walks through the process of information creation — how long it takes different types of sources to be produced and shared — through the story of a baby elephant.
Canvas Commons modules on News Literacy and Reliable Sources help students evaluate sources and bias. The three modules can be uploaded by any UW instructor, into any course.
Journalism Ethics and Standards, a subject guide created for Communication Studies, includes the Society of Professional Journalists Code of Ethics and other resources about ethics and standards in journalism.
A spring mini lecture series offered students, faculty, staff and the community new ways of understanding and combating fake news and misinformation.
It’s hard enough to understand the problems of fake news and misinformation — the complex factors that have contributed to them becoming primary issues facing our society — let alone, how to combat them.
UW students at Jevin West’s lecture, “Cleaning Up Our Polluted Information Environments.” Photo credit: Shantelle Xiu
“As a community, UW thrives in our access to all types of information — we are constantly producing and consuming information. This lecture series reminds us as a community to protect our integrity as information producers, and as ethical information consumers.”
— Tim Tiasevanakul, class of 2020, Law, Societies and Justice major
Enter the experts: Kate Starbird, UW assistant professor in Human-Centered Design and Engineering; Jevin West, UW assistant professor in the Information School; and Berit Anderson, CEO and Editor-in-Chief of the media company Scout.ai. These three have extensive experience working in the information trenches, in a variety of ways. Starbird and her team of UW students have been researching social media platforms for years, to better understand how rumors, “alternate narratives” and conspiracy theories spread after crisis events. West is a data scientist who co-developed the wildly successful UW course, “Calling Bullshit: Data Reasoning in a Digital World.” And Anderson, a technology journalist and former editor of Crosscut Public Media, started her company with the goal to “democratize technology,” just as fake news undermines democracy.
The series, which was sponsored by the Office of the Provost and ran through April 2018 on the Seattle campus, attracted students, faculty and staff from across the UW, as well as members of the local community. It demonstrated how hungry we are, as a community and as global citizens, for ways of understanding and tackling these problems — and offered us knowledge and strategies from three engaging, expert perspectives. All three lectures are now available to watch online in full.
Kate Starbird: Understanding the “muddied waters”
Kate Starbird, UW assistant professor in Human-Centered Design and Engineering
As Kate Starbird told us on April 18, “We are all targets of disinformation, meant to erode our trust in democracy and divide us.” This doesn’t mean we have to be completely vulnerable — but for Starbird, it’s crucial that we understand why we’re as vulnerable as we are.
Starbird’s lecture, “Muddied Waters: Online Disinformation During Crisis Events,” opened with a discussion of what makes human beings susceptible to disinformation and political propaganda. In general, says Starbird, it’s our unconscious cognitive biases that lead us to take stories, real or faked, as “true” — when they confirm our pre-existing beliefs. And our biases can be targeted by technology companies. For example, social media algorithms create “filter bubbles” by showing us, and getting us to click on, what we already want to see. It has become increasingly possible to only encounter information of the kind we want or expect to see, on each side of the political divide — making the divide ever greater.
“If it makes you feel outraged against the other side, probably someone is manipulating you.”
– Kate Starbird
Starbird and her students have been analyzing the marked increase on social media in the spread of disinformation: information disseminated with the intent to confuse, on both the left and right of the political spectrum. They study Twitter, for example, to see how disinformation follows crisis events such as school shootings, and how those rumors “muddy the waters”: casting doubt on the credibility of those who experienced the crisis, and even on the crisis itself. “Disinformation is very effective,” she says, at muddling our thinking. And when we, as consumers of information, feel muddled, we give up trying to understand — and worse, are confused into inaction.
Jevin West: Combating information “Polluters”
Jevin West, UW assistant professor in the Information School
Jevin West, in his April 24 lecture “Cleaning Up Our Polluted Information Environments,” discussed the problem as largely one of “un-trained editors.” Where we once relied on professional journalists, editors and fact-checkers to serve as information “gatekeepers,” we can all now share information widely, but without clear ways to evaluate credibility. In addition, West said, studies have shown that falsehoods travel faster than truths — and it takes far less energy to create bad information than it does to refute it after the fact.
West displayed examples of fake news in its everyday forms, from the more benign (flat earth conspiracy theories) to the more dangerous (false medical information) — and explained how new technologies will make it continuously harder to tell what’s real from what’s faked. He also showed the audience how easy it is to misrepresent information through statistics, and to create graphs that can seem to say anything. We’re especially vulnerable when it comes to numbers, he says, because “numbers carry authority” — and it can be very hard to tell when they are presented out of context.
For West, fake news and misinformation pose an “existential threat.” “I think it’s the most serious issue we’re dealing with in society right now,” he says.
Berit Anderson: Tracing the rise of “AI Propaganda”
Berit Anderson, CEO and Editor-in-Chief of the media company Scout.ai
Following the 2016 U.S. presidential election, Berit Anderson and her colleagues at Scout began investigating the causes for what seemed like surprising results key swing states. What they found, she says, is that “there’s a new global electioneering platform built on the backs of the tech industry.” In early 2017, they published an article entitled, “The Rise of the Weaponized AI Propaganda Machine” detailing how automated propaganda networks can influence elections around the world, which garnered attention and support from some of the biggest players in the industry, including Google.
Anderson’s lecture on April 30, “The New Global Politics of Weaponized AI Propaganda,” outlined the steps that have allowed Russia, in particular, to target elections in the U.S. and Europe. These include, “Turn voters into disinformation agents” via companies such as Facebook, who allowed advertisers to target voters in swing states by zip code and income. Another tactic is to “Deploy the bot armies” — that is, create vast networks of fake social profiles to spew politically divisive, often faked “news.” In some cases, the same bot accounts have been deployed in different parts of the world in support of different agendas, from the Arab Spring to Brexit to the 2016 U.S. Presidential election.
Looking ahead to solutions
The series as a whole provided powerful strategies for combatting these problems — many of which overlapped across the three talks.
“A big take-away from all three lectures was that I have a responsibility as an individual to be more self-aware and a better consumer of information. I walked away feeling empowered.”
– Katie Harper, graduate student in Library and Information Science
All three discussed how new policies and better regulation of tech companies (or “pollution facilitators,” in West’s terms) can help. For example, the new E.U. General Data Protection Regulation (GDPR) will hold Facebook and other companies accountable to clearing users’ histories when requested, making it more difficult for “polluters” to exploit personal information. Anderson emphasized how new policies can help create change internally at tech companies as well. Facebook has taken a “big step,” she says, by committing to requiring advertisers to label political ads as such, and provide “paid for by” information. Somehow, we need to “restore trust in our information systems,” says Starbird — and that means that tech companies need to be more transparent and trustworthy.
At the same time, all three emphasized the ways in which individuals can help — by becoming more conscious information consumers and producers. A good rule of thumb, says West, is to “think more, share less”: to become more thoughtful and selective about the information we share, thus becoming better “editors.” We should also pay attention to how information affects us emotionally and engages our biases, says Starbird. “If it makes you feel outraged against the other side,” she says, “probably someone is manipulating you.” “Don’t become a cog in the outrage machine,” echoes Anderson. “The most important thing you can do as an individual is not to let yourself become angry at people with different political views,” she says.
“It was so interesting to learn about the fake news and misinformation that we might encounter every day through popular media. This series as a whole is valuable to the UW community because it spreads awareness about misinformation, and increases our knowledge of ‘fake news’ and credible sources in media and otherwise.”
– Selah Lile, class of 2021, pursuing double major in Psychology and Spanish
For West, the most powerful solution lies in education — especially for younger generations as they learn information literacy. “The biggest thing that we can do is arm the consumer,” he says, with education. Requiring media literacy as part of grade school curricula, for example, could go a long way toward creating a more critical and savvy consumer public. (In 2017, Washington state passed a bill into law to do just that.) Starbird and Anderson agree. We have to understand how online media works — “how it affects our lives, our economy and our global politics,” says Anderson.
The speakers suggested other ways to address the problems as well, including West’s prompting to make better use of existing resources such as reliable fact-checking organizations, and Anderson’s suggestion to reach out to individual developers we might know at tech companies here in Seattle, to work toward change together.
Anderson says she’s optimistic: “I’ve spoken to politicians in Europe and also here in the U.S., and I’ve found that people are very motivated to find a fix for this,” she says. “It’s a time to be bold and stand up for the things we believe in.”
