Engineering  |  Honors and awards  |  Research  |  Science  |  Social science  |  Technology  |  UW News blog

November 28, 2017

UW students win Amazon’s inaugural Alexa Prize for most engaging socialbot

A team of University of Washington students and faculty has won Amazon’s inaugural Alexa Prize, a university competition designed to produce an artificial intelligence agent capable of coherent and sustained conversation with humans.

The UW team developed Sounding Board, a conversational agent designed to provide engaging and informative conversation and to transform how people interact with everyday devices in their homes. The team from the UW Department of Electrical Engineering and the Paul G. Allen School of Computer Science & Engineering took home the $500,000 first prize, which will be shared among the students.

The UW Sounding Board team (left to right: Hao Fang, Hao Cheng, Ari Holtzman, Mari Ostendorf, Maarten Sap, Elizabeth Clark, Yejin Choi) wins Amazon’s inaugural Alexa Prize. Credit: University of Washington

Their challenge was to produce a socialbot — an AI agent capable of coherent conversation — that could converse about popular topics and current events for a goal of 20 minutes. Teams built their socialbots using the Alexa Skills Kit and received continuous, real-world feedback from millions of Amazon customers who interacted with teams anonymously through Alexa.

Amazon selected the winning team from three worldwide finalists on Tuesday at the AWS re:Invent 2017 conference in Las Vegas.

“Our philosophy in developing Sounding Board was to bring a variety of relevant content into a natural conversation,” said team leader and electrical engineering doctoral student Hao Fang.  “Ultimately, we hope Sounding Board can become a conversational gateway to online information that users enjoy talking with.”

The Sounding Board socialbot earned an average score of 3.17 on a 5-point scale from a panel of independent judges and achieved an average conversation duration of 10:22.

The runner up team Alquist from Czech Technical University in Prague, which attained an average score of 2.72 and had an average conversation duration of 3:55, received a $100,000 prize. The third-place team from Heriot-Watt University in Edinburgh, Scotland, received a $50,000 prize for an average score of 2.36 and an average conversation duration of 4:01.

The UW Sounding Board team combines expertise in natural language processing, speech technology and human-AI collaboration from additional team members EE doctoral student Hao Cheng and Allen School doctoral students Elizabeth Clark, Ari Holtzman, and Maarten Sap. EE professor Mari Ostendorf is the lead faculty advisor for the team, working in collaboration with professors Yejin Choi and Noah Smith of the Allen School’s Natural Language Processing research group.

“The students started from scratch, with no experience building a dialog system or working with Alexa skills, but together they brought a breadth of perspectives on language processing and a passion for understanding both the technical and human factors challenges of conversational AI,” Ostendorf said.

The Sounding Board design is both user- and content-driven. The system aims to understand user comments in multiple dimensions, from directives to sentiment and personality, in order to best serve user interests. At the same time, the system relies on having interesting and timely things to talk about. It actively harvests online content and leverages a knowledge graph to provide connections between related topics that can be used to steer the conversation.

“Sounding Board is unique in its ability to understand what type of person the user is, and is able to adjust parts of the conversation based on who it thinks the user is,” said the Allen School’s Sap.

The UW team relied on the collaborative environment at the university, both for getting feedback on technical ideas and for user testing. Faculty and students from across the UW-NLP community — in computer science, electrical engineering and linguistics — provided input on the many different versions of Sounding Board as it evolved.  In addition, a key resource in system development was access to real Alexa users nationwide. “It is impossible to anticipate all the types of reactions and questions people will have, even the different ways that a simple yes-or-no question can be answered. Learning from actual user data is critical,” Ostendorf said.

More than 100 teams from universities in 22 countries applied to be part of the inaugural competition. The finalists were selected from among 12 semifinalists whose socialbots were evaluated based on customer ratings of their interactions during hundreds of thousands of conversations last summer.

The three finalists announced in August continued to improve their socialbots by leveraging customer interactions through Nov. 7, and Amazon selected the winner based on assessments of a panel of judges listening to conversations with three interactors.

Amazon will publish technical papers from all participating teams in the Alexa Prize Proceedings as a way of sharing their work with the broader research community.

“We envision that conversational AI will be integral at the interface between humans and machines, and the Alexa Prize makes an important step toward that vision,” said Choi. “It has been an exciting journey to build Sounding Board, and we look forward to working on crucial research challenges that we have identified along the way.”