January 3, 2011
Engineering students hack Kinect for surgical robotics research
The UW’s Biorobotics Laboratory is experimenting with its new Kinect. In case you somehow missed it during the holiday season, this new add-on for the Xbox 360 ditches the controller by detecting a persons gestures and interpreting them as commands.
Researchers and hobbyists have hacked the system since its November launch to explore uses beyond video gaming. Microsoft, the maker of Kinect, has said it welcomes such tinkering.
Among the so-called hackers is Fredrik Rydén, a visiting graduate student from Sweden working with UW electrical engineering professors Howard Chizeck and Blake Hannaford. During a weekend in December, Rydén reprogrammed a Kinect to add sensory feedback.
“I bought the Kinect on a Friday and was done on Monday morning,” Rydén said.
Rydén and fellow doctoral student Hawkeye King then recorded a video of the modified tool and posted it on YouTube. The project has been covered on Engadget, MSNBC and seattlepi.com, and the video has been viewed more than 23,000 times.
“As soon as we saw the Kinect, it seemed like, yeah, lets play around with it. It just seemed obvious and cool,” King said. “We work in haptics, building virtual environments that touch remote locations. This is an interesting way for building virtual worlds from real worlds, and even combining the two.”
The Biorobotics Labs research in telerobotic surgery aims to make it possible for surgeons to operate at a distance on patients in disaster areas, on battlefields or in other inaccessible places. The labs research also hopes to advance existing surgical robots.
“In telerobotic surgery, surgeons are basically unable to feel what theyre suturing or cutting,” Chizeck said. “Wed like to get that sense of feeling back.”
Experiments using the Kinect are a step toward that goal.
In the video, the Kinects infrared camera points at a table where Rydén is seated. The upper right-hand corner shows King operating a stylus that gives force feedback, in the form of a push, when he meets an object in the virtual scene. The red dot shows where the stylus is positioned.
When the red dot moves over an object the stylus feels the resistance, a field known as haptics. The video shows that King can not only tap solid objects, like the table, but can also feel a new object (in this case a Styrofoam head placed on the table) and even shake Rydéns hand in real time.
“The virtual world is generated in real time from the real world,” Rydén explained. “In this case he was in the same room, but he could have been anywhere and seeing the information on the Internet.”
Another bonus, he said: “Kinect is cheap because it comes from video games and its mass produced.”
The UW team will explore the Kinects potential to improve surgical robotics by creating off-limits areas that would protect vital organs by building so-called “virtual fixtures” around body parts that should not be touched.
The Kinects simple interface, low cost and portability make it well suited for doing experiments.
“We could have had students spend several quarters to do what one student has done in less than one quarter using the Kinect,” Chizeck said. “This is just a very efficient way to track moving objects.”
“Its unlikely that any final device is going to have the Kinect in there,” Chizeck added, “but its a way to test ideas quickly and develop them.”
###
For more information, contact Ryden at 206-724-9160 or ryden@uw.edu, Chizeck at 206-221-3591 or chizeck@uw.edu and Hannaford at 206-543-2197 or blake@uw.edu.
