Frequently Asked Questions (FAQ)

  1. What is DO-IT Video Search?
  2. Why was DO-IT Video Search created?
  3. How is DO-IT Video Search funded?
  4. How does it work?
  5. Is this approach scalable to larger video collections?
  6. What video player are you using?
  7. What is the HTML5 <video> element?
  8. Are the video players accessible to screen reader users?
  9. Is the Flash video being streamed?
  10. Why are you using closed captions instead of open captions?
  11. Are the videos audio described?
  12. Can I customize the interface?
  13. What is your procedure for making your videos accessible?
  14. Is the DO-IT Video Search source code available?
  15. What is on your To-Do List?
  16. Who do I contact with feedback or suggestions?
  17. Who do I contact for support in using JW FLV PLayer or any of the other products mentioned in this FAQ?
  18. Are there other examples of video search applications?
  19. What are some other resources on multimedia accessibility?

What is DO-IT Video Search?

DO-IT Video Search is a web-based application that allows users to search the full text of DO-IT's video library. Over the years, DO-IT has produced more than 30 videos on a wide range of topics related to people with disabilities accessing curriculum, using technology, and pursuing challenging careers. our videos are full-text searchable because they're closed captioned.

Why was DO-IT Video Search created?

We had three main motivations for creating this application:

  1. To provide a means by which users can search the full text of DO-IT's video library, and quickly jump to video scenes where their search words are being spoken.
  2. To demonstrate a secondary benefit of closed captions. If accessibility alone isn't enough to motivate organizations to caption their media, our hope is that the added benefit of being able to search the full text of video archives will be added motivation.
  3. To provide a test platform for exploring accessible media delivery. See our To-Do List for our future plans in this area.

How is DO-IT Video Search funded?

DO-IT Video Search was developed with funding from the National Science Foundation (NSF, Cooperative Agreement #HRD-0227995) and continues to be updated and maintained with NSF funding (Grant #CNS-0540615). The contents do not necessarily represent the policies of the U.S. federal government, and you should not assume their endorsement.

How does it work?

All of DO-IT's videos are closed captioned for the deaf and hard of hearing. Closed captions come in a variety of formats depending on the type of media being used, but most of these are text or XML files that can easily be parsed by a simple script, and used for searching. For our purposes, we store each individual caption and its associated start and end time as records in a MySQL database. The user interface is written in PHP. When a user searches for a keyword or phrase, a MySQL full-text search query is executed, and results are delivered to the user. Results are grouped by video with the most relevant videos (according to MySQL) listed first. Each item in the results is caption text, including a link that starts playing that video in the embedded media player at the indicated time where that text is being spoken. For the nuts and bolts of the search algorithm, we depend entirely on MySQL full-text search, which we have found to be adequate. However, we are testing a variety of search queries and working to make adjustments where we can.

Is this approach scalable to larger video collections?

Admittedly, our database is quite small. We currently have nearly 40 videos, and approximately 5000 captions. We have not tested our approach on a larger collection. For comparison, see our list of other examples of video search applications.

What video player are you using?

We offer users a choice of video players, which can be configured on the Preferences page. One of the purposes of DO-IT Video Search is to provide a test lab for a variety of media delivery options. The most accessible option that we current support is the JW FLV Media Player, named with the initials of its developer Jeroen Wijering. Jeroen's player is highly customizable through a wide variety of user-defined variables and a powerful Javascript API. It also has an accessible interface for screen readers that support Flash accessibility, plus support for closed captions and closed audio description (see Question 11 for more on our support for audio description). For experimental purposes, we also provide a standard YouTube video player and a player that is generated by supporting browsers using the HTML5 <Video> element. For more on HTML5, see Question 7.

Both the JW FLV Player and YouTube player are Adobe Flash players, and one problem with Flash objects in general is that most browsers don't support keyboard access to a Flash object unless the user first gives it focus by clicking on it with a mouse. This is a problem that affects all known browsers except Internet Explorer 6 and higher. Obviously this is a problem for users who are unable to use a mouse, unless their assistive technology is able to bypass this barrier. Fortunately, both the JW FLV Media Player and YouTube player include Javascript API's that allow us to control the Flash movie externally using HTML buttons. These alternative player buttons may also be useful to other users, including users with low vision, or users who simply like them better than the standard controls. By default both sets of controls are available on the screen, but this can be changed on the Preferences page.

What is the HTML5 <video> element?

The draft 5th major version of HTML (HTML5) includes, for the first time, <video> and <audio> elements, that allow video authors to very easily add multimedia to web pages, and allow users to play these media directly in the browser, without installing a plug-in. As of March 2010, the <video> element has been tested and is working in Firefox 3.6.2, Safari 4.0, Google Chrome 4.1.2, and Opera 10.51 (Windows only). Safari supports a different video format (H.264) than the others do (Ogg). Google also voiced support for H.264 but supports both formats. Noticeably absent from this list is Internet Explorer, which does not yet support the <video> element at all.

Accessibility is currently a problem with HTML5 <video>. There is no mechanism built into the spec that allows video to be accompanied with closed captions or audio description. There have been proposals and studies regarding how best to approach this problem, and even a few experiments that successfully displayed captions using Javascript, but none of this has resulted in accessibility features being standardized within the HTML5 specification. DO-IT Video Search is currently using JQuery (specifically, jquery.srt.js) to extract captions from an .SRT caption file and display them one at a time in an empty div. This is an experiment, and only seems to work in Firefox and Opera.

Other current problems are that browsers provide little or no control of the built-in video player by keyboard. Opera 10.51 currently provides the best implementation: All controls are accessible by tab, and controllable with Enter or arrow keys. In Firefox 3.6.2, play/pause can be toggled with space bar once the player has focus. Neither Chrome nor Safari provide any obvious means of controlling the player by keyboard. The player can be controlled with Javascript, which allows accessible custom controls to be created by the web author as a temporary solution. Also, the native video controls are not currently accessible to any screen reader in any browser.

Much of the lack of accessibility can be attributed to HTML5 being a work in progress. As accessibility support improves, we will add additional accessibility features to our HTML5 implementation on DO-IT Video Search.

Are the video players accessible to screen reader users?

The JW FLV Media Player is accessible to screen reader users whose assistive technology is current enough to support Flash accessibility (e.g., Window-Eyes 4.2 or higher, JAWS 4.5 or higher). Prior to version 4.6, JW FLV Player had intermittent accessibility. Historically the player has included meaningful labels on buttons and controls that were read by screen readers. However, as new versions of the player were frequently released, the new versions sometimes broke accessibility.

The YouTube Player has a similar problem. The player is often updated, sometimes with negative consequences for accessibility. In general, key buttons are announced by screen readers, but often they are accompanied by superfluous graphics and buttons that have no meaningful label nor clear function.

As noted in the previous question, the HTML5 video player is not currently accessible to screen reader users in any browser.

Is the Flash video being streamed?

Traditionally there have been two techniques for delivering Flash video: Streaming and progressive download. Streaming requires a streaming server such as Adobe's Flash Media Server. This product can be expensive for some budgets, as can the commercially hosted solutions. The University of Washington's media services group does not support Flash streaming either, so we are currently unable to provide true streaming. The alternative, progressive download, allows video to be played as it's being downloaded via HTTP, and requires no special technology. However, it does not support direct access to any point in the video until the video has been downloaded up to that point. For our purposes, this would mean that if a search result points to a caption that appears 10 minutes into the video, the user would have to wait until 10 minutes of video had been downloaded before the user could access that video content. This is clearly unacceptable for our needs.

An alternative is PHP streaming, in which a small PHP script is used to deliver a new Flash video header, starting at any designated keypoint within a larger video. This technique has evolved since it was originally proposed by Stefan Ricther in 2005, and the entire development cycle can be tracked at FlashComGuru.com, in the articles Streaming flv video via PHP, take two and Flash Video Streaming via PHP - Optimized by xmoov.com. One of the tools that has emerged from the PHP Streaming movement is the PHP Streaming Extension for the JW FLV Player, which is the tool we are currently using to pseudo-stream our Flash video.

In order for PHP Streaming to work, the Flash video (FLV) file must be encoded with keyframes at the same time as caption start times, or at least frequently enough so that the video is sure to include a keyframe close to the time at which any caption will begin.

Additionally, the FLV file must be injected with meta data indicating the position of each keypoint. This is easily done either with FLV Meta Data Injector (FLVMDI) or Captionate, both from the Manitu Group. The latter of these products has the added benefit of providing a complete solution for adding closed captions to Flash videos.

Why are you using closed captions instead of open captions?

Those of you who are familiar with DO-IT videos know that we have a long history of producing all our videos with open captions. Open captions are mixed into the video and are always visible, whereas closed captions exist independently of the video and can therefore be toggled on and off. All our videos are originally produced for broadcast and for distribution on hard media such as VHS and DVD. We continue to add open captions to these video for many reasons:

For the above reasons open captions continue to be viable in general. However, there are at least two cases in which closed captions are a preferred means of delivering captions:

Screen capture of a DO-IT video, a blurry image viewed over a modem

Are the videos audio described?

Yes, all of DO-IT's videos include audio description. Audio description is a narrative voiceover that describes content (including key actions or screen text) that is otherwise presented only visually, and is therefore not accessible to someone who is unable to see it. Since the added narration may be distracting to some users, we have produced two versions of all our videos, one with audio description, and one without. By default, we show the audio described version of all videos in the DO-IT Video Search application. However, this can be changed on the Preferences page.

One of the reasons we selected JW FLV Media Player (described in the earlier question regarding our choice of media players) is its support for closed audio description, to our knowledge the only Flash media player that has such support. With this feature, audio description can be delivered in a distinct MP3 audio file, and can be toggled on and off by a button on the player's control bar, similar to the way closed captions work. We find this to be a nice feature because it allows us to deliver one video that meets the diverse needs of all audience members, rather than having a separate video for users who require audio description.

Since closed audio description is new to us, it is only available on videos produced in 2008 or later. The following is a complete list:

To sample audio description, you must have "Closed Audio Description" checked in your Preferences. Also, your choice of media players determines how closed audio description is rendered (none are perfect yet). JW Media Player 3.16 provides an audio description toggle button on the player control bar, identified with the letter "A". However, this version does not allow audio description volume to be controlled separately from the program audio. As a result, the audio description is difficult to hear over the background audio. With the release of JW Media Player 4.2, audio description has its own independent volume control (internally, not yet controllable by users). However, there is no audio description button so audio description can not be turned off.

Can I customize the interface?

Yes. You can change many of the default settings of the DO-IT Video Search website on the Preferences page. Doing so requires that your browser be set to accept cookies.

What is your procedure for making your videos accessible?

The following is a step-by-step guide through the process by which our videos are produced:

  1. Write the scripts. All our videos are created from scripts. Often this includes unscripted content (e.g., interviews, footage filmed on location at the University of Washington), but even this content is integrated into the script. Starting with a script results in a more polished presentation and an easier video editing process later. Also, the script content can easily be converted into captions, eliminating the extra time and cost of transcribing the video post-production.
  2. Plan for accessibility during production. Since all our videos include open captions, we are careful when filming to avoid critical content appearing near the bottom of the frame, where captions will later be displayed. Similarly, since all our videos include audio description, we are careful when scripting to pace the video in ways that ensure that there is enough space for narration to be mixed in.
  3. Produce and encode the videos. Our videos are produced by University of Washington Television (UWTV), and the final deliverable includes media for broadcast television, DVD, and the Web. For the latter, videos are encoded into Quicktime, Windows Media, and Flash Video (FLV) formats. The FLV files are encoded with keyframes at frequent intervals, necessary for PHP streaming and linking to particular points within the video. Captions and audio description are added during production by third party vendors, but since our videos are initially scripted we save on the cost of caption production. Videos prior to 2008 were open captioned, and caption data was no longer available for retrofitting the original pre-captioned versions of our videos with closed captions. For these videos, we used Automatic Sync Technologies, a company whose CaptionSync service enabled us to quickly (automatically, using speech recognition) convert our existing scripts to closed captions in all the required caption formats. For more recent videos, our videos are closed captioned during production, and are delivered in the required caption formats including Quicktime Text (Quicktime), SAMI (Windows Media), and Timed Text, the W3C standard format supported by JW FLV Player.
  4. Inject the FLV file with meta data using Captionate. This step is described in more detail in Question 9, above.
  5. Prepare for Video Search. Our final step is to save all materials to our web and streaming media servers, then tell the Video Search application where to find these materials by completing a simple behind-the-scenes web form developed for DO-IT staff. This information, including each caption and its start and stop times (extracted from the caption files) is stored to a MySQL database, and is immediately ready for searching.

Is the DO-IT Video Search source code available?

The PHP source code is not presently available since it is highly specialized for the unique characteristics of DO-IT's video collection. With more work (and funding), we may consider making it available in the future. Meanwhile, the trickier parts involve using JW FLV Player and PhilFlash extension, both of which are well-documented elsewhere. Feel free to consult our HTML and Javascript files using your web browser to see how we've implemented these resources.

What is on your To-Do List?

We're currently working on the following tasks:

Who do I contact with feedback or suggestions?

We welcome and encourage your feedback, especially on accessibility and usability of DO-IT Video Search. Please send feedback and suggestions to Terry Thompson, Technology Accessibility Specialist at DO-IT.

Who do I contact for support in using JW FLV PLayer or any of the other products mentioned in this FAQ?

Although we have used a variety of software tools to create the DO-IT Video Search application, we are not the best source for obtaining support on these tools.

For support related to JW FLV Media Player, the user community is very active and supportive. Consult the JW Media Player Support Forum. This forum may also be the best source of support for third-party utilities that are used in conjunction with JW FLV Media Player, including the PHP Streaming Extension and many others.

For support with any of the other products mentioned in this FAQ, consult the product websites for user forums or contact information.

Are there other examples of video search applications?

Most of the Web's major video portals, including YouTube.com, video.aol.com, and others, provide the ability to search video, but base that search on titles, descriptions, and/or key words rather than full text. A few companies, including Docsoft and EveryZing.com, are attempting to harness the growing power of speech recognition technology to automatically transcribe and caption video. However, a close look at the transcripts produced by EveryZing.com will reveal the current limitations of this technology.

Also, Automatic Sync Technologies (AST) has developed a video search system that is available for purchase. AST specializes in automatic generation of closed captions from an existing transcript, so captions naturally provide the searchable data for AST's system. More information is available on AST's Streaming Media Video Search page.

The most high profile example of search that uses captions is Google Video, which encourages users to add captions to their videos in Google's own closed caption format, and for videos that are captioned Google supports direct linking within the video, just as DO-IT Video Search does, albeit without full attention to accessibility of the interface. For additional information, including a link to an example showing within-video linking, see Closed Captioning Search Options on the Official Google Video Blog.

What are some other resources on multimedia accessibility?

For additional information about captioning, audio description, or other issues related to multimedia, consult the DO-IT Knowledge Base. A good entry point is the AccessIT article How do I make multimedia accessible?.