MIME Types #

MIME types (also known as Internet Media Types) are a way of defining file formats so that your system knows how to handle them.

# AddType TYPE/SUBTYPE EXTENSION
AddType audio/mpeg mp3
AddType audio/mp4 m4a
AddType audio/ogg ogg
AddType audio/ogg oga
AddType audio/webm webma
AddType audio/wav wav

<audio>
   <source src="elvis.mp3" type='audio/mpeg; codecs="mp3"'>
   <source src="elvis.oga" type='audio/ogg; codecs="vorbis"'>
   <!-- add your fallback solution here -->
</audio>

Knowing in Advance: canPlayType Can Help, Probably #

Fortunately the audio API provides us with a way to find out whether a certain format is supported by the browser. But first, here’s a quick recap on how we manipulate the audio element via the API.

var audio = document.getElementByTagName('audio')[index];

// or, if you gave it an id attribute
var audio = document.getElementById('my-audio-id');

var audio = new Audio();

Once you have your audio element, you’re ready to access its methods and properties. To test format support, you can use the canPlayType method, which takes a MIME type as a parameter:

audio.canPlayType('audio/ogg');

You can even explicitly include the codec:

audio.canPlayType('audio/ogg; codecs="vorbis"');

canPlayType returns one of three values:

1. probably,
2. maybe, or
3. “” (the empty string).

The reason we have these odd return types is because of the general weirdness surrounding codecs. The browser can only guess at whether a certain codec is playable without actually trying to play it.

So to test for support, you could do this:

var audio = new Audio();
  var canPlayOgg = !!audio.canPlayType && audio.canPlayType('audio/ogg; codecs="vorbis"') != "";

All we’re doing here is checking that canPlayType is supported (!! effectively casts to a boolean) and then checking that canPlayType of our chosen format doesn’t return an empty string.

Current Browser Codec Support #

Let’s check the codec support in the current crop of modern browsers.

† WAV since Chrome 9

The good news is that at the time of writing, it’s estimated that around 80% of browsers now support HTML5 audio.

The bad news is that there is still no consensus on which codec to support, so you’ll need to provide both MP3 and Ogg Vorbis sources in order to take full advantage of HTML5 audio.

var audio = new Audio();
var duration = audio.duration;

Buffering, Seeking, and Time Ranges #

The situation is improving in this area, as browser makers start to implement key parts of the spec.

The API provides attributes called buffered and seekable that can be used in situations where we want to ascertain which part of the media has been buffered, preloaded, or is ready to be played without delay.

Let’s first take a look at the buffered and seekable attributes. Both return a TimeRanges object. The TimeRanges object is a list of time periods containing start and end times that can be referenced by their indexes.

// returns TimeRanges object of buffered media
var buffered = audio.buffered;

// returns time in seconds of the last buffered TimeRange
var bufferedEnd = audio.buffered.end();

------------------------------------------------------
|=============|                    |===========|     |
------------------------------------------------------
0             5                    15          19    21

// Is the player currently seeking?
var isSeeking = audio.seeking;

// Is the media seekable?
var isSeekable = audio.seekable && audio.seekable.length > 0;

// Time in seconds within which the media is seekable.
var seekableEnd = audio.seekable.end();

Well-Played #

It’s worth mentioning the played property in passing. This property tells us which time ranges have been played within the media. For example:

// returns a TimeRanges object
var played = audio.played;

Media Events #

Underpinning both the native audio and video APIs is a comprehensive set of events. A full list can be found at the WhatWG version of the spec.

Attaching handlers to media events allows you to easily react to changes in state. For example, it might be useful to update the time info in a custom-built player each time the timeupdate event occurs.

A quick summary of the more commonly used media events:

Other useful events:

Again, the HTML5 Media Event Inspector is your friend.

Additionally, you can test browser support by using areweplayingyet.org. It’s worth checking out the code behind the tests to understand the goings-on under the hood.

Streaming #

Streaming audio is a common requirement. This is an area that until recently has been dominated by Adobe’s Flash technology. Proprietary server technologies and protocols are well-established. One of the leading examples of this is Adobe’s Real Time Messaging Protocol (RTMP). However, current browsers only support streaming over HTTP, and so something like Flash is required to process RTMP streams.

Currently, audio-enabled browsers only support SHOUTcast and Icecast servers that stream audio over HTTP. SHOUTcast is propriety and allows streaming of MP3 and AAC files only, while Icecast is non-propriety and supports Ogg Vorbis as well as MP3 and AAC.

An Evolving Spec (Or, “Whoa, this thing is moving!”) #

As the audio spec is still evolving, there are a couple of inconsistencies in browser implementations to watch out for. These generally affect older browsers.

var audio = new Audio();
audio.setAttribute("src","mynew.mp3");
audio.load(); // required for 'older' browsers

What’s New? #

There are a few new features and whole new APIs being specified for web-based audio, so let’s take a look at what’s around the corner.

Advanced Audio APIs: The Future Sound of Browsers #

When it comes to more advanced audio functionality, Mozilla was first to enter the fray with an early experimental Audio Data API for Firefox. This was intended to facilitate audio manipulation via JavaScript and created a platform for low-level tinkering.

Not to be outdone, Google released the Web Audio API for Chrome and put it forward as a W3C proposal. The Web Audio API is a higher-level implementation that takes an audio-node based approach and provides many useful functions.

Both implementations address the desire by developers to do more with audio — to create, manipulate, analyse, and mix audio on the fly. In effect, we may one day be able to do in a browser anything we can currently do with native applications.

So what if we want to play with advanced audio in the browser? Well, two separate implementations currently exist. If you want to write applications that use both APIs in any reasonable way, you need third-party bridging libraries to abstract away the differences.

Also fairly new on the scene is Mozilla's MediaStream Processing API, which takes another approach to audio and video streams to allow us to manipulate at both high and low levels.

In fact, using advanced APIs and taking advantage of the speed of modern JavaScript engines, we can even write decoders for unsupported codecs. Currently, we have the JSMad MP3 decoder and the lossless Alac.js decoder. Apparently, FLAC and AAC are in the pipeline. There is hope, then, that in the future, we may not have to worry about which browsers are supporting which formats.

For those curious about advanced web audio, I wrote about the differences in approaches and the different libraries that allow us to write cross-browser solutions in HTML5 Audio APIs: How Low can we Go?

The good news is that all relevant parties are talking, and things seem to be buzzing on the W3C Audio Working Group. It looks like everyone is coming together to put their ideas into a unified Audio Processing API, which is rapidly approaching publication as a Web Standard.

Summary #

Although browser implementations of the current HTML5 audio spec are improving, there are still a few issues to watch out for when creating comprehensive cross-browser solutions. You certainly need to be aware of browser limitations on platforms like iOS. Hopefully, as support matures, these issues will disappear.

Positive news on the "advanced audio" front too: a new standard is being established, and we're close to get a unified spec for browser makers to work from. In the meantime, JavaScript libraries are available to help bridge the gaps between existing implementations.

There are also signs that we may be seeing the end of the requirement for different browser-dependent codecs. Opera Mobile already supports MP3 where the underlying OS supports it, and Mozilla look like they may enable this too. If this option exists for mobile browsers, it's not a huge stretch to imagine that desktop browsers will follow suit.

So lots happening and lots already achieved. The rough edges of existing browser implementations are being smoothed, consensus and standards are being forged, and we can see the foundations of some very exciting new technologies being established.

The future of web-based audio looks bright!

Further Reading #

• "Probably, Maybe, No": The State of HTML5 Audio
• The State of HTML5 Audio
• Mastering the HTML5 <audio> tag
• Unlocking the power of HTML5 <audio>
• Everything you need to know about HTML5 video and audio

getUserMedia #

getUserMedia is an API that gives a web page access to a user’s camera and microphone via JavaScript. It’s supported in Opera Labs (latest Android build, latest desktop builds) and WebKit in Chrome Canary builds (instructions).

Like many other APIs, it’s not part of the “real” HTML5 spec. It started life as the HTML5 <device> element, then got moved into the W3C as part of the webRTC specifications. But let’s not taxonomise when we could be having fun.

The first thing we need to do when using super-cool new UIs is detect whether it’s supported:

 if (navigator.getUserMedia) {
  // do something cool
} else {
  // fallback code
}

For this article, our “do something cool” will be to grab the dominant colour of the current input stream from the camera. As a fallback, we have several options.

One option, if we’re using the default Android browser, is to show a form that automatically starts the camera when focussed:

<form enctype="multipart/form-data" method="post">
  <input type="file" accept="video/*;capture=camera" />
</form>

There’s also the capture attribute, a proposed extension from the W3C Device API Media Capture API that tells a user agent where to get the input data. Android extends the accept attribute. See more in David Calhoun’s Device API article. Alternative capture arguments are camcorder (for video), microphone, and filesystem.

Browsers that don’t understand the still-draft Media Capture API will simply ignore the capture attrbute and prompt the user to browse to a file on their file system for uploading.

Users on iOS are out of luck, as their devices don’t allow access to the file system.

getUserMedia API #

Let’s assume (because we’re optimists) that your user has a getUserMedia-enabled device. The API is nice and simple, as all Web APIs should be.

navigator.getUserMedia accepts two required arguments and an optional third.

The first argument tells the device which media you require access to, and it’s passed as a JavaScript object. So, if you only require access to the microphone, the first argument would be {audio: true}; for video-chatting, you would use {audio: true, video: true}.

The device decides which camera to use: User agents are encouraged to default to using the user’s primary or system default camera and/or microphone (as appropriate) to generate the media stream.

A previous version of the specification allowed hints to user agents about which camera to use. The API could specify “user” (the front camera on a phone) or “environment” (the rear camera on a phone). Debate continues about whether to reinstate this (read the ongoing conversation). One argument against is that letting sites know what cameras a device has could help Dr Malice of Evil Corp. fingerprint users after luring them to his site (which I find to be overly-cautious). Harald Alvestrand poses a conundrum: conferencing units may have a room camera, a document camera and a lectern camera, neither of which is attached to the physical box; which one of these is front?

But let’s not worry at the moment. Devices with more than one camera generally have a mechanism that allow the user to choose which one is used, so this gives the user control. Note also that the spec says User agents may allow users to use any media source, including pre-recorded media files.

The second argument is the success callback, the function to be executed on success, assuming that the user allows access and the device supports your request.

There is an optional third argument. This is the failure callback, the function to be executed if something went wrong. It’s optional, but only optional in the sense that washing your hands before you eat is optional: if you don’t do it, you leave yourself open to all sorts of bugs and your mum will shout at you.

Of course, it’s important to do feature detection before making an API call to ensure that the browser and device are capable, but even if feature detection succeeds, there is still much that can cause failure. For example, the user could deny permission or turn off the camera via a hardware switch, or the device itself could disable the camera.

So here’s how we recommend you use the API:

navigator.getUserMedia({audio: true, video: true}, success, error);

function success(stream) { 
  // ... use 'stream' ... 
} 

function error(){ 
  // display fallback content 
}

To check whether it’s working correctly, we can attach the output of the camera’s stream to the input of an HTML5 video on the page, like this:

<video autoplay></video>
<script>
if (navigator.getUserMedia) {
  navigator.getUserMedia({audio: true, video: true}, success, error);
  function success(stream)
  { 
    // ... use 'stream' ...
    var video = document.getElementsByTagName('video')[0];
    video.src = stream; 
  }
  // ...
</script>

video = document.getElementById('video');

How to Make and Link to a WebVTT file #

All you need to make a WebVTT file is a simple text editor. Type WEBVTT as the first line of the file and save it as a .vtt file. In the future, we expect existing captioning tools such as Universal Subtitles to export to WebVTT format.

WEBVTT

That’s all you need to get started. Next, we have to link to the file in the HTML document. We do this via the <track> element, which is a child of the video element. The <track> element has several optional attributes:

• the source WebVTT file (src),
• the language of the track (srclang),
• a user-readable label, and
• what kind of track it is. The values of the kind attribute come from the list above (i.e., subtitles, captions, etc.).

In the following example, we’re using a <track> element for subtitles:

<video width="640" height="480" controls>
  <source src="video.mp4" type="video/mp4" />
  <source src="video.webm" type="video/webm" />
  <track src="subtitles.vtt" kind="subtitles" srclang="en" label="English" />
  <!-- fallback for rubbish browsers -->
</video>

A few notes about the attributes:

• If no kind is specified, the default is subtitles.
• If the kind is subtitles, then srclang is required.
• There should not be two tracks of the same kind with the same label.

In the above example, we use a <video> element with two different <src> elements (for cross-browser loveliness). After the sources comes the <track> element. You can have several <track> elements as you might have subtitles, captions, and descriptions all in different languages.

<track> doesn’t presuppose a particular file format. Microsoft supports the TTML format, but this format will not be implemented by other browser vendors.

WebVTT Contents #

We now know how to make a WebVTT file and how to reference it in an HTML document, but what goes inside it? Within the file, we list what are known as “cues”. The WebVTT file might only have one cue, but it can contain as many as you want. Each cue starts with an ID, followed by the time settings, followed by the text. Each cue is separated by a blank line. Here’s an example of captions:

WEBVTT

1
00:00:01.000 --> 00:00:10.000
This is the first line of text, displaying from 1-10 seconds

2
00:00:15.000 --> 00:00:20.000
And the second line of text
separated over two lines

The above example has two cues. Times must be written in hh:mm:ss.mmm format, so the timings in this example occur in the first twenty seconds. The second cue will split the text over two lines automatically.

If you have a segment of text that needs to appear in a karaoke/paint-on caption style, then you can place timers inline with text:

1
00:00:01.000 --> 00:00:10.000
Never gonna give you up <00:00:01.000> Never gonna let you down <00:00:05.000> Never gonna run around and desert you

Styling Options #

The previous examples specify the minimum configuration you need for subtitling and captioning, but you can style your captions too. Let’s start with the cue settings, which are done on the same line as the time settings:

D:vertical / D:vertical-lr

Display the text vertically rather than horizontally. This also specifies whether the text grows to the left (vertical) or to the right (vertical-lr).

L:X / L:X%

Either a number or a percentage. If a percentage, then it is the position from the top of the frame. If a number, this represents what line number it will be.

T:X%

The position of the text horizontally on the video. T:100% would place the text on the right side of the video.

A:start / A:middle / A:end

The alignment of the text within its box – start is left-aligned, middle is centre-aligned, and end is right-aligned.

S:X%

The width of the text box as a percentage of the video width.

To make use of these settings, put them alongside the time settings, like this:

00:00:01.000 --> 00:00:10.000 A:middle T:50%
00:00:01.000 --> 00:00:10.000 A:end D:vertical
00:00:01.000 --> 00:00:10.000 A:start T:100% L:0%

which would result in something like the following:

Along with the above cue settings, you can use inline styles for text:

Bold text

<b>Lorem ipsum</b>

Italic text

<i>dolor sit amet</i>

Underlined text

<u>consectetuer adipiscing</u>

Ruby text

<ruby>見<rt>み</rt></ruby>

You can even apply a CSS class to a section of text using <c.myClass>Lorem ipsum</c>, giving us many more styling options.

Finally, you can add a declaration representing the name of the voice: <v Tom>Hello world</v>. This declaration accomplishes three things:

1. The caption will display the voice (Tom) in addition to the caption text.
2. The name of the voice can be read by a screenreader, possibly event using a different voice for male or female names.
3. It offers a hook for styling so that, for example, all captions for Tom could be in blue.

<track src="chapters.vtt" kind="chapters" srclang="en" />

WEBVTT

Chapter 1
00:00:01.000 --> 00:00:10.000>
Introduction to HTML5

Browser Support #

One slight glitch with WebVTT: not a single browser currently supports it. All major browsers have started working on implementations, so we should see some results soon. Thankfully, in the meantime, there are several JavaScript polyfills available:

• js_videosub
• Playr
• MediaElementJS
• LeanBack player (and upcoming new version)
• Captionator 

Demo #

We’ve put together a quick demo which uses the Playr polyfill. We started using MediaElementJS, but it doesn’t sport as many features as Playr, such as separate lines of text and CSS classes. In the demo, the subtitles start at 2 seconds and 15 seconds and use bold, underline, and custom styles. Here’s the associated WebVTT file.

Conclusion #

This article covers the basics of creating a WebVTT file suitable for subtitles or captions for a video. We know how to add cues and chapters and how to add styles and change how the text appears on the video. Although no browser yet supports it, there’s a lot more to come for accessible video, so stay tuned to the W3C Web Media Text Tracks Community Group.

What are your thoughts on WebVTT? Are any of you using it now? How can it be improved?

Finally, let’s thank @silviapfeiffer for taking the time to answer some questions about WebVTT and for her tremendous work in this field.

Reading #

• Follow @silviapfeiffer
• W3C Web Media Text Tracks Community Group
• Recent developments around WebVTT
• Presentation: HTML5 video accessibility and the WebVTT file format
• HTML5 video accessibility and the WebVTT file format – Audio Described
• A review with notes and thoughts for LeanBack Player
• WebVTT validator
• WebVTT and Video Subtitles
• The Open Video Alliance
• Understanding WebVTT file format (draft)
• Creating subtitles and audio descriptions with HTML5 video