Which constraints of the web do see as being prohibitive specifically? I’ve been reading a lot lately on how the costs of storage, bandwidth and processing are predicted to exponentially decline in the near future. Do you think these factors affect the evolution of such standards? Also, what about a cross-media format which might have a rich life of use “offline” that could eventually be ported to the web in the future…

I think this is a bit too ambitious for right now. Describing such multi-layers in a simple manner has been tried in SMIL and MPEG-7 and I truly think we are not yet ready for it on the Web. We are moving only slightly in that direction with media fragments and the idea of addressing spatial objects.

I raise this question because I truly believe there are creative, knowledge and financial motivations to see this sort of “object-oriented” approach to the elements within the frame be included in a video’s metadata. Video is needlessly and artificially flattened by modeling it on celluloid.

Here is the link to the QuickTime “Track Header” description:
http://developer.apple.com/mac/library/documentation/QuickTime/QTFF/QTFFChap2/qtff2.html#//apple_ref/doc/uid/TP40000939-CH204-BBCEIDFA

It contains the following fields (amongst others):

Alternate group – specifies a collection of movie tracks that contain alternate data for one another, choices may be based on e.g. playback quality, language, or the capabilities of the computer.

Layer – describes the priority in which tracks overlay each other

Track width/height – pixel size of this track

Volume – loudness of this track

Further, here is the QuickTime “Media Header” description – “Media Atoms” define a track’s movie data:
http://developer.apple.com/mac/library/documentation/QuickTime/QTFF/QTFFChap2/qtff2.html#//apple_ref/doc/uid/TP40000939-CH204-BBCHCFID

It contains the following fields (amongst others), which I understand and used to chose tracks:

Language – the language code for this media

Quality – the media’s playback quality

I am not sure how this information is actually used in practice, but there is quite a bit of information that could be exposed through a JavaScript API if hooked up with the browser. Similar is necessary for Ogg (though we should make it simpler).

So, let me explain the idea for how this information could be used with the HTML5 video element:

Assuming we have a multi-track video linked in the src attribute of a video element, the browser would need to identify what is in the tracks (along the lines described in the blog entry) and then do something useful with it.

By default, it would use the main audio and video track of the default language that the browser is set to and play them back.

For blind people (which would need to be a setting in the browser) it would additionally activate the audio description track in the default language. It would also probably disable all video tracks.

For deaf people (again a browser setting) it would additionally activate the caption or subtitle track in the default language and the sign language video track in the default sign language. It would also probably disable all audio tracks.

Additionally, the browser would provide a right-click menu that lets you activate/deactivate all tracks individually. If the controls attribute is set in the video element, this menu is also added to the controls bar.

Additionally, there would be a JavaScript API through which the Web developer can identify the available tracks and turn them on/off selectively.

It is possible that to make this available in a container-format independent way, an external description format such as ROE is necessary.

As long as the spec allows for content negotiation with ROE descriptors, I think we’re in business.

- such a name, being a list of attributes, might be better off as actual attributes (eg, in Skeleton for Ogg), as it will both avoid having to parse a string into its constituent attributes, and make it easier to add new attributes should the need arise.
- Sending attribute preferences in the original request from the client (either via HTTP headers or URI parameters) will allow the server to select tracks and stream them to the client, needing only one roundtrip, as opposed to two if the client first requests a list of tracks, then requests a subet of these tracks, after having parsed them and worked out which ones it’s interested about). This, if keeping names like proposed, also moves parsing complexity from client to server, a good thing I think.
- names might still be interesting as unique identifiers, for those clients that know exactly what they want (eg, editors), but do not have to carry the semantics.
- Track numbers are easy to break by remuxing a video (eg, simple edition, or even adding a subtitles track).

A deaf person may argue the same for any audio track of a video, and a blind person for any video track – they also don’t want to waste bandwidth on something that is not necessary to them. This naming scheme is a first step in the direction of enabling content negotiation on muti-track media files through the media fragment URI scheme which allows addressing of tracks.

Take sign language — I have perfectly good hearing, so I don’t want to waste bandwidth downloading it. On the other hand, somebody without hearing won’t want to download the audio track.