Hmm… there seems to be a lot going under NB. The director’s comments that Paul suggested would be displayed on screen, whereas these more metadata like comments would probably only be machine-readable. It may well be better to create another type for this. How about META?

The codec types are what a user in e.g. a Web browser would choose to be displayed, if it is so available and if it understand the format of content type, i.e. the format of the description.

For example, DFXP, smilText and realText can be used to describe any of these. But one instance document should only represent one text codec type. I.e. one instance DFXP document should be either a caption or a tickertext or a karaoke file.

When encapsulating this into Ogg, you want to know which it is. You want to know this at the level of encapsulation because the server might need to do content adaptation and throw away some of the tracks. Also, it is easier for the client to know this in a quick way and at a defined location in the bitstream, such that it can present to its user the ones that the user prefers.

So, it doesn’t matter if you have a format like DFXP which is able to specify multiple of these types, or a format like srt which is only for subtitles (and maybe captions). The author will specify which format it is and which codec type as it gets multiplexed into Ogg and you’re sorted.

As for your comparison with video codec types: you are right that video codec types, in particular the “codecs” parameter in a MIME type, has a very different meaning. I may have chosen a bad name to describe what I mean. If anyone has a better suggestion for what to call them – I think one person suggested codec category – let me know.

BTW: the actual MIME type of the text codec is also encapsulated into OGG, so you don’t need to worry about that missing.

There doesn’t seem to be a straight mapping between your codec types and encodings (neither 1-to-many nor many-to-1), for example, DFXP or smilText or realText could be used for any of subtitles, captions, karaoke and tickertext. And probably some of the others as well. So, only the document author would be able to set the codec type, based on the content of the track.

This make the use of these codec types completely different (from a client application point of view) from other codec types like for video. For the latter, the codec type signals (to the app) whether it’ll be able to decode and display the track, nothing about its semantics. For these text codecs it’s exactly the reverse: the client app only gets semantic info from the codec type, and cannot determine whether it’ll be able to decode the stream…

I have just read it up on http://en.wikipedia.org/wiki/Fansub and apparently you are right. Strangely enough, this relates to “closed” and “open” captions – open captions are hardcoded into the imagery and cannot be removed, while closed captions are in a separate data stream that can be turned off.

Thanks for noticing.

DVD overlay subtitles are I guess a corner-case, since they’re pre-rendered but those prerenders are triggered by timing values. But I’m pretty sure they’re hard subs because they’re not in a textual format.

On the other hand, I believe the MKV format has a way of including a soft sub stream in the actual container (rather than as an external file), but I don’t know how standard that is. I think mplayer supports it…

Anyway, as I understand it, what you’re talking about here are all soft subs. Hard subs are handled by the encoder processing them into text at encoding time.

In the end it really depends on how it’s being presented, because there will be default displays for the different types of text annotations. For example, captions and subtitles will be at the bottom of the screen in a certain area, probably in black with a white boundary. However, we need input by designers, GUI experts and the like for what defaults make sense. It’s not a trivial exercise overall.

The other type of text annotation I’ve seen recently is a director’s commentary or behind-the-scenes information. In “Who Framed Roger Rabbit” they have a ‘Toontown Confidential’ version of the playback where text gets superimposed on the movie at given points to describe extra information about the scene – such as the number of different effects that a particular scene used or information about particular characters. I think this might fit into your ‘NB’ category but it’s not strictly metadata about the movie _file_ – I think it fits more into a ‘movie information’ (e.g. ‘NFO’) or ‘trivia’ (‘TRV’) tag.

Hope this helps,

Paul