gsnedders

Just over a month ago, there were all kinds of accusations going around about Apple, Microsoft, and Nokia paying to get Ogg/Theora/Vorbis removed from the HTML 5 draft. However, as someone within both the W3C HTML WG and the WHATWG I find these accusations wholly unbelievable.

Unquestionably the first issue I'll have to clear up is why they would be removed otherwise: without the support of Apple, Microsoft, and Nokia (the former two who make up a large percentage of the web browser marketshare) any mandated format for the media elements (audio and video) is pointless. The entire point of mandating a format is to have a common format that can relied upon (and, as such, it really needs to be a MUST not a SHOULD) — if major companies consider Theora a large financial risk they simply will not implement the specification, and what is a specification worth if it does not create interoperability? Without interoperability, there is no difference from the specification not existing at all.

But why is Theora a large financial risk, and why only for major companies?

• Video codecs are known to be a hotspot for submarine patents (and no matter how complete a patent search you do, you can never be certain there are not unpublished patents covering a format).
• The financial risk is huge — when submarine patents do arise the fines tend to be in the hundreds of millions (if not billions) of USDs.
• Anybody who holds a submarine patent will wait a while before pouncing, and even then only on to a major company (for a small company won't have shipped enough products in breach of the patent and have enough money to be worthwhile suing).

"But I read that Theora was patent-free! Surely this is all irrelevant‽" has been asked all too many times — it is known that Theora isn't patent-free (though the known ones held by On2 have got an irrevocable royalty-free license), and it is not known if there are any submarine patents (by their very definition (i.e., patents that aren't published for a long time after they are applied for) they cannot be known about).

If video codecs are patent risks, how come companies continue to add new support for codecs such as H.264? The answer is a simple one: they will add support for new codecs when the new codec is technically superior to what is already supported or there is a significant body of already deployed content that requires support for the new codec; though many people have tried to spin this issue by omitting is technically superior to make an argument that they can't under their argument add any new codecs ever (as the last clause is mainly for legacy codecs, and as this post later explains it is highly unlikely to be used to add any new codecs nowadays).

Many have argued that Wikimedia's use of Theora is a significant body of content, however, compared with the number of videos on the web (undoubtedly predominantly pornography) this is wholly insignificant (it is extremely unlikely to be as large as even 0.001% of video on the web).

So, to get support for a codec such as Theora we need people to distribute data in Theora. There are major issues with this: most people will only use codecs that have encoders that are bundled with software, and that are widely supported. We don't get encoders for codecs unless they are technically superior to what is currently available or are widely supported by decoders; and we don't get decoders for codecs unless they are technically superior to what is currently available or there is a significant body of already deployed content that requires support for the new codecs (which doesn't exist until we get support in the encoders which we don't get until we get support in the decoders, ad infinitum). Realistically, this means the only way to get support for a new codec (and not one that is already widely deployed) is by it being technically superior to what is already used.

What can we use as a base video codec, though? This is a very hard question, and not something that can easily be answered. It is, however, clear that Theora is not an option (nor is any MPEG standard, as these all have known non-royalty-free patents). Arguing that Theora should be put back in is pointless — it's not what we want in a specification. We want interoperability. There are a large number of people both within the W3C and outside of it looking for a suitable codec. To quote what the specification currently lists as requirements:

• We need a codec that is known to not require per-unit or per-distributor licensing,
• That is compatible with the open source development model,
• That is of sufficient quality as to be usable,
• And that is not an additional submarine patent risk for large companies.

Any suggestions that meet all four of these requirements would be warmly welcomed by the HTML WG.

However, I have kept Ogg/Vorbis out of the entire body of this post. Neither of these is an issue. Both already have major companies shipping them in bulk (Microsoft, Epic, and Rockstar all have used Ogg/Vorbis in major products). That said, it cannot be said for certain that these can be used with the video codec (some video codecs require certain containers or audio codecs). Why remove these, though? It makes little sense to define what MUST be used for some of the media elements but not for others.