Google Has A Problem: VP8 Is Not As Good As H.264, On2’s Quality Claims Unfounded

When On2 announced their VP8 codec in September of 2008, they made some pretty big claims comparing the improved quality of VP8 over H.264. Specifically, On2 said that VP8, "delivers over 50% bandwidth savings compared to leading H.264 implementations" and stated that VP8, "surpasses the compression efficiency and performance of every other video format on the market." While the industry was quick to take note of these statements, many were also quick to question On2's claims. Now, based on test results published, it's clear that the quality of VP8 is not as good as On2 said it was and makes one wonder why Google isn't setting forth proper expectations since they now own the VP8 codec.

For more than a year, myself and others asked On2 time and again to provide the technology to the industry so that third-party evaluations comparing VP8 to H.264 could take place. While On2 had been in the codec business for a long time, they also garnered a reputation for making a lot of big performance claims with their other codecs, like VP7, that turned out not to be true. When Google decided to buy On2 last year, most people assumed that whatever On2's claims were about VP8 must be true or else why would Google be interested in the codec? While that would be a reasonable assumption to make, based on new test results, it's clear that VP8 is not better than H.264 at all.

To be fair to Google, at no time that I recall did Google ever say VP8 was 50% better than H.264. That was language that came from On2 and when Google acquired the company, they didn't make any performance claims comparing VP8 to any other codec. But the problem that Google now has with VP8 is that they also didn't refute anything On2 had been saying for the past 18 months. On2 set expectations for what VP8 could do and since Google didn't say otherwise, many are now going to be disappointed when they find out those claims aren't real.

Based on test results from two different codec experts, Jan Ozer (test results) and Jason Garrett-Glaser (test results), they both came to the conclusions that the VP8 codec provides similar quality to H.264, but in most cases, H.264 is still better quality wise than VP8. Both also stated that most won't notice the difference between VP8 and H.264, but that's not what VP8 was suppose to be about. VP8 was touted as the video codec that was suppose to replace H.264 because it could offer better quality at half the bandwidth, something both reviewers said is not possible.

Google did mention yesterday at their I/O
conference that by encoding videos for YouTube with the VP8 codec it would enable Google to save money on their bandwidth costs. But what Google didn't say is how much it would save them or even give any kind of comparison numbers highlighting the differences between H.264 and VP8 when it comes to the number of bits delivered. Considering that's suppose to be one of Google's selling points for VP8 and no details are given anywhere on their webmproject.org website regarding this subject, one has to wonder what Google is referring to.

After reading Jason Garrett-Glaser's long and very detailed review of VP8, it's clear that Google has some major problems that go a lot deeper than the subject of bandwidth savings. Jason found so many major problems with VP8 that he summed up his review by saying that, "VP8 is not ready for prime-time", "the encoder’s interface is lacking in features and buggy" and most importantly, it appears that Google has not improved upon the code they acquired from On2.

Jason says that, "the VP8 software basically is the spec–and with the spec being “final”, any bugs are now set in stone. Such bugs have already been found and Google has rejected fixes." While the specs may be final, in fairness to Google the website does say that the encoder/decoders are just preview releases and one would expect those to be improved upon over time. But if you are Google, why rush VP8 out to the market so fast when clearly it still has a lot of flaws? This is not a short-term play for Google and it's not like they rushed VP8 out the door to make money off the licensing.

Also, keep in mind that VP8 was out in the market for almost 18 months before Google acquired it and in that time, not a single content owner that I saw ever adopted VP8 from On2, even though On2 made a big stink about how many content owners were testing their VP8 codec. Now we know why we never saw any press releases in those 18 months announcing any customers for VP8.

Frankly, the way Google has handled the On2 acquisition and the release of VP8 simply has me puzzled. Google's not a bunch of dumb folks and I don't see why they would rush an inferior product out to the market, make it open-source and then hope it would unseat H.264 all while not getting any input from developers before the initial first release, even if it is beta. If Google permits changes to the spec and allows the developer community to help fix some of the glaring issues with VP8, I could see that approach. But at no time during Google's presentation yesterday did they frame the release of VP8 that way or ask for help from the development community to fix issues that they must know exist.

There are a lot of unanswered questions here and hopefully someone from Google will address these points on their blog since it's only going to be another 12 hours or so before these issues with VP8 come out of the small developer community blogs and into the major media spotlight.