Having to deal with multiple video formats means either increased storage requirements or processor requirements. I believe the reason for trying to standardise the supported video formats to a limited selection, is the same one for limiting the number of image formats officially supported by web pages: ensuring the content is viewable everywhere. If the specification said do what you want, we would see half a dozen different formats, browser supporting some of them and the users being caught in the cross-fire.
The day an Ogg endcoder/decoder is made available for things like Adobe Premiere, Final Cut, Quicktime and Windows Media Player, using a BSD style license and also focus on quality for a given bit rate, then we aren't going to see widespread adoption.
While Ogg might be fine, it is not packaged as a solution suitable for commercial products. At the same time the MP4/H264 licensing means it is not suitable for open source. We have clash of cultures and each is wanting to stand in their ivory tower, and not come down to Earth.