Doom 3 was not a failure from a technology stand-point.
It did just what it set out to do. You try writing an engine with full dynamic lighting that runs on a "Geforce 2". That's right, Doom 3 has a render pipeline
that is DX7 compatible.
If anything Doom 3's only technical foible is that it relied on OpenGL which happened to become a "big mess" around the time Doom 3 was in development.
He was lucky that OpenGL even offered shaders at that point and had to write them in ARB because GLSL was so broken it's not even funny.
Megatexture is showing up from other engine developers now too. The concept of data hierarchy wont go away simply because most mappers have gotten by better with tiled texturing. Eventually that limitation will be broken down. The only failing there is that John tried too much to eat his own dog-food. Rather than use a combination of standard tiled textures and Megatexture he tried to force everything to be compiled from the Megatexture. The hardware wasn't ready.
Doom 3 BFG's source code is a treasure trove of improved rendering concepts afforded by better OpenGL standards. If we had the OpenGL then that we
have now, it would have held-off UE3 no doubt. Some informal tests on a few hundred dynamic lights show you can have a scene with the original Doom 3 render at 15fps while BFG will render at 300fps. Easily competitive now and it's not even a deferred renderer.