You need at least:
1. Kernel driver for hardware init, power management, mode setting, GPU buffer management and command submission
2. Userland library for GPU buffer management and command submission
3. OpenGL implementation
In the open source graphics stack, the kernel driver exposes KMS and DRM interfaces and potentially others. Parts 2 and 3 are part of libdrm and Mesa respectively. The display server can (I think) be built on top of KMS, libdrm and OpenGL and be independent of the hardware. However it will need an extension to OpenGL called EGL which will be specific to each display server protocol.
Currently X doesn't usually work that way for historical reasons - it used 2D acceleration first and still supports hardware that has only 2D acceleration. So it has hardware-specific drivers for each family of GPUs. However there is the 'Glamor' library that supports 2D acceleration genericallly on top of OpenGL, and I would expect to see a gradual move to that, not least because it's the only option for 2D acceleration in XWayland.
Getting back to Nvidia, their problem currently is that they don't implement the same interfaces as the open source stack and therefore don't work with the new display servers that depend on those interfaces. Implementing KMS gets them a long way there. However it sounds like they still need to reimplement the EGL, not because it's hardware-specific but because their OpenGL implementation is entirely independent of Mesa.