No. The nVidia drivers share around 90% of their code between all platforms (Windows, Linux, FreeBSD, Solaris) and the open source ones all use the Gallium framework, which is designed for portability from the ground up.
Modern GPU drivers require a set of services from the kernel, mostly related to memory management. They need to be able to get access to the device's I/O range in the physical address map and they need the kernel to grant access to texture memory in both main memory and the device. That's about all that they need from the kernel.
At the top, they need a state tracker that manages 3D API state (which is fairly minimal on modern APIs, as they aim to be stateless for performance reasons) and that translates the shader programs into some intermediate representation.
The majority of the device-specific driver code lives between these two layers, which are usually handled by abstraction layers so that they can be plugged into different APIs. You use the same Gallium driver with an OpenGL 2, OpenGL 3, OpenVG or Direct3D state tracker.