Its possible, and No, it wouldn't be huge but... (Score 2)

It wouldn't be to hard to make a cross OS driver developement library. The drivers them selves would not be that much bigger either, but they probably would end up being inefficient. Each OS has certian parts you can accelerate in different ways, and there's little tricks you can do specific to your card for a certain OS to get it to run faster, these would be lost in a generic cross OS driver developement kit.