you'd have a vast library of libraries. Something like CPAN or something you'd get in the C world. Libraries written to perform some task and nothing more. Then documented with care and the API published.
Anyone wants to do something, they take the library that appeals to them and adds it to their program and build up a program from these bits.
Now the problem today is that a) some only use libs that come with the OS or language framework, b) the libraries that are out there are shit, written quickly and for a bit of a mishmash of scenarios.
For example, you can get an XML parser and it will work perfectly. It will only parse XML though, but then, that's what only what you want from a XML parser library!
So the problem is not so much that we have libraries, but that the libraries we have are not good enough as library code.