This isn't really to do with BGP or IPv4 as such, it's an inherent problem in the way "The Internet" regards addresses.
You might be able to get some efficiencies in IPv6 by incorporating formerly-unrelated address allocations under a single prefix. But that doesn't solve the problem of a continuously growing network, increasingly complex (and commercially controversial) peering arrangements, the fact that IPv6 addresses are actually larger and the fact that you're going to have to support IPv4 anyway in parallel with any IPv6 transition (I don't personally believe it will ever happen, but that's a different story).
You could, however, get rather more efficiency in core routing tables if network addresses only had a very transient existence and were related to the source/destination route to be employed (eg: look up a domain name, do some route pre-computation, allocate some addressing tokens that make sense to the routers on the path, recalculate the route periodically or in response to packet loss). That's not IPv6, though. IPv6 has the same order of dependence on every router knowing about every destination network as IPv4 does (give or take the slightly greater prefixing efficiency).
TL;DR - The Internet is getting bigger. Buy more kit.