Well the exponential growth has been going on since the very first computer systems where made way back before Moore's law even existed, when they where still using vacuum tubes.
Arguably it goes back even further if you consider it to be systems for information processing rather than strictly computer systems. Things like writing, transcribing books, the printing press, mathematics and so on.
The growth will probably stop eventually but it's unlikely to occur before any kind of singularity happens, even if it does the world will be drastically different. The planned 2012 IBM supercomputer should have about enough processing power to emulate a human brain (its not doing that but they have the blue brain project underway), By 2025 a $1000 computer should have that power (consider what the super computers of that time will have). Unless you think Moore's law is going to kick us all in the nuts in the next 15 years we should be well on our way. Traditional Moore's law (as it applies to transistors on silicon) should continue till some time around 2030 (although some earlier limits are as low as 2020 and it might slow down things leading up to the point). This doesn't take into account the dozens of other non-traditional technologies under research that aren't Moore's law relates: memsistors, photonic computing, DNA/quantum computing (only useful for some specific computation but AI might apply), 3D-ICs, carbon nanotubes, graphite, spintronics.
After some kind of singularity (assuming we survive) we have no idea what the limits are, can we make new sub universes where the laws are better optimized for computing? or change the laws in some specific area? Can we use the theory of relativity to speed computation up (ie I leave a computer on the planet and travel at close to the speed of light in a circle until it finishes number crunching, or hopefully some similar system on a chip)? Can we find some ultimate universal loophole for infinite energy/computation? A cpu that works in an infinite number of parallel universes? Maybe we will hit the universal wall, but by that point it won't matter so much.