I agree about crawling. I was talking about building the index. Google does that better than anyone (by a long short) and still doesn't get it right. Because they are one "small" company in the large scheme of things. Look at WolframAlpha and Freebase. There are so many different ways of dealing with search and Google needs to stop believing it can figure them all out on its own. Developers should be allowed to create local branches of the index and work on them. Google should try letting go of some of the control, to allow innovation, else innovation is constrained by the number of people they can afford to throw at the problem. And most of the people they throw at the problem spend most of their time keeping things from falling apart. Right now developers aren't allowed to "scrap" the index. This doesn't make sense in terms of allowing innovation in search. My assumption is that, a developer (with experience in data analytics) in a small town in Alaska, if given access to the (realtime) index of data Google has collected on her town, can extract value relevant to that town, better than anything Google can achieve.

There aren't too many organizations public or private left in the world, that can replicate Google's indexing capabilities. That index has unimaginable untapped value and it needs to be opened up (unlimited api access). It's value can never be tapped by a couple hundred engineers sitting inside Google. Giving a small group within Google complete access to it, is equivalent to only giving a couple scientists access to human DNA data. I hope they find a way of opening it up before the regulators do. If there is anything worth building a platform around. more than a Google+ or an Android or TV or googledocs or chrome or youtube it is that index. I don't know how it would work but they need to start thinking about it as an end goal.