Crawlers are a KLUDGE (Score 1)

Webmasters currently have very little control over the indexing thus indexers have little intelligence. What is needed is a whole new protocol.

Raw indexing of HTML leads to raw results that are often of no use whatsoever. What the indexers need is a way to query a 'site' for pages that should be indexed, how often to index them, what the general topic areas of the pages are etc. also needed are html tags that indicate the 'content area' of a page (so that navigational header and footer crap can be ignored) and a means to apply relative weight to areas of the page.