I wouldn't concentrate too much on what search engine software to use, since it didn't really sound like you'd built out much of a requirements list of what you wanted it do. Think about how you wanted it to function and then look for the tool that matches accordingly. I would even go so far as to create a search page, and a results page. A good place to start would be
searchtools.com.
As for building out a Topical or Hierarchical structure like Yahoo's or DMOZ, you need to have meta data about the document. You can pull meta data from a URL, as was suggested earlier, but I wouldn't advise that. In order for this to work, your URL's start to look like 'http://city.gov/CityCouncil/MinutesOfMeeting/2000 /10/15/index.htm', which also makes it difficult to move things around, and makes it difficult to return the string 'CityCouncil', properly formatted when you build the hierarchy on the fly. If you include the hierarchy information in the page, you can get around some of these problems. I think the standard way to do this would be using meta tags. View the source of this page for an example of this. The downside to this is that you have to structure your hierarchy in advance, which goes back to getting & building requirements early, you also have to convert old documents to include these tags, and you have to make sure that future documents will have these tags properly implemented. If you're using URL paths to qualify the documents, you may be able to take advantage of an already existing directory structure to build your heirarchy. I just don't like using document locations to figure out what a document is about--its always seemed like a brittle solution to me. On the other hand, I've used the URL string to help make a first pass at placing meta tags into documents, usually using a Perl script. I still had to go back in and check the documents though.
I've used both FreeWAIS-SF and Verity for implementing searches like this, as well as home-grown solutions. I wouldn't advise using Verity since I think that it's prohibitively expensive, unless it has some feature that you require and are willing to pay for it. And I didn't think the home-grown solutions worked as well as the off the shelf products that were customized.
