Comment Hire an Information Architect. (Score 1) 211
Seriously. This is what we get paid to do. There's far too much to communicate on a forum, and if the SNR here is typical, you'll get awful, unrelated and just plain wrong advice.
If you can't hire, see if your school has a library science program and look for a good intern.
Failing that, read the Polar Bear book (Rosenfeld & Morville, pub: O' Reilly) yourself and follow the threads to resources particular to your problem.
Tactical help: populate the kewords, title, subject properties in your PDFs and Office docs. If you populate in Office and make a PDF, the properties come along. They're in File>Properties... Filling them out will help any search engine that can consume binary docs make sense of your content.
And there are bulk scan-to-OCR packages out there. Funnel into PDF and populate the properties.
Worth saying again: populate the properties.