It's about being able to deflect them and prevent them from doing bad things.
Sure, it's easier to deal with something you can detect in the first place, but if you can effectively block them by putting up barriers, physical and electronic you will have succeeded in your primary goal. So here's my approach..
1. Do your best to detect them, use sound, video and detect the RF signals emitted by the device and the pilot.
2. Erect physical barriers that are not visible to the operator or the device. I'm thinking there is a LOT you can do with simple fishing line in this regard, but I'm sure a lot of tall trees would serve an excellent purpose too. Put up an obstacle course.
3. Put up electronic fences using short range GPS and WiFi jamming around the "protected" area. You can effectively reduce the ability of a drone to find it's way around and make it impossible for it to be remotely controlled.
4. Concentrate your efforts on finding the PILOT. They will likely have an RF transmitter in their hands, so it shouldn't be that hard, unless the drone is self guided (which is why you jam GPS and provide physical barriers).
5. And Finally, if you do detect something flying where you don't want, come up with some non-lethal ways of bringing it down. You don't need to fire anti-aircraft guns at it, there are ways I can think of which wouldn't present much risk to people, but would be effective in bringing down your average hobbyist's drone.
So I say again, detection is but a small piece of the total security puzzle here, and trying to use audio detection is about the LAST way I'd try it...