I just don't buy that audio can't be used. With an array of high quality microphones spread over an area fed into a software radio and some pretty hefty computing power you should be able to look for the rhythmic audio that your typical copter type drones will generate. Because even if they change the size and shape of the rotors all that does is change the frequency not the amplitudes you will see from the rhythmic action.
With some proper math and the right computing power you should be able to identify drones pretty rapidly with fourier analysis and the same microphone array could use Doppler effects to calculate position, and the directional vector. Combine this with some systems to double check such as heat, RF and conventional high band doppler radar and you should have a system that will get 99% of the drones and even provide targeting to a shoot down type system.
The key here is you need some pretty dam good microphones spread over a pretty large area feeding into a pretty massive computer array. It wouldn't be cheap or easy. The easy thing is use a doppler radar system that cross-checks it against RF emissions to eliminate birds. But IMO the best system would use all three, high band doppler radar, RF emissions and audio (and maybe even heat). With three cross checks you should be able to get pretty good accuracy.