The only reason manned flight works is air traffic lanes, it's a real highway system up there for the manned guys.
That is simply untrue. The reason manned flight works, and the basis for the system, is the concept of "see and avoid". The only time that "see and avoid" is not rule number 1 is if you are flying in the clouds. If you are doing that, then it has become ATC responsibility for "separation" -- i.e. keeping other people flying in the clouds from running into you and vice versa. If you aren't operating on an IFR clearance, then there are rules about how close to the clouds you can go which are based on giving you and the guy who pops out of a cloud time to see and avoid each other.
Otherwise, "see and avoid", wherever you happen to be.
Now, there are "airways" that are defined routes, but there is no requirement to actually fly on them (unless you've received an ATC instruction to do so), and even large airliners don't need to fly on them. They'll often get instructions like "direct XYZ direct ABC" which bypass those airways. And the FAA Next-Gen concept is to do away with such airways and operate mostly direct. That's because there will be better tracking of aircraft enroute.
And the guys that fly "off-road" (some manned Cessna and tv copter pilots fly over my house! And I found is a big no-no & should be fined) are penalized if caught and yes do create potential dangers.
It is not a danger nor is it illegal to fly "off-road". You are quite incorrect in your belief that it is a "big no-no". It IS a no-no to fly closer than a certain distance from people or structures, or below certain altitudes (1000' over a populated area, unless in the process of landing or taking off), but "off-road" is quite common and quite safe. And that 1000' restriction doesn't apply to helicopters. Their rule says they have to fly at or above a level from which a safe landing can be made in an emergency.
5miles of an airport? I live 3.5 miles from one & can't fly in my own back yard?
Life sucks, but yes. Many airports have controlled airspace from ground to 3000' AGL in a 5 mile radius, specifically to protect aircraft operations into, out of, and in the vicinity of the airport. That's the general boundary of control of ATC at a towered airport, and it is mirrored in uncontrolled fields.
manned operations? really no one (even the ATC in some ways) has good ADB-S yet.
ADS-B has nothing to do with a prohibition on interfering with manned flights. If you can see your toy, then you really ought to be able to see that much larger manned aircraft that is passing by. You don't need radar or transponders.