You can get away with that using smaller "park flier" planes as they are less able to cause damage, but there is risk involved.
In this case, size does indeed matter. AMA defines a park flyer as less than 2 lbs, less than 60 mph and electric only. Many of our fit into that category, but others do not. My heaviest plane is around 5 lbs., with a 60" span and some of my faster planes will fly in excess of 80 mph. We don't have many that fly larger craft in these spaces on a regular basis, but we do have internal combustion craft. If we need more room, we find an adequate location. We always work to ensure that everyone is safe, even those in surrounding areas. We also stay below 400' AGL when flying LOS for practical reasons. Most often, I'm under 150' AGL. FPV is another matter altogether.
If you hurt someone you will be personally liable and wont be insured.
There is a question of liability, but your comment seems a little broad. I may not be insured by the AMA (I'm never quite sure what that policy covers), but I certainly carry insurance in addition to AMA membership.
By comparison any official club will register with international model aircraft organisations. They train their members in the rules of safe flying, and provide insurance if something goes wrong. A few years ago my local club was $180 per year, not cheap but not too bad considering they have to maintain grounds as well.
As for fees, $180 would be high in my area. Most are between $50 and $75 annually. The advantage is apparent the first time you fly at one of their fields. Still, I'll fly at the local parks whenever I can and avoid the 20 - 40 mile commute. All of the AMA sanctioned fields are a good distance away.
We always select the venue with the aircraft requirements in mind.