In fact, if you're on the lands managed by the FS and you don't have a permit you're likely there illegally. It's the national parks that you have free access to.
Where do these people come from? Free access to National Parks? Hardly. While I admit that a National Parks Pass is one of the greatest deals around, it is certainly not free. National Forest, on the other hand, I have never once paid to use (outside of camping with friends that prefer to use the paid campgrounds instead of open forest access). I did pay I think $7 to get a firewood permit when I lived in Summit County (allowed to cut standing-dead beetle kill only), so that wasn't free, but it was hundreds of dollars cheaper than buying the firewood for the winter. Also, I will admit that tomorrow the National Parks will be free (instead of the $10-25 entrance fee you would normally pay).
Most of the lands you are going onto as an outdoorsman are managed by the NPS and not the FS.
I would be curious to see what you come up with as a citation. The majority of Federal land is in the West. The majority of that land in the west is part of the Forest Service. There's lot of National Parks as well, but their area is nothing compared to NFS and/or BLM land.
I am an avid fisherman, and I can probably count on a single hand the number of times I've bothered to fish in a National Park. With few exceptions (Slough Creek and some of the other "Holy Grail" type streams that exist in National Parks) there are just too many people to have any fun. National Forests, on the other hand, are generally the best places to go for public fishing access.