Well, yes, it is primarily shooters to blame. I'm not afraid to admit that sometimes gun owners can be our own worst enemy, and this is one of those instances.
Oddly, some of the worst behavior I've seen was from stupid/bored/drunk/high townies that live nearby, and basically call these areas their back yard. After all, they don't have to drive an hour to get there, and don't consider shooting opportunities as a scarce resource. A fair share is also due to campers (more like squatters sometimes), and no doubt hikers as well, as even that demographic has two divisions: people who basically leave no trace, and pigs like everyone else.
As a hiker, I'm always picking up hiker related rubbish on the trail (energy bar wrappers being the most common), but there is a practical limit to how much damage one hiker can do, namely the weight they can carry on their back. As a shooter, I always bring along a rake and shovel and at least a couple huge industrial grade trash bags for cleanup after I'm done. I often fill at least one bag and bring it back to the city for disposal.