I think the thing you are not considering is that we are currently paying both ways: researchers pay page fees to publish, and their institutions pay subscription fees so the researchers, grad students, etc. can access the journals. Both of these payments come mainly from the same place: research grants from major government science agencies. The researchers get grants and include publishing costs. The researchers' institution taxes the grants ("overhead fees") which generally gets distributed to the library who purchases subscriptions.
So, if all grant-funded research was publicly available, and researchers had to pay higher publishing fees, but the library had to pay lower (or no) subscription fees, it all balances out to roughly the same amount of money. And it's better to have the researchers doing all the payment, so funding agencies can limit the amount of publishing costs they will fund. What do you think the big journal publishers will do if NIH and NSF suddenly say all grant-funded research must be open access from day 1, and they will only fund publishing costs up to $50/page? They will have little choice but to live with lower revenues.