Not quite. It has always been a balancing act
All you've identified there is a gap between what copyright policy requires and what we actually have implemented. I'd be the last person to say that our copyright laws, as enacted, have lived up to our proper policy goals. But that doesn't change what the correct policy is.
Copyright doesn't exist absent affirmative action by the government, and it is wholly utilitarian in nature. This means that there is no policy of balancing interests. Rather, it is a question of how it can best serve the public interest; if giving something to authors may accomplish that, then we should do it to an appropriate extent, and if not, we shouldn't do it.
It's little different than the farmer who wants to haul his carrot harvest to market in a wagon pulled by a mule. He might have to feed the mule some of the carrots to get it to pull the wagon, but there's no balancing act between the farmer and the mule. (Indeed, as soon as it's more cost-effective for the farmer to just get a gas-powered truck, the mule gets sent to the glue factory)
copyright has been deliberately adjusted to make sure that it's society that benefits from the release of works into the public domain and not a second degree economic interest.
That's not true. You're arguing in favor of monopolies controlling commodity goods, which is an odd stance to take. Society benefits tremendously from works being in the public domain, and available for the economic exploitation of any and every party that cares to give it a go. So long as anyone is free to make copies of Shakespeare, it doesn't hurt society if some publishers charge for copies of it. Given that competition is possible for copies of the same public domain work, all that will happen if one publisher tries to charge too much is that someone else will step in and sell it for less. This all works to bring the price of copies down, which in turn increases the public's access to the work, which is necessary for the work to be of use.
after the discussions about estates providing for heirs began to get serious in the 1830s and later, to make sure that families wouldn't be unduly burdened by the premature death of their income earner.
The widows and orphans argument has always been unmitigated bullshit. Works usually have zero copyright-related economic value; of the few that do have such value, they usually burn through the vast majority of it within a short time after the first publication in a given medium. Only the tiniest fraction of works have long-lasting copyright related economic value.
Suggesting that the survivors of a deceased author need longer terms in order to live off the value of a copyright requires that it be a copyright of this sort. Given the rarity of such works, it's as stupid a suggestion as saying that you might as well leave them a shoebox full of lottery tickets.
If you actually care about providing for your family, you need to take out a life insurance policy, and you need to save and invest your money wisely in a diverse portfolio. And just to be safe, you'd better vote for politicians who will enact government programs to provide actual, useful assistance to poor people.
The reality behind the widows and orphans argument is that a handful of authors and publishers who already won the lottery, as it were, by holding the copyrights on works with long-lasting copyright related economic value, wanted to preserve their gravy train. It's as if the winner of a $100 million dollar jackpot used some of that money to successfully lobby for a retroactive increase to a $200 million dollar jackpot.
Fundamentally, the idea of a copyright term that exceeds the commercial relevance of the work is to discourage people from being able to step in due to expiration and start profiting from the works of others, in furtherance of the incentive to produce new works of cultural enrichment, by making it harder for moochers to swoop in. We've gone too far because of a small number of intensely valuable outliers, but the answer is not extremism in the other direction, either.
I'm not arguing in favor of extremism in any direction. I don't think that copyrights should be short, I think that they should be no longer than absolutely necessary. An overly-long copyright is harmful to the public because it is waste. An overly-short copyright is harmful to the public because it doesn't incentivize authors as much as is appropriate. What we need are copyright terms (and scope) that hit the sweet spot where we get the most efficiency: the most works created and published yet for the least restrictions on the public.
But this also means that your disrespect for 'moochers' is totally inappropriate. Ideally we could grant copyright terms (we'll set aside scope for now) on a case-by-case basis. If the minimum copyright incentive that author Smith needs to write and publish his book is 3 years, then we grant him 3 years. If the minimum copyright incentive that filmmaker Jones needs to film and distribute his movie is 10 years, then we grant him 10 years. If the minimum copyright incentive that painter Brown needs to paint and sell copies of his painting is 0 years, we don't grant him a copyright at all. Does this allow for third parties to compete against Smith in 3 years, Jones in 10 years, and Brown from day one? Sure. But who cares? Granting one day's worth of a longer term to any of them is pointless, because they've already got the minimum amount they needed to do what we want them to do: create and publish works. It's as wasteful to grant them more as it would be to offer a construction project to the lowest bidder, yet to then double the payment to the winner just for the hell of it.
In practice, we can't fine tune copyright grants that well; we'd need to staff the Copyright Office with a legion of psychics. But we can still try to make it work efficiently. For example, requiring registration helps us weed out authors like Brown who have so little reason to care about copyright that they wouldn't bother to register. Offering short terms and renewals helps us weed out authors like Smith, who only care about copyright for a little while, and then stop because it's no longer valuable enough to them to merit filing the renewal. (We know that few works were ever renewed historically, so that's a real thing) And for authors like Jones, longer maximum term lengths -- up to a point -- could still be available. They just wouldn't be automatic, so that we don't inadvertently grant such long terms to Smith and Brown, who don't need them.
And as for authors like Black, who create a work but insist on a copyright that lasts forever, or at least for an immensely long time, even if that really is the necessary term in order to incentivize the creation and publication of the work, we can say fuck it; Black wants more than the work is worth to the public. It might be nice to have that work created and published, but a sane copyright system is more valuable than that particular work, so we'll just all have to live without, and Black can get a job doing something else.
At no point however, is the idea that we should discourage third parties from being able to compete freely ever considered, because it's dumb, basically. The copyright monopoly should not last any longer than it needs to to get works created and published. If this allows for third parties to step in while a work is commercially viable (and given that people still reprint works from antiquity, that can be a very long length of time indeed!) then so be it. There's nothing at all wrong with it. In fact, it's great, because it drives down prices and increases access to works.
