It was impossible to do this until the past 2-3 years.
Untrue.
In 2004 I bought my last subsidised phone - a Sony Ericsson P900. And the only reason I did this was because there was a loophole in the Orange contracts that meant I could get it dirt cheap by getting it on an expensive tariff and then change to a cheap tariff after the first month (they closed this loophole shortly after). My next phone was an HTC Dream in 2009, bought used off eBay. On Three's PAYG tariff that worked out pretty cheap. When the HTC Dream died in 2012, I imported a Samsung Captivate Glide and just swapped my Three SIM into it.
So the option to buy a handset and put it on a tariff of your choice has been there for years, if you actually look. But almost no one *advertises* off-contract handsets, so a lot of people don't even realise that you can do this, so they get a standard subsidised handset on, what they seem to think, is a good deal because they pay a low low upfront price and then a fixed monthly fee which gives them way more inclusive minutes/texts/data than they are ever going to use. If they had actually investigated their options, a lot of people would've realised that it was cheaper for them to buy a handset and put a PAYG SIM in it, because they're never *really* going to use those 10,000 minutes per month that they would've got with the subsidised phone.
Then, after 1-2 years, the MNO writes to their customer to say they can get a "free" (or low price) upgrade if they renew their contract, and you'd be stupid to turn down "free", right? Again, people don't investigate their options - if they did they would often realise it would be better to stick with their existing phone and move it onto a cheaper tariff.
And of course, no one in the industry wants to change this - the MNOs are making lots of money through these overpriced contracts, the phone vendors love the fact that everyone chucks away their perfectly good phone every 2 years and gets a replacement, and the customers usually don't know any better.
I guess add to that that for some crazy reason, phones are seen as a status symbol and therefore everyone's always got to have a brand new phone. TBH, from my perspective, the more recent phones don't seem anywhere near as good as the older ones, so I am loath to "upgrade" my phone. The HTC Dream may have been slow, but the form factor was fantastic; the Samsung Captivate Glide that I replaced it with is verging on the "slightly too big" side and the keyboard isn't anywhere near as nice to use; If I had to upgrade now, I'd be hard pressed to find anything to replace it with - none of the current phones have hard keyboards at all and screen sizes seem to have become stupid - everyone seems to be competing to be the first to make a phone that's even less likely to fit in your pocket/hand than their competetor's. Yes, the internals of phones are getting way better, but the form factors are far worse.