Yes, and if the UK rejects freedom of movement, a) Japan won't be happy as per the document referenced above and b) I'm not sure the UK can hope for a better deal than tariff-free exchange of goods. Certainly there will be countries going "well, if Polish plumbers can't sell their services in Manchester, UK banks will have to open EU offices to sell financial services to Berlin". And this still wouldn't be "punishment", but grounds for a fair deal, IMO.
And yes, I know that May has rejected freedom of movement. But she has also rejected border checks between Ireland and Northern Ireland, so apparently there will be free movement of people through that border?
Also, yes, Spain is a bit strange, unemployment is very unevenly distributed. But that's a topic for another thread, I think. :)