That half of your immigrant population that comes outside EU is mostly a legacy of UK being a former empire. If skilled immigrants from the EU would have it more difficult than they have now, they will just move elsewhere where they are more welcome. I mean, why should Poles and Lithuanians go to UK if they have jump through a lot of hoops to do that if they can just as well go to USA?
And UK will just get more people from Pakistan.
And when it comes to the retirees, thanks to EU, health insurance and government pension funds are interconnected, making it easier for the retirees to live anywhere in the EU using the local services (this is why, by the way, I am going to retire to Czech Republic or Slovakia - it is very easy for a EU citizen to do and I already speak the language somewhat). For non-EU citizens using the local health services would be much more difficult.