Except that these are two separate problems. One is how to grow the economy. The second is how to distribute the outcome of the growth. The problem with your position is that you have already given up on the second; you are reconciled to the fact that the 0.1% is going to get the lion's share of the outcome of the economy.
You can hold off on immigration; but eventually robots will take over and do most of our jobs for us. If we still stick to the current capitalist model of society (and robots count as somebody's property/capital), and do not find a way to distribute the output of our robotic friends equitably (without requiring each and everybody in society to do meaningless jobs just to participate in the economy), we are all in trouble.