Nobody said that automation would happen magically with no effort. So if you're offended by the claim that you imagined, you can stop. Rather obviously, it's a huge investment of effort/money to automate any complex process well, and that's invested because it makes the ongoing economics much better.
And while construction is harder to automate, there are companies automating construction. It's a complex collection of tasks to automate, and construction by definition is done in the field which complicates things, so it's not as far along as factory automation. But there are companies automating production of home components, for assembly on site, which does in fact make home construction much more efficient (and higher quality). And there are companies doing POCs with huge-scale 3D printing (using concrete) and pick-and-place (using bricks) to automate construction.
I'm not sure why you think that construction being fairly manual right now means that millions of other jobs aren't being automated out of existance. Or that construction jobs won't be increasingly automated.