It's not the lack of imagination. Far from it.
When the buggy whip makers went out of business, the car industry was already in full swing. They were already outputting enough cars to replace the buggies. The buggy whip makers could actually see the workers working to make them obsolete. At this time, it was wellknown how many jobs the automobile industry was creating. And it was wellknown that the new automobile not only replaced the horse carriage, it actually made it better, allowing for more trips, for more load hauled, for higher speed. The car helped to make the whole transportation business to grow more productive, and not just a few percent, it was a multitude of improvement. The demand for transportation at the same time was also growing because transportation got so much cheaper that goods or persons which would never have been transported so far and so often before, now could. Replacing the buggy with the car as the means of transport actually increased the transporting market.
Buggy whip makers didn't need to imagine the new jobs. They knew what the new jobs were, as they could see their neigbours already having them.
But if you just replace a worker by a machine, there is not necessarily a new job opening waiting. The manufacturer of the machine already has the people to make the machine, as he was able to built it. And it's not as if his business has to be growing, as the market for his worker-replacement-machines is limited to the number of workers his machines can replace. It happens that not only the worker who is replaced by the machine is out of the job, also the people installing the machine are also out of a job, because their job is now finished. And maintaining the machine surely will require either less man-power or less qualified man-power than the man-power it is replacing. Otherwise there would be no point in actually replacing them.
Automatisation of jobs in general does not create new jobs. It just frees up human labor. If that allows for huge gains in productivity (and we are talking huge gains. The mechanical loom improved the productivity tenfold, and so did the spinning machine), there might be new markets and thus there might be new demand, creating new jobs. But just replacing the human by a machine does not. Having cheaper sport news does not increase the market for sport news. The replacement of the financial advisor by a computer does not increase the demand for financial advise, because the requestor does not get a tenfold improvement on his ROI. As a maximum, he saves the few percents the human financial advisor got as his premium. The same is valid for legal expertise. People will not want to have more need for legal advise just because it is cheaper. Most people prefer not to be involved in legal quagmires at all. Compare that with the demand for cars! People love to buy cars. Or at least, they used to love it. But the demand for new cars is already shrinking at least in some parts of the world. Young people in Europe list the desire to own a car quite low in their priorities already. A similar trend can be seen in the U.S.. And which new job is replacing the car manufacturer's job? Simply none. Completely different than it was when the welder's job at a car factory replaced the buggy whip maker.