They would have a very uphill battle.
For the common customer, it's generally 'nvidia or bust'. That being said, supply chain limitations might have people looking for a more affordable and accessible option, which isn't exactly a huge appealing prospect for IBM, particularly since the market has AMD trying to already fill that niche. General problem is it isn't just about specs, if it isn't CUDA then people are just not confident.
IBM hasn't really been known for in-house top performance in about 20 years. Last time they bothered to go for top performance was almost a decade ago, and even then it was tossing nVidia GPUs into POWER9 systems, and nVidia was the real meat of the solution.
To get traction, they'd need a market opportunity of folks that aren't afraid to try to leave nVidia behind, but not so adventurous as to just make their own chips instead, and for whom AMD offerings are not good enough. IBM would have to go from zero software in the space to credible. They'd have to scale out even bigger than they used to despite largely giving up on scale out years ago.
IBM hasn't really done anything to earn the benefit of the doubt in a very long time. People's memory of the once potent tech giant has faded. They've alienated all but their most loyal customers by ditching anything where they had to be vaguely in competition with other companies, so they volunteered themselves into obscure niche, and the niche is high-priced, highly compatible, but not necessarily top performance offerings.
I suppose some of us old timers would be stoked if IBM actually made a serious effort, but I think the prevailing sentiment would be skepticism after decades of neglect on this sort of effort.