You are aware that any attempt to change the mandatory part of the IPv4 header would result in an incompatible header in any case, yes? So what does that mean?
It means that no matter which way you twist and turn the problem, you cannot escape the fact that in order to increase the address space of IPv4 you *must*, per *definition*, create an incompatible protocol. Which makes sense, of course. If you change a fundamental part of the protocol well, then it's not really the same protocol anymore, is it?
So instead of putting yet another band-aid on a protocol the engineers stepped back, took a long, good look at the problem and said "hey, IPv4 just can't do all the things we want it to, and since we will break compatibility anyway, let's create a new protocol that actually handles all of these edge cases we didn't see last time we designed a protocol, while we're at it!"