Going from 32-bit addresses to 128-bit addresses at the outset would have meant the IMPs and then the NSFnet routers (Cisco AGS+s) would have had the routing tables take up 4x the space (well 257/65s but who's counting). That would have meant a more severe and quicker router memory exhaustion with backbone routes, and a quicker move to CIDR (where we specify NETWORK/NUMBER-OF-CONSECUTIVE-1-BITS instead of NETWORK/MASK).
For every choice we ever make (routers, IP design, freeway offramp selection, etc.) if the choice has no downside and an upside... we do it... because it's stupid not to. For most of the choices, however, there's a tradeoff of an upside and a downside. In the IPv4 design 32 bits was (in the late 1970s!!! long before PCs, smartphones, IoT, etc.) was assumed to be more IP addresses than anyone would want... which is why more than 1/4 of them were never made available for assignment (Class "D", Net10, Net26, Net127, etc. and RFC-1918 taken out because sysadmins left their boxes configured with mfg default IP addresses.)
I appreciate the contributions Vint Cerf made but I can tell you that running IPv4 on 9600bps modems (back then "MODEMs") meant that we tried to send LARGE packets to make the payload-to-header ratio high but then SHORT packets to make the "responsiveness" (latency) seem acceptable. If each packet header went from 40-64 bytes to 192 bytes larger... that ratio would have significantly impacted any "real-time" (low-latency) performance. That would have impacted those of us using TELNET (ssh not yet invented), accessing BBSs, BiX, Compuserve, etc.
It's wishful thinking to look at a world where our smartphones have 32GB of memory and our data communication speed are measured in 10s of Mbps and long wistfully to have designed the header structure of RFC-793 where computers had 64KBytes (not MBytes) and communication speeds of 110baud (1970s).