Well 64kb for the BIOS+BASIC, 64KB for video, and then people wanted network card ROMS's, SCSI ROMS, EMS pages, and even more crap. The 384kb window was pretty small too.
1MB was too small for a 16bit processor, it's more so Intel's fault. And IBM for not selecting the 68000 processor which had a much larger 24bit (16MB) address space capability. But knowing IBM, they would have gone with the 68008, which had an 8bit data bus for those glorious 8bit ISA slots, and was available in a 20bit address variant, because 1MB is more than enough.
But heck in CP/M land we were trying to squeeze by in 64kb. 640kb a 10x improvement seemed astronomical.
The move from 32bit to 64bit hasn't felt as earth shattering though, I mean it's nice having 16GB of ram directly accessible, but I just wind up running a bunch of 32bit stuff that can get a full 2GB of space (since the 640kb grew into 64MB with early 386's, then 512M, now Windows NT split the 4GB 50/50 and when 2GB wasn't enough 'enterprise' gave us 3GB, and shrunk hardware to a single GB, now in 64bit space we can have 128GB of ram (and growing). But people want to map their video cards 100% into processor space, which grows just as fast. Considering VGA worked in 64kb (EGA/CGA/MDA in much less), now video cards with 4GB aren't that uncommon.
It's a never ending race.