Doom ran in 32 bit mode too? Segment addressing were no problem, the framebuffer was addressable as 64kiB at 0xA0000 by default. I don't know what you mean by "Data was packed into 16-bit values", are you talking about internal structures in Doom?
"Mode X" is the unchained 320x240 mode with square pixels using a planar layout, making the whole 256kiB VGA address space available. Horizontal: pixel 0 => plane 0, pixel 1 => plane 1, pixel 2 => plane 2, pixel 3 => plane 3, pixel 4 => plane 0, pixel 5 => plane 1 etc. Each plane was 64kiB in size so 256kiB in total. Getting good performance from unchained 256 color modes wasn't easy and essentially reduced to trying to avoid changing the active plane.
The 320x200 mode was more efficient for some types of rendering but had the disadvantage that it only used 64000 bytes of the video memory, the framebuffer address n mapped to plane n&3, address n>>2 as viewed by the CPU. This meant doublebuffering wasn't possible, the alternative was rendering in a memory buffer and then copy to the framebuffer.