Comment Old hardware... (Score 3, Informative) 176
Old hardware is your best bet. Anything new would be unsupported by the older 2.2/2.4 kernels, PCIe, SATA, chipsets etc.
*Slot 2 Pentium II or III CPU's and Socket 370 CPU's are perfect. If you want multiprocessor, a Tyan or Supermicro dual slot/board is a good bet but stay away from any board with RDRAM using the i820 or i840 chipsets. They did however realize how big a mistake RDRAM was and Intel made SDRAM->RDRAM bridge chips so those chipsets could use PC-100/133 SDRAM. Tyan made a dual processor i840 board with dual slot 1 and SDRAM using the bridge chips.
*At least 256 meg of ram, 512MB - 1GB is ideal. Make sure your board supports the RAM you have.
*An AGP Riva TNT card or better yet, a Geforce 1, 2 or 3 graphics card. 3D support may not be available*
*Sound Blaster Live!, Ensoniq, Turtle Beach or Aureal sound cards should all work. Though the Sound Blaster Live! is probably your best bet.
*You are also going to need an ATA hard disk (2+GB) and CD/DVD rom drive, I am unaware of any P2/3 board that supported USB booting so you need the optical drive.
*If no onboard LAN card is present (most common scenario) you want a PCI 3Com 3c905B/C, or any PCI card based on the DEC Tulip chipset (21040/21041/21140/21142/21143). Many older Netgear FA311 cards also worked flawlessly, based on a well supported National semi chip that I think was a tulip clone)
*Bonus: decent 19"+ Trinitron CRT monitor. I still have a 21" Sun Trinitron.
Stay away from ISA cards as much as you can. I had a hell of a time getting my old ISA Sound Blaster AWE 64 Gold sound card running under Mandrake back in the day. And that was a "plug and play" card without jumpers. As for why to use Pentium 2/3 boards and not a pentium 4, the p4's after socket 432 willamette generation might not run a 2.2 or early 2.4 kernel. Socket 478 gained things like SATA and PCIe so its a crap shoot. Pentium 2/3 is a guarantee.
*Nvidia hardware 3D support does not appear to be supported on 2.2 kernels. I checked the README for the oldest Linux Driver and 2.4 and 2.6 kernels were mentioned. Have a look here: http://www.nvidia.com/object/linux-display-ia32-71.86.15-driver.html and check the hardware issues section in the README!
Have fun kickin it old school.