My kids have had computers at their disposal since they were 2. My computers of choice? OLD ONES!!!
Their first computer was a 800MHz cast-off which my office sold off at a cheap price. Perfect kids computer, plays most kids-oriented games, and best of all, I didn't have to worry when they used crayon on the screen or yanked hard on the CD tray or spilled chocolate milk all over it. It's since been "upgraded" to my wife's previous computer.
If this is the first time you've had a very young kid around a computer, then you can't imagine how tough he will be on technology. He may be gentle when you're around, but as a 2-to-4-year-old, he will test his environment and his surroundings (including the percussive durability of his surroundings), often when you're not around to stop him.
If you really really really want to buy new, then you may want to consider the OLPC, currently on a get-one-give-one deal at Amazon:
One Laptop per Child XO Laptop (Give a Laptop, Get a Laptop). This is the legendary "durable-for-kids-in-third-world-countries" laptop, and when you buy one for your tyke, you're also buying one to get shipped overseas to a disadvantaged tyke somewhere else in the world. It may serve your needs, and helps out a good cause.