My kids (now 3 and 6) use old thinkpad t22 - it's totally kids-proof. It was dropped from the table several times, hit with ball, you can see tooth marks on the lid etc.
There's great tux-paint drawing app (there's no need to be able to read to use it), there are few great sites for youngsters, eg. www.uptoten.com
Remember to limit the time spend at computer (like 30 minutes/day for 2-5 years old - it may sound long but we don't watch TV at all).
You may need to prepare content-filtering proxy (Squid+addons serves well) and turn on "safe" mode in google (my son uses google since 5).