I've found that a trackball works very well for my toddler's hands. The larger the ball and buttons, the better. I found that the ability to hold a mouse still while clicking buttons was frustrating for my son. Also, it was not intuitive for him to pick up the mouse when it got to the edge of the mouse pad. The concept of a trackball works very well for him. He's 3-1/2 now and knows how to log in to Windows (clicks his picture, no password), click on an icon on the desktop to play his games and close the game when he's done.
Software:
We've had good luck with the JumpStart series of games. We have three of them, starting with JumpStart Toddler, which is mostly button mashing, mouse clicking. The other two have games where a picture is obscured with bubbles, for example, and he has to roll the mouse over them to 'pop' them and display the picture. We also have some Edmark Software games, like Bailey's Book House which he's starting to get into more, and Millie's Math House, which he's starting to get into more with counting and basic addition/subtraction. The software is not very well coded -- it tends to crash a lot and the clickable area on certain objects are too specific for a toddler's hands (like clicking on the fishing pole), but the things that it teaches are very age appropriate and keeps my boy interested.
Hardware:
I have a built-in desk in a common area of the house. Under the cabinets, I have a laptop on a shelf with a LCD monitor bolted to the wall. I have a Logitech Trackman Marble and an old keyboard that is easily replaced for $15.