Arduino has a really short friendly learning curve - the system is designed so a random not-very-technical artist can pick it up, start doing blinky lights and sensors, find lots of interesting community support and demonstrations and applications. All the pieces you need to get started are right there - hardware, software, IDE, sensors, output devices, documentation. The Arduino hardware is fancier than a bare-bones AVR chip on a breadboard (and building one of them is a good second project), but it's still pretty cheap. The software may hold your hand a bit too aggressively, but once you've learned what you're doing you can get deeper (think of it as a mostly-C scripting language.) If you'd rather use gcc to write your programs at the bare-metal level and avrdude to download them, you can, but Arduino lets you do your work at higher levels until you need that. You could buy an ISP programming tool for $20-50 to program raw AVR chips with, but you can also use a ~$30 Arduino to do that job, so just go buy one.
Once you've used the Arduino a bit, you might want to branch out to a TI or STM development board, or something like Propeller with a lot more CPU horsepower if you need that, or PIC (if you want to know what people used to learn on before Arduino.)
Stuff you're going to have to buy - whatever prototyping board you want (I'd recommend Arduino), a solderless breadboard or two, solid-core wire in a couple of colors, some LEDs, assorted resistors and capacitors, probably several different types of sensors and output devices, maybe a power supply (USB gives you 5v, which is just fine if you're doing everything tethered to your laptop or have a USB phone charger around.) If you don't have electronics stuff around home already, you'll probably end up spending $100 or so, typically for a kit from Sparkfun or Adafruit or MakerShed, plus some random shiny-looking parts from their catalogs, plus you'll start to find Radio Shack very useful when you need to stop in and get some more LEDs or various connectors (and get yourself a bag of assorted resistors and a bag of assorted capacitors if you didn't have enough from a kit.) If you're going to solder boards, you'll also need a soldering iron, solder, and some breadboard to work with.