How to get out of jail free?

Meshugga writes: "Ok, so now you tamed your editor, mastered the true zen of your favorite shell and people hear you talking regex in your sleep. But have you ever had a complete messup on your *nix box? I'm thinking along the lines of rm -rf / on a mission critical system, using killall on Tru64, a hanging LKM for an important piece of hardware...

How did you mess up? Were you able to fix it? Possibly even without a reboot? And more importantly: How? What tools did you use? What else did you try?"