Freaking Christ on a stick. DTrace might be among the two or three most powerful tools I've ever used.
Try to remember back to when you were first learning to code. Some kind soul took pity on your printf-debugging booty at some point, and showed you how to use a source-level debugger, and it felt as though scales had fallen from your eyes. "So this," you thought, "is how software development ought to work!" You set breakpoints, and watchpoints, and stepped through line by line, isntruction by instruction, all the while cackling in euphoria at your newfound omnipotence.
DTrace is the kernel developer's version of that. I predict that people will someday be amazed that performance work was possible without this tool. Old-timers telling debugging tales of the days before DTrace will attract the same hushed, awed crowds found around today's old-timers talking of punched tape and soldering irons.
Unfortunately, dtrace only exists for Solaris. I love Solaris. I learned to program there, it was the first real OS I got to know with any intimacy, and in many ways it still feels like home to me. I also have friends who work on Solaris, and they are some damn fine engineers. However, unfortunately for us all, Solaris seems to be a platform rapidly spiralling into quasi-irrelevance. Sun, IBM, anyone with a stake in Linux and some smart people, please sit some folks down with the dtrace manual, and rip off this great piece of software for Linux.