Hello, I couldn't find another way to contact you, so here we are.
I'm finishing up a PhD in scalability & performance analysis, and have done a lot of work in instrumentation. A userland instrumentation tool is part of my final research. Instrumentation is in a terrible, terrible state -- save a few points of light -- and I'm happy to see someone else in this area!!
So, as you're starting out, some tips:
1) If you haven't already done so, investigate dtrace. While available on Mac OS & FreeBSD, it's worth picking up a virtual machine image of opensolaris & playing with it there.
2) Pick up a copy of: R. Jain, "The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling," Wiley- Interscience, New York, NY, April 1991, ISBN:0471503361. It's my new Bible.
Good luck, and hit me up if you'd like to chat. For my email address, I'm [my first name].[my last name]@gmail.com
Cheers,
-ls