I have lost four of my friends to airplane accidents. Two were pilots -- in one case the it clearly his own fault, and in the other it was extremely bad piece of luck. The other two deaths were the direct result of naively trusting the wrong pilot.
I see two flavors of comment so far. Non-pilots saying they think the idea is scary, and pilots saying "aw, pshaw, I am well trained, what is the problem?". Well, I am a pilot myself (commercial pilot and certified flight instructor), yet I strongly agree with the "that's scary" crowd. I've flown many thousands of hours in all sorts of locales, weather, and equipment. I've handled numerous emergencies, with never a scratch. I've taught hundreds of other pilots to fly. But, in all that time, by far the scariest moments I have ever had in the air were occasions where I made the mistake of riding as a passenger with the wrong choice of pilot!
Those who place their faith in the FAA's training standards, simply fail to understand that the ratings indicate compliance with the bare legal minima -- essentially they mean nearly nothing.
Nor does safety correlate with pilot rating. I've met some mere student pilots that I'd sooner trust with my life than many commercial pilots. The variation from one individual pilot to the next, regardless of qualifications, by far exceeds the variations from one rating to another. That variation comes from preparedness, attitude experience and common sense. Bottom line, with the exception of airlines (where I have no choice!) I will NEVER ride with a pilot whose experience, skills, and attitude I do not personally know first hand. And, I'd never advise friends or loved ones to ride with "just any old pilot".