What we need to assure the privacy of medical information is technological means to place the control of the data squarely in the hands of its rightful owner -- its subject. My doctor shouldn't have my file, I should.
I don't see any way to make that work without adding a HUGE drag on all aspects of providing health care. All the participants in the process need to access the patient's data even after the patient has gone home to complete the care.
Imagine that my health care record was stored on something like a USB flash drive that I carried with me (when I wanted to), and ONLY there. I think that's what you're proposing. Here's how a typical 6 month checkup would work for me:
Go to doctor's office, let doctor view record so he knows what to check about me, get examined, let doctor load instructions to lab for some tests into the medical record. Pay doctor in full for visit. (See below) Go to lab, let lab tech see record so he can draw blood, etc. Go home. Lab calls, results are ready. Go to lab, let lab load results onto my medical record. Pay lab in full. Go to doctor's office, wait, let doctor see lab results, compare to previous results, etc. Doctor adds a new prescription (or changes existing dose) to medical record. Pay doctor for second trip in full unless included in first payment. Take record to pharmacy, let pharmacist see prescription. Wait for it to be filled. Pay pharmacy in full. Go to local office of my health insurance company, let claims person see record so they know how much to pay me. Insurance company agrees to send me the money, but it will be mailed from the home office in SlowMail, Nebraska. I go back to work, having missed work for at least three occaisions.
If no one is allowed to have medical records on you except you, the system devolves back to sneakernet -- your sneakers. And no one can bill your insurance company without some record of what they did or why, which they aren't allowed to have. I can only imagine what their financial auditor would think about having such sparse records of where their income came from. And when the DEA visited the pharmacy, and asked where all the Percoset went? They'd have no records!
I'm not a fan of any EHR proposal I've seen yet. But your health care providers need to keep records of what they have done to/for you. That's not to say that you shouldn't have access to them, but they need them, too.