The clinic I worked with (as programmer) once announced that they HAD to destroy records after 7 years. As I recall, HIPAA only required them to keep the records for at least that long -- not keep them then destroy them at 7 years + 1 day. But with that kind of misunderstanding, we can't trust our doctors' offices to be the keepers of our data.
Also, this isn't just about us keeping and reading our own data: it's also about making it easier to go from doctor to doctor. Sure, a hospital/ER might willingly give out data to patients -- but is it in a format that's easy to import at all other doctors' offices? No? Well damn. How does one doctor know what else you're seeing doctors for? You ... tell them. In that painful intake form, that's hand-written, every time you come into the office. My sister went to the trouble of taking all her records from other doctors (painful to get) to a new doctor, only to have him dismiss all of it and start his diagnostic path from scratch. Why bother to read through someone else's notes? Pshaw! Just start over. No reason to save time by not repeating experiments, when a patient's well-being is on the line, and they came to you because previous stuff didn't work.
We should have our data. And we should be able to present it when we go to any doctor, to bring them instantly up to date on your history -- generally as a patient and specifically to the conditions you want treated.
The closest I've seen is this: vets' offices. Maybe it's thanks to a monopoly in the vet industry, but quite a few of the vets we've seen over the years (as we moved from state to state) used the same software. They could read the print-outs we brought from other vets' offices (we had to keep all that in a folder ourselves, nothing electronic) because it was in exactly the format they were used to. That's at least a step up.
France, by the way, uses (used?) a nationally-issued "carnet de sante" in which all your doctors would write their notes, at least for kids. Standard format, all vaccinations and other procedures and observations recorded, dated, with doctor information in case a phone call was necessary. And it was carried by the patient. I don't recall french doctors' office having large shelving units full of old patient data -- they relied on the patients themselves to bring that back in each time. And they took the time to re-read old notes, both from themselves and from other doctors. It IS doable.