On the note of forgetting about half of what you had learned, it reminds me of a pedagogy that I am a fan of: learn the underlying concepts and how to apply them, and you no longer have to "remember" because you "know." With your example, I cannot rattle off hardly any of the trig identities -- but I can derive all of them quite easily.
I think that this is a major issue with our education system when it comes to math: memorization. For example, in my daughter's math class, when going over exponent laws, the teacher said, "anything to the 0 power is 1. Why? Because that is the way it is; it is just one of those things we need to memorize." The same kids who learn this way find themselves in a math class a few years later and cant remember if it is 3^1 = 0 or 3^0 = 1. I wrote him showing how bloody easy it is to learn the correct way -- by looking at what exponents are (repeated multiplication) and that if we work backwards from 3^4, to 3^3, 3^2, we see that we are dividing by three each time, so it is easy to see why 3^0 has to be one. Better yet, this lets students understand why negative exponents are what they are. With this proper understanding, the student can re-derive the exponent laws anytime they may need them.
I completely agree with you that Khan is not a replacement. There is something to be said for us social beings being, you know, social when learning.
The system that I envision, at least as far as math is concerned, is something that I dub "modular math" -- though, that term should not be confused with modulus. I think the curriculum should be broken up from arithmetic to calculus in models (sets, pods, mods, levels, whatever).
A student meats the material at their level, and progresses through each model. This allows a student to quickly move through material that is easier for them and to have the time required for material that is more difficult. I imagine a system whereby students participate in a math curriculum whereby they progress on their own through CORRECT (yet to be defined) use of online lectures and quizzes and tests.
A teacher is ever present and provides the one on one work that the student requires when they hit a topic that they have trouble with, or need further or alternate explanation. The puts the teacher as a facilitator that needs to be familiar with all levels of math from basic arithmetic through calculus -- and by familiar, I mean able to actually teach the concepts. A teacher may have a student in model 4 and model 10 in the same room, and when each student has an issue, the teacher would need to be able to step in at that topic and work with the student.
I have begun development of the requisite online tools for this, but Khan has, by far, a lead on videos and lectures -- and even presence. I thought of this way before Khan was popular enough for me to have heard of him, but, with most things, it comes down to who implements it first. I think that it could be an exciting next step for the Khan Academy.
The best book on programming for the layman is "Alice in Wonderland"; but that's because it's the best book on anything for the layman.