Hmm, odd. I don't remember it being like that at all. If anything, I had a lot more time to learn things than I do now at college.
As a kid, my mom read all sorts of great books to me, and when I was older she could literally just leave library books on the kitchen table. I had learned that books are interesting, so of course I read them. Besides fiction, I loved books about science and history. I even tried to read some Platonic dialogues in 4th grade. I was really into spy stuff for a while, so we also did a lot of codes and ciphers, which quickly translated into the fun parts of math. I knew I wanted to go to college, so we did some formal curriculum for a couple hours every morning in middle school. It was mostly lame, but it was helpful with math at least. I also joined the "Homeschool Film Club". I learned Adobe After Effects and did a lot of camera work for the local public access channel.
In ninth grade I decided to try a fairly rigorous Christian private school in the area. It was fine, I got straight As, but it was boring. The kids had no motivation to learn, and I could progress in most subjects on my own faster than at the school. (Math was again the exception, I had a fantastic math teacher.) So I went back to unschooling in tenth grade. I was really into popular science books at this point, and I read a lot about theoretical physics and evolutionary biology. I was also reconsidering a lot of the religious ideas I had, so I was reading a lot of hardcore theology.
I discovered UC Berkeley's online lectures around this time, and listened to a bunch of college-level psychology. Eventually I became interested in the philosophical side of psychology, and started investigating philosophy. Fortuitously, UC Berkeley has a philosophy professor who likes podcasting, so I listened to a few of his series of lectures. I went through the first half of Heidegger's "Being and Time" this way. It's a hideously difficult book, even for philosophy, but I had a lot of free time!
During this same period I was working with a professional theatre in a neighboring town, as well as a couple local community theatres. Since I didn't have set hours for school, I was able to be there whenever they needed me. I acted in several shows, and worked on lighting and general tech work.
Oh, almost forgot. I also took a couple community college classes, in physics and writing. They were both absurdly easy and I didn't learn anything, but it looked good on my transcript to have some formal classes at the college level.
I decided I wanted to go to the University of Chicago, if I could. It's ranked 8th in the country, but in my opinion it's academically better than the Ivies and other colleges ranked above it. (MIT and Caltech are the exceptions, but they are also more narrowly focused.) I applied, got in, and am currently attending. It's awesome being around other academically engaged people, but I kinda miss the chance to learn on my own. Luckily, I have the summer to do that!
I admit I'm a bit of an outlier, and I probably would have done fine in the public schools. Not everyone who is unschooled will have a natural passion for academics. However, if anything, unschooling is even better for people who don't want to become academics. My younger sister has some minor learning disabilities, and is far more into the arts than the sciences. She spent her high school years learning about theatre and music, and has become a fantastic actress. She spent the summer working with the professional theatre company I mentioned, but in a much more intensive way than I ever did. However, while it's not her focus, she still loves learning intellectual things as well. She's currently doing some pretty substantial research into psychology and counseling. She's trying to decide between theatre and counseling as possible careers. Either way, I'm convinced she's in a better position than she would be at the public schools, where she'd likely be forced into special LD classes and not allowed to explore the things that actually interest her.
It's entirely possible to do unschooling badly, but that doesn't mean it's inherently a bad idea.
