Yes, there's a point of diminishing returns. Absolutely. I hit that wall myself many times in college.
And yes, course loads can get down-right evil. And commuting time can be a killer for some.
I'll gladly concede that your second year left you no free time. But what about 1,3 and 4? Summer terms? While you obviously had far less free time than most, are you saying you had 80+ hour weeks every week for four solid years?
I may appear to a bit dismissive to time concerns, but I've always been leery of people who claim they have no time. I heard that a lot in college from folks who were out partying every night and most weekends. Who made trips home every other week, and spent entire summers on vacation. I hear "I have no time" and it immediately translates in my head to "I'd rather be doing something other than X."
Personally, I squandered many opportunities for personal projects in college. But it wasn't because I didn't have the time to do them, it was because I wanted to do something else more (sleeping, drinking, dating, playing Frisbee etc.). Most people aren't honest with themselves over the difference between "time" and "priorities"