That's because Caltech has a notoriously brutal curriculum. Consequently, grades are lower than other schools of that caliber, and Caltech has the lowest freshmen retention rate of any "top tier" school. There was a rumor going around that it was possible to do two years of a physics degree at Caltech, transfer to Harvard as a senior physics major, graduate and be back at Caltech for grad school in physics while your classmates were still undergrads. Supposedly someone did this - although they were probably taking the same classes as their former undergraduate classmates when they returned.
This was also very evident at the annual job fair. Several companies from the Bay Area that drew a large amount of employees from Stanford wouldn't even talk to anyone that didn't have a 4.0 GPA at Caltech. Other companies that employed a large number of Caltech alums openly admitted to students that they would mentally add a full 1.0 to a Caltech GPA when looking at a student's resume.
FWIW the vast majority of classes at Caltech use takehome exams. The "weeder" EE course switched to an oral exam when I was there, but that was 11 years ago and I don't know if they still do that.
Disclaimer: I attended Caltech for undergrad but did not graduate, and my wife has a Ph.D. from Caltech.