As a graduate student and college instructor, I would argue that one thing that students will lose from skipping the lecture is the horizontal social connections between students. Even if there's no discussion or opportunity to stop the teacher and ask questions, attending class gives students the opportunity to forge social relationships before and after class that allows them to compare notes and share experiences. Students could time-shift a lecture and discuss it later, but it seems less likely, and there's something to be said for talking when the lecture is still fresh in their minds.
I also wonder how attentive students would be watching a podcast compared to sitting in a lecture hall. Sharing the same physical space demands at least the appearance of attentiveness.