Record albums are kind of the pop equivalent of a symphony. They last roughly the same time, the length of a CD. It's a myth that it was chosen to be the length of Beethoven's 9th, but the intuition seems to be about right: that's about how long people are willing to listen to music before they need a break.
Not all albums are constructed well, but a good album has some kind of structure and forms a complete unit of music, rather than just being a bunch of songs. It's about as unified as the movements of a symphony. Songs don't quite correspond to movements (a movement is likely to be 15 minutes, a song 3), though as you say there is yet more structure within a movement.
These attention spans are probably not absolutely fundamental to human nature, but they're at least deeply culturally embedded.
It is too bad that things seem to have settled in a place that have eliminated long-form songs like Bohemian Rhapsody and Stairway to Heaven (about the length of a symphonic movement, and each definitely composed of sections that are very different musically). I would be very happy to see those return, though songs like those are rare epics, requiring tremendous skill and insight to construct. I don't know if Pandora will want to play them to you, since it means they get paid just once for feeding out bits that they could have been paid 3 or 4 times for, but I suspect that if somebody writes a great song like Stairway there will be demand for it. The streaming services will want to serve that demand, and since they have control over how often it comes up, it may cut only a tiny bit into their profits.