I wouldn't say you're stupid, but you're not understanding the problem right. ISPs configure high buffers on low bandwidth links so that the total throughput is higher, at the expense of latency, since, as you say, packets have to wait in queue until they go out when you max out the connection.
This is NOT the right way of doing things, buffers should be smaller, if you max out your connection, your packets will drop, this will cause either TCP (at the protocol level) or the application if there is no support at the protocol level (UDP for example) to back out and lower the transmission rate, WHILE keeping your latency at normal levels.
You'll still have some packet loss, sure, but the overall experience should be better if applications act nice.
And no, QOS is not the solution to this, QOS should work end to end on a connection and that's simply not possible for internet users. What you mean is shaping, or policing, at the interface level, but if you're doing that, you're just avoiding the filling up the buffers, so that you achieve what I described earlier.