Prioritization only comes into play when there is congestion. Yes, QoS can be designed to let the little game packets ahead of the big video packets, but as a network engineer, I constantly see this:
1. Congestion starts
2. Someone implements QoS, taking a TON of time and using all sorts of advanced features on the gear. Sometimes this causes CPU use to spike, requiring more faster hardware. Sometimes you run into a bug that only relates to QoS, etc. Lots of time, money, and maybe some downtime before the dust settles.
3. Everyone is happy for a week because it works right.
4. Since everyone is happy, no additional capacity is ordered
5. Traffic continues to increase, causing even small packets to get delayed
6. Buy extra bandwidth anyhow
7. QoS tuning done before is not used because there is ample bandwidth
8. Traffic increases, reaching bottleneck again
9. QoS engages again, no one notices the increase
10. High-priority packets start dropping again, requiring more bandwidth that takes a long time to show up.
11. Order more bandwidth, and piss everyone off as they wait.
12. GOTO 3
Why do all of that when you can:
1. Monitor usage. Look at history to predict congestion
2. Order more bandwidth 90 days before you must have it.
3. Repeat
Also, keep in mind that QoS only works on traffic your're SENDING, not the traffic you get. By the time you get it, it has already dropped the packets and your link is full.
So, all of this QoS work needs to be done by the people that want you to buy more bandwidth. This is why it will never happen at the ISP level.