Doing something like prioritizing VOIP packets over FTP, for instance, is perfectly acceptable,
Is it? I'm not sure I agree.
If my connection is saturated while I am using both VOIP and FTP, it is entirely acceptable to me that my ISP prioritizes my VOIP traffic over my FTP traffic.
If my ISP's total uplink connection is saturated (whether or not this should happen is another discussion), it is entirely acceptable to me that the ISP throttles its users that are currently using the highest amounts of bandwidth. Ideally, they throttle every user using more than X amounts of bandwidth down to X, where X is the highest number that they can sustain; and not do anything for all the users using less than X. This done without looking at the type of traffic of the different users, only the total bandwidth use. Of course, within the scope of a given user's such-throttled bandwidth, that's user's VOIP traffic may be prioritized over that user's FTP traffic, per the above clause.
But it is not acceptable to me if your VOIP traffic is prioritized over my FTP traffic independent of our total bandwidth usages. If I am trying to use 100 Mbit/s of FTP and you are trying to use 100 Mbit/s of VOIP and the ISP can only sustain 120 Mbit/s total, then it can throttle us both down to 60 Mbit/s, but it must not throttle me down to 20 Mbit/s instead because VOIP outprioritizes FTP. And when I am trying to use 50 Mbit/s with my FTP and you are trying to use 100 Mbit/s with your VOIP, then you go down to 70 Mbit/s, while my bandwidth stays intact.