Well put, sir.

It's an issue anywhere there is congestion. Which happens anywhere people don't like wasting money.

Buffering versus line utilization is an interesting relationship. You NEED buffering if you want to keep the line highly utilized because data comes in randomly from various places -- if you have small buffers, then the various TCPs will often collide, drop, and backoff even though the line isn't highly utilized. The "right" amount of buffering is a function of how much you want to pay, the utilization you want to achieve, the latency you are willing tolerate, and, assuming an adaptive protocol like TCP, the round trip time of from one endstation to the other (not just on your segment).

In the core, the lines are fast, but also very expensive. Providers want those things well utilized -- a 10Gb pipe which is only running at 20% isn't earning enough money. So the core tends to have deep buffering, lots of simultaneous flows, and hence runs at high utilization. It's not uncommon to have buffers on the order of several megabytes per port (providers often will measure it "milliseconds").

In a LAN, gigabit lines run everywhere, and the wires are short and cheap. They tend to be cheap switches, with shallow buffers. No one cares too much if packets are dropped, there is plenty of bandwidth to resend things. It's not uncommon to have 2-4 megabytes shared for the whole switch (24 ports or whatever).

In the first case, prioritizing helps avoid large latencies, since the core has deep buffers, and enough users to keep the buffers busy.

In the second case, prioritizing helps avoid packet drops, since the LAN has shallow buffers.

-netchipguy

It's not as simple as whether the ISP is running at 100% or not (i.e. your packet will get through, or not). Some apps are very sensitive to latency (voice, gaming, etc) while most are not.
The switches have buffering which gets emptier, fuller, emptier, fuller. When it runs out of buffering, i.e. your 100% situation, packets get dropped, and TCP "backs off" to try to avoid that happening again in the immediate future. In fact, Random Early Discard (RED) protocols will drop the odd random packet, with increasing probability as the buffer fills, to let TCP know to backoff.... before LOTS of packets start getting dropped. If you do get to 100%, it shouldn't last long.
However when your time sensitive packets are in the same queue as it gets emptier, fuller, emptier, fuller, then even if the buffers never fill, you still suffer from increased latency... and also latency variation (jitter), which can be even more problematic (when do you decide the packets never coming and you need to fill in the gap?). Furthermore, you'd like to avoid dropping these time-sensitive packets with RED (which of course would be another "non neutral behavior"). That's because these kinds of apps generally send a steady stream, they will sorely miss the data in that dropped packet, and anyway they won't backoff in the face of drops, defeating the whole purpose of RED.
Enterprises who use IP Telephony will usually put that traffic at a higher priority (and, for that priority, disable RED). Not because their gigabit LANs are at 100%... they do it because it makes the telephone calls almost as robust and low-latency as "fixed lines".
The whole Net Neutrality debate would perhaps get somewhere if people agreed on what they were talking about. There's too many very different ideas bundled into the same name. The version that makes it illegal to willfully delay/block/etc will get 90% support. The version where it's illegal to prioritize ANYTHING is much more debatable. Those who have actually rolled out services over shared medium (IP telephony, video conferencing, etc) will have a lot of information to share on the latter.
Simply throwing bandwidth at the problem is not (yet) a viable solution, since folks are still figuring out ways to use all the bandwidth they can get. It's like saying "I don't need background threads and foreground threads, just treat them all the same and make the CPUs faster". Sounds nice, in theory.
Think of a service like Skype. Assume we want that kind of innovative service to prosper. This absolutely requires that service providers don't block/delay Skype packets. Meanwhile, to hit the quality and reliability of "fixed lines", some way to mark that small number of packets as "important" would help A LOT.
-netchipguy

I don't "sell my software" at Best Buy. I sell "upgrades" to an appliance that you must already own. Your right to copy the software isn't based on whether I chose to build on commodity PC hardware, or design my boards or even chips.