If people would just accept a decent MTU none of this would matter.
The max is 64 K but we're stuck with 1500 (including overhead) because you can't be sure that every hop will support your MTU.
Internally you can enable jumbo frames and shit will work, but once you need to go out over the internet all bets are off, so you limit your shit to 1500 and your performance goes to all hell.
We're basically delivering UHD movies via telegram.
Packet size is a tradeoff - for high throughput you want big packets, for low latency you want small packets. So fine, just tailor the packet size to your application - well no, when you're sharing a network, the packet sizes used by other applications have a significant impact.
So lets say you're doing something that requires a low latency, such as VoIP. And lets say you've got QoS set up to ensure the small VoIP packets are always inserted in front of any big packets, since that's a sensible thing to do. Look at 2 scenarios:
Scenario 1:
Transmit queue is empty, VoIP packet goes straight to the network card.
Scenario 2:
Transmit queue has a bunch of packets already in it. The VoIP packet goes straight to the head of the queue, but the ethernet card has already started transmitting another packet, so we have to let that finish before the VoIP packet can actually go out onto the network.
On a busy system, scenario 2 would be the norm, so the latency of the VoIP traffic will vary and the receiving end has to even out this latency with a jitter buffer. Lets assume an MTU of 1500 - the transmitting side has only just started transmitting a 1500 byte packet when the VoIP packet enters the queue, on a 2Mbps connection it would take 7.5ms to send this packet before the VoIP packet can start to be transmitted, so you're looking at a 7.5ms jitter on your VoIP session. If the MTU was 64K, the jitter would be a whopping 328ms, which is verging on unusable for VoIP.
Now, you may say that 2Mbps is a slow internet connection, and you'd be right, but it is also a very common speed of internet connection, so doing stuff that breaks it would be bad. Don't forget that you get latency introduced for each hop you do through though - on a 100Mbps connection with a 64K MTU you add up to 6.6ms of latency per hop, so if your traffic goes through 10 100Mbps hops, you're looking at potentially 66ms of latency.
Ideally you'd set the MTU of each interconnect independently of the rest of the network and base it on the jitter level you'd like to achieve (therefore it would be based on that link speed). And indeed this can be done - clients can do path MTU discovery to figure out the minimum MTU on the route between hosts, irrespective of the local MTU. Unfortunately, too many idiot sysadmins set up firewalls to block ICMP packets and that breaks PMTU discovery. Which means that if you're using a "nonstandard" MTU (i.e. not 1500) you _will_ have connectivity problems because your traffic will sometimes traverse firewalls that are set up by said idiots.