Ok, excerpt from a tcpdump. The network works like this:
Erythrocyte<->XEN bridge<->Endothelial<->PPPoE<->Teh Intertubes<->m4e5e36d0.tmodns.net
11:36:20.625832 IP (tos 0x0, ttl 64, id 21777, offset 0, flags [DF], proto TCP (6), length 2880) erythrocyte.squiggleslash.link.www > m4e5e36d0.tmodns.net.9971: . 1:2841(2840) ack 529 win 211
11:36:20.626425 IP (tos 0xc0, ttl 64, id 41691, offset 0, flags [none], proto ICMP (1), length 576) endothelial > erythrocyte.squiggleslash.link: ICMP m4e5e36d0.tmodns.net unreachable - need to frag (mtu 1492), length 556
IP (tos 0x0, ttl 64, id 21777, offset 0, flags [DF], proto TCP (6), length 2880) erythrocyte.squiggleslash.link.www > m4e5e36d0.tmodns.net.9971: . 1:2841(2840) ack 529 win 211[|icmp]
11:36:23.623281 IP (tos 0x0, ttl 64, id 21779, offset 0, flags [DF], proto TCP (6), length 1460) erythrocyte.squiggleslash.link.www > m4e5e36d0.tmodns.net.9971: . 1:1421(1420) ack 529 win 211
11:36:23.731117 IP (tos 0x0, ttl 185, id 48684, offset 0, flags [none], proto TCP (6), length 40) m4e5e36d0.tmodns.net.9971 > erythrocyte.squiggleslash.link.www:
., cksum 0xcae5 (correct), 529:529(0) ack 1421 win 2190
The tcpdump is on Erythrocyte's pseudo-Ethernet device, though for what it's worth I get the same stuff on Endothelial's.
Now, some background:
- Erythrocyte and Endothelial are both virtualized servers running on the same physical hardware using Xen. squiggleslash.link is my internal DNS domain for IPv4 NAT addresses (10.x). Just saying. This doesn't involve IPv6 or anything else like that.
- The MTU on Erythrocyte's eth0 is 1500. Though for shits and giggles I did reduce it to 1460, but nothing changed.
- The MSS on Erythrocyte's default gateway set up was unset. Again, for shits and giggles I changed it to 1460. Nothing changed
- Erythrocyte seems, for the most part, to be having problems with Endothelial's requests to send smaller packets. Essentially, I'm tracing all of this because if I try to hit my website from an external address, such as Chloe (my N800) over T-Mobile's Internet connection, it takes ages to load pages, with many objects just not being transferred.
- This doesn't appear to be related to PPPoE. The pppoe driver is correctly notifying Erythrocyte that all packets to Chloe need to have a smaller MSS. And, in any case, I'd expect Endothelial to have a fit about 2880 byte packets regardless of whether the external connection is PPPoE or regular Ethernet.
I may be misreading the log, but it looks like somehow Erythrocyte is able to send a giant, 2880 byte, packet fragment to Endothelial, despite hardware (or virtualized hardware) that makes that impossible, and OS settings telling it specifically not to do that. Am I misreading it?