Ok, I got a little dramatic in the title of the post. Body is accurate though, I really did get a lot out of the discussion here. People thought it was just a load balancer, and everyone was very wary of yet another subscription. It was real feedback that helped explain a lot about what was going on with the campaign.

Absolutely, that was the main scenario we were thinking about as we designed this. It does a nice job of putting links together so you really can a skype video call from the hotel. Since then we've been amazed to find out how many other scenarios there are: people with 2 DSL lines to the houses, business disaster recovery scenarios, people needing to get video streams into or out of areas where they can only get 3G service... the list keeps growing. But yes, the business traveler was who was in our mind at the start.

If we get a packet but haven't yet delivered the previous one in the sequence, we hold onto it until we either get the missing packet, so we can deliver what we have in order, or until we've seen a packet come in on all interfaces, and can declare the missing packet as lost, and then deliver everything we have.

Oh, I see what you're thinking. No, we don't make a mess of anything.

Look at our typical scenario: two ISPs, let's say 7Mbps DSL and 10Mbps cable and latencies that differ by, let's say, 20ms, with careful reordering you can see single-socket performance of about .95*(10+7) with average packet latency somewhere between the two. Without some careful reordering, TCP is very good at slowing way, way down, and a lot of UDP media streamers just fall apart completely.

If you're us, and you get a packet but you haven't yet delivered the previous one in the sequence, the right thing to do is to hold onto it until you either get the missing packet, so you can deliver what you have in order, or until you've seen a packet come in on all interfaces, and you can declare the missing packet as lost, and then deliver everything you have.

As far as SACK and D-SACK, you don't really want to do that for the 30% of your packets that arrive out of order. From what I've seen in the real-world, those RFCs are not intended for coalescing streams where potentially a lot of the packets are out-of-order (as they would be in the DSL + cable example).

Thanks again for your interest.

Yes, and this isn't load balancing. Load balancing puts each socket on one internet connection where it stays for "life". We divide the traffic between the multiple connections at a packet by packet basis. Which means that unlike load balanced solutions, we can even speed up single sockets like streaming video, uploads and VPNs.

MPTCP is really interesting, but if you want to combine connections to speed up a connection to a server that just supports regular TCP (Netflix, or Box, Dropbox, etc), even having MPTCP to somewhere else in the cloud, like a VPN server or connection aggregation server, doesn't get you very much. Running TCP over TCP in the real world is generally a bad idea, for reasons laid out pretty well here: http://vpnhaus.ncp-e.com/2011/06/30/sstp-the-problem-with-tcp-over-tcp-part-2/(cutting

Of course there are buffers everywhere. We have buffers. Bufferbloat != "having some buffers". Bufferbloat is "when excess buffering of packets inside the network causes high latency and jitter, as well as reducing the overall network throughput", As always Wikipedia is good place to start http://en.wikipedia.org/wiki/Bufferbloat

"It's worse than you know." "It usually is."

The latency situation is much more complicated than you describe. You said that we were adding latency to the highest-latency connection. But that's not right, for software that routes every packet optimally: we're looking at a latency of (high-latency connection - ((highest latency connection - lowest latency connection) / 2)).

But it's more complicated than that too, because Internet connections' latencies are not constants.

We don't add to buffer bloat, we go war against it. Buffer bloat is the enemy of Switchboard. Switchboard wants to know where every packet is, and devices that claim to transmit but instead stick them in a buffer are problems. Switchboard is smart enough to sniff this behavior out, and give less packets to such devices, which hopefully leads to smaller queues, and better behavior, and lower latencies. Which is why in many cases Switchboard helps with latency... a typical 4G card will have 25 ms latency when you ping on it, but when you start a file transfer it will soar north of 800 ms. Switchboard will see this change and start routing you around that backlog in real time.

You hit the nail on the head. Dealing with latency differences has probably been the hardest part of all this. We identify packets that are more latency sensitive, and route them over the lowest latency link.... doing this properly effectively divides the difference in latency between the links in half. We also take the QoS headers from the VoIP traffic and use them on our packets when sending them across the net. If your networks do something special for VoIP traffic, they will do the same for our traffic, when we're actually carrying VoIP.

Yes, out of order packets were a real problem, caused by the difference in latency between paths. We get them back in a semblance of order coming out of our network connection.

We rolled our own so we could overcome SCTP's problems with NAT traversal (which only seem partially fixed by trying to tunnel SCTP over UDP) and so we could do things like track the flight time of every packet

Oh, no, there's nothing like that. But you asked if I "guarantee privacy". I read slashdot daily, so I know that if Hide My Ass has to comply with UK judges and FBI warrants, then we will too: http://yro.slashdot.org/story/11/09/25/0415213/hidemyasscom-doesnt-hide-logs-from-the-fbi

No problem, thanks for the question anyway.

Ah, thanks for the question. (sorry I gave this answer to someone else too)
MPTCP is really interesting, but if you want to combine connections to speed up a connection to a server that just supports regular TCP (Netflix, or Box, Dropbox, etc), even having MPTCP to somewhere else in the cloud, like a VPN server or connection aggregation server, doesn't get you very much. Running TCP over TCP in the real world is generally a bad idea, for reasons laid out pretty well here: http://vpnhaus.ncp-e.com/2011/06/30/sstp-the-problem-with-tcp-over-tcp-part-2/
That's why Switchboard uses UDP as much as it can, and defaults to TCP when it absolutely has to, similar to what OpenVPN does.