Google Caught in Comcast Traffic Filtering?

marcan writes "Comcast users are reporting 'connection reset' errors while loading Google. The problem seems to have been coming and going over the past few days, and often disappears only to return a few minutes later. Apparently the problem only affects some of Google's IPs and services. Analysis of the PCAP packet dumps reveals several injected fake RSTs, which are very similar to the ones seen coming from the Great Firewall of China [PDF]. Did Google somehow get caught up in one of Comcast's blacklists, or are the heuristics flagging Google as a file-sharer due to the heavy traffic?"

Not me...

I'm on Comcast, and haven't had any problems. Doesn't mean they're not doing it elsewhere, but they don't seem to be doing it here.

Re:Not me...

I'm on Comcast and I do notice some unusual "connection reset" errors every now and then. More than I would normally expect, at least. They happen when I'm trying to telnet/SSH into my Linux box from outside, when I try to download something on Steam, in fact during nearly anything that requires a connection to be established for any significant period of time. I never used to have this problem before Comcast assimilated my previous cable provider. Makes me wonder if it's deliberate.

Re:Not me...

I'm rather certain the root of your woes is Comcast. I am not certain it's intentional.

Furthermore, the problem is very likely far more simple and less sophisticated than this issue of packet spoofing.

Set up a continuous ping to something "nearby" (your gateway, your DNS ser ver, your neighbor, whatever) in your Comcast network and tee it to a file. Leave it up for days and you'll likely see periods of time where you have no service for patches of time... often long enough to kill sessions.

I very often have problems with any sort of sessions (SSH, VPN, etc.) staying up for long periods of time because the underlying line level reliability is so poor. I can watch my cable modem logs and see many resets, timeouts, etc.

I laugh whenever asked about phone service via Comcast. Sadly, however, this pathetic reliability also precludes Vonage and the like. And I find this a bit sad since while I do not consider Comcast capable of running a world class network, I loathe the phone company. Those guys are more competent but much more directly evil.

Re:Not me...

choke on it... it IS comcast. Your intermittent problems keeping a session open are inarguably unacceptable in view of the wider experience of broadband users in North America. My provider is rock solid in my area. I regularly keep open as many as 6 sessions that do not see lost packets, never mind service unavailable. for example: active SL connection(s), Vonage call, Internet Radio, NNTP session, and active web browsing. None of these suffer a problem. In fact, the only problems I've had were / are on the wireless links. My microwave and wireless router apparently disagree on the topic of which is more powerful.

If we look at what is promised, what is purchased, what is possible, and compare that to what is experienced, it is clear that some ISPs suck, and there is a reason that they suck. Suckiness is not 'normal' or 'average' or acceptable. With the FCC ruling to allow multiple ISP connectivity to many homes, the quality of service should improve to prevent customer churn. My advice is to switch if complaints are not resolved if you can. If not, register a complaint with the authority who gave your ISP broadband monopoly in your area. Document the complaint process and responses. The BBB, I believe, can be consulted in cases where they clearly are not giving you what you paid for.

Re:Not me...

But in this case it just sounds like they can't figure out how to do it right.

It's not that they can't figure it out, it's that they aren't even bothering to try and shape traffic. They'd rather interfere with it.

Back in my ISP days we ran our entire operation (400 dial-in lines and about 60 WISP clients) off two un-bonded T-1s (they went to different POPs for redundancy). We couldn't afford to add more bandwidth at the edge, so I hacked together a traffic shaping setup using Linux. It prioritized ssh, telnet, TCP ACKs, icmp packets, and the VPNs of our business clients. VoIP wasn't a big concern in those days but had it been I would have prioritized it as well. When online gaming started becoming big we started giving that traffic priority over bulk transfers as well.

The bulk downloaders/p2p'ers didn't notice or complain. They still got the lions share of the bandwidth -- and are you really going to notice if your transfer gets 139KB/s instead of 140KB/s due to that ssh packet moving ahead of you in the queue? During peak hours my T-1s were running at 90-95% of capacity but my users were all still humming along quite nicely, none the wiser. There was more to this then just traffic shaping (we also had a pretty slick squid setup), but the point is we got along just fine with our limited resources.

If we could fucking do it, then sure as hell Comcast could. They have apparently decided that it's better to block/drop the traffic then shape it. If they had real competition they'd probably pay for this over the long run.

Re:Not me...

Nah, the basic problem is that the bigger the company, the higher the density of PHBs. Once you get to a certain concentration, you hit stupidity critical mass. From the outside it looks like malice, but it's really just highly focused incompetence.

Re:Not me...

Oh, if that were really the only problem.

There are two kinds of big mistakes you can make: those that are big for a company your size, and those that are just plain big. In a big company with lots of customers, small mistakes are multiplied by volume into just plain big mistakes. If you've got gross revenues of a million dollars, a mistake with a potential $100,000 impact is big for your business, but not that big. You can survive it, you can reestablish credibility with your customers (whom you know face to face) by personally eating a helping of crow in front of each and every one. If you're in a company a 100x as big, you're talking maybe a $10M impact that if laid to the account of any individual employee is a disaster beyond that individual's ability to make right.

That's why large companies can develop a special kind of stupidity, preferring a status quo that is certainly wrong to any alternative that is only probably right. Individuals protect themselves using exactly the same strategy that schooling fish employ. Any decision has to have so many fingerprints on it that firing the people who can be tied to a mistake is like cutting off your right arm. That's why big defense contractors are probably the most bureaucratic organizations on the planet. Ordinary mortals have to make decisions that can have impacts measured in hundreds of millions of dollars. In any such situation, you obviously need a form of collective responsibility, the question is what form it takes. It's all to easy to develop an organization that protects individuals by being unable to detect and respond to most problems. We didn't know about it, if we had we probably couldn't do anything about it, and if we could have, it wasn't my job.

The problem is not that a typical PHB is necessarily stupid. The problem is that organizations are built in a way that rewards people for acting in a stupid way. But stupidity is all too common. Even stupid people can manage to be cunning in bad organizations, because they are problems in an organization built around willful blindness to problems. It's more of a challenge for intelligent people I suppose, because it's hard for people with imagination to find much satisfaction in what it takes to get ahead in these places. It has even been suggested that sociopaths make good managers, which I doubt. But I can well believe that feigned stupidity is better in some cases than the real thing.
 

Re:Not me...

Thanks for adding anecdotal noise to the discussion that adds absolutely nothing to the discussion.

Gee, I think that anecdotal evidence is interesting, especially if you're interested in understanding what rules Comcast uses to decide which packets to block. Questions like: "Is it the whole network or just portions (I suspect just portions)?" or "Is it all the time or during peak demand?" Please try to be civil. If a comment isn't valuable, it won't be modded up. If it is valuable it will.

Get the facts

70% of all "file sharers" use Google. Anyone with even a small background in statistics can see that Google is behind all this piracy. Comcast is simply watching out for our economy. I say good for them. Now if they would only do something about that wretched Slashdot and its wanker community.

Re:Get the facts

-1, Troll? This should have been modded funny. Or ignored. Or overated if it bothers you that much. But troll? I hope you pay in meta-mod.....

Re:Get the facts

Lately?

I have noticed this stuff happening for over a year or more. Of course I speak my mind on a lot of issues that goes against the grain. For instance, stuff like the domestic spying- I usually point out that it is far from domestic which get troll, flame bait, and overrated modifiers all the time. It has been a situation for a while now and I have a working theory on it.

The theory goes something like this. When we started seeing the politics sections appear (that was supposed to be temporary but stayed forever) I started seeing political motivated posts that were basically rehashes of some party line talking point getting moderated insightful while common sense posts about the topic in hand was being modded off topic, under rated or some other negetive moderation. I began watching and it appear that either an organized group or groups of people have signed up in order to press a particular view or the sites own administration is doing it to some extent. Judging by the constant links to political sites like media matters and moveon.org by posters themselves, I'm starting to think it is a group of ideolgs doing it.

Of course I can prove anything other then by saying it is my personal observations. But if you start looking at it in this light, you will likely see the trend happening too. Of course to what degree will probably depend on your political bias. But you should definitely see a pattern rising that will worsen coming to a major election time.

Google *is* the file-sharer

After all, doesn't Google host more copyrighted content than any other person/company in the world? ;)

Gmail Notifier

Starting yesterday my Gmail Notifier Firefox extension stopped working at home where we have Comcast, but at work it works just fine. I thought maybe the plugin had broken due to some API changes or something but I thought it was odd it worked one place and not the other. This really seems like it's related and even though I believe Gmail Notifier is a third party extension, it's still accessing Google's servers.

Comcast is really pissing me off. But what's my other option: Qwest DSL.

Re:Gmail Notifier

Comcast is really pissing me off. But what's my other option: Qwest DSL.

Thankfully, I had RCN as an option. I pay them $20 extra per month for a static IP and run my home Web server and mail gateway there. I've never had a problem downloading Ubuntu or Fedora distributions with BitTorrent; Web traffic incoming or outgoing; or... well, anything.

Call your city. Ask them to re-evaluate Comcast as the local Cable provider or do what my town did: offer RCN as a competing provider.

I hope they get slapped

Hard. Nothing worse than a pissed off multi-billion dollar company suing your ass off. That will teach them.

unfair competition

Is the title clear enough? I can't imagine any judge or jury saying Comcast is allowed to impersonate Google and tell Comcast customers they're not allowed to use Google's services or that Google's services are overwhelmed and shutting down connections. That's essentially what forged, fraudulent RST packets from a MITM attack are doing. That can't possibly be considered a legitimate business practice in court.

Re:unfair competition

That's essentially what forged, fraudulent RST packets from a MITM attack are doing

I fail to see how they think these types of "traffic management" tools will work in the long run. It's only going to encourage the P2P users to adopt more protocol masking/encryption techniques to hide from these devices. And then what are you left with? Blocking encrypted traffic? Breaking the internet by refusing to route packets directly between end-users and only routing them to major sites?

In a fair world with a fair marketplace they'd have two options. They could choose either one and the market would decide which was best: 1) Stop selling unlimited service and switch to a metered model. 2) Upgrade their friggen network to support it.

Re:unfair competition

I'm still not convinced the bandwidth is Comcast's major concern. Comcast still makes the majority of their money from being a cable company, and only uses Internet access as a diversification method, don't they? All the Comcast commercials I see are for cable TV, not for Internet access.

It seems to me the whole rage against P2P traffic (which is how lots of games are played, BTW, and how almost all VPNs are set up) is not so much about capacity as about a conflict of interests on the part of Comcast. They're the content delivery network for TV programming and music (they have music channels like DirecTV does, don't they?). They are wanting to make sure you use your cable TV for getting video and audio, because that's where they get a bigger cut.

Re:unfair competition

That's an interesting take on it. And as far as I'm aware there is no DSL provider in the United States doing anything like this. It certainly seems to be the case in the wireless world. The carriers removing or blocking features that may compete with their own content offerings.

One wonders what the solution to this is. Prohibit someone from being in the content business AND the delivery business at the same time? They'd fight you tooth and nail on that -- and you'd have the "free market" types after you as well.

In any case I think they will shoot themselves in the foot in the long run. What happens when all P2P traffic is encrypted and looks like any other encrypted protocol (ssh, ssl, etc)? At that point you may be able to identify WHICH subscriber is using p2p (bittorrent stands out like a sore thumb for the sheer volume of connections it establishes) but how will you identify which individual packet is p2p and shape it? Or will they just start sending random RST packets to ALL your connections, including (as TFA suggests) Google?

If bandwidth IS the issue then in the long run they only have two options. Invest in some upgrades or stop selling "unlimited" service. Personally I'd take the best of both worlds. I'd offer a "premium" package aimed at p2p users (no monthly bandwidth limit and/or higher speeds) and use the money from that to expand my network.

Would be kind of awesome...

If Google were being wrongly flagged, and Google ends up suing the ass off Comcast to put an end to this bullshit.

Push it one step further...

What if Google, a (justifiably) huge advocate of network neutrality, is deliberately sending the type of RST packets that imitate Comcast's faked packets, specifically to Comcast IP addresses, knowing the inevitable fallout that would result? It would make an already bad situation for Comcast far, far worse, and it's likely that the requested Senate investigation would turn into nails in the coffin for those who want preferential treatment of packets on the Internet.

For a company that does no evil, if they could pull it off, it would be absolutely diabolical. But then, it could easily be one of those "ends justify the means" kinds of situations. At any rate, all I can say is "MWAH HAH HAH HAH HAH!!!! Suckers!"

(No, I don't actually believe that's what's happening, but man, what an AWESOME plan to make network neutrality happen once and for all.)

iptables fake RST detector

use connection tracking on this one:

iptables -I INPUT -j LOG -p tcp -m tcp --tcp-flags RST RST -m conntrack --ctstate NEW,INVALID

The fake RST will probably not have a valid sequence number for the established TCP connection, so the Linux stack will flag it as a NEW connection, and the fact that you're getting a RST for a NEW connection should be good enough alarm.

Or maybe it would also work with just the matching code

iptables -I INPUT -j LOG -p tcp -m tcp --tcp-flags RST RST -m state --state NEW,INVALID

What do y'all think?

Go even further and ignore fake RST?

This looks like it could be extended - add a -j DROP rule after the -j LOG (log the offending packet, and then send it to the bit bucket).

Re:iptables fake RST detector

The problems with a fake RST detector are two-fold. The RST bits are being set on TCP traffic sent in both directions on a connection, so even if you ignore RST teardowns, the other side will tear down the connection. What Sandvine boxes do is just flip the RST bits on TCP packets flowing through them, so the sequence numbers will appear correct in the connection tracking table because the TCP packet is a valid one from the other side of the connection.

If Comcast truly is using Sandvine boxes, then this could be a network controller station with the preset examples still in place. The Sandvine sales presentation shows how to load up the system with all the prefixes from AS36561, and then interfere with a tiny percentage of TCP traffic after the first few hundred packets are transferred. What this does is provide a way of denying they are completely blocking those packets, but will blow away any connection hoping to do streaming video or cruise around on a web page heavy in graphic content like a mapping function.

The business model after installing Sandvine boxes is to then extort regular payments from large content providers to allow access to their network. Comcast, SBC/ATT and a few other monopolistic ISPs would like to see both sides of a connection pay for traffic in both directions, not the current economic model where each side pays for their own access or transit.

What Sandvine boxes do is break the end-to-end model of the internet. Even a tiny percentage of broken connections will put an end to all the cool applications everyone is currently enjoying. Streaming video and audio sessions, VoIP calls, file downloads, p2p exchanges, search engines, mapping and geolocation, and heavy web content sessions like social networking sites. The only traffic that can survive this kind of interference are from applications that make repeated attempts at connection in case of unexpected interruptions, like SMTP.

P2P protocol designers are pretty agile and clever. In the face of regular faked TCP RST bits on a connection, they'll evolve the protocol to make shorter connections, and to make repeated attempts to reconnect when an unexpected RST is received. Expect tuning "knobs" in clients very soon now, on how resilient to make the connections or how many bytes to transfer before tearing down and rebuilding the connection. There could also be a way to limit the numbers of attempted connections so as to fly under the radar of systems like this. I can open any bittorrent client with a single popular file, and see over 1000 completed TCP connections within 2 to 3 minutes. Limiting the number of new connections per minute could throw a spanner in Sandvine's current design.

the AC

Google could fix Comcast's ass tout suite

When loading a Google Page, an intermediate page pops up saying

"Your ISP is interfering with the transmission of data requested from Google our users, and as a result we are unable to consistently provide advanced services to you. You will be redirected to a more basic version of Google's services so that we can provide as much as we can in the manner you have come to expect from us".

Wait 10 seconds, then redirect to Google's non-AJAX pages.

I predict hordes with torches and pitchforks (led by a little old lady with a claw hammer)

Re:Google could fix Comcast's ass tout suite

It would be great if they also provided links to various federal and state fraud statutes...

And links to your state's AG office...

And little adwords ads on the side for local law firms.

going on for months with google maps

I have been unable to use Google maps for months now on Comcast. I have called them, but, you can guess how that went. Yahoo maps and Mapquest work fine, but on Google I get about half the tiles filled in before it stops. And I mean it stops. It ends up looking like a checkerboard. Occassionally it will finish a couple of minutes later, but typically it never does.

Getting Comcast to fix it seems unlikely.

time for IPSec?

IPSec would thwart this sort of attack (since it encrypts at the IP layer, you can't forge a RST packet in the TCP header). Yeah, it costs more CPU, but that's not a problem for modern PC clients, and I suspect Google can handle it, too. Is it time for this to become SOP?

Now, whether MS would be cooperative in that, I dunno... I know XP supports it, but not too much about configuration specifics.

Comcast shenaigans

I recently moved from one house serviced by comcast to another and I can tell you there is DEFINTELY something screwy going on, and it's not just bittorrent trafic.

I've done bandwidth tests and my upstream STARTS at a nice 1.5MB/s and then 15 seconds later drops to 30K/s EVERY TIME.

What this does is give false results when people are doing speed tests. When you do your test you get great results (in my case 15Mb/s downstream and almost 2Mb/s upstream) for the first 15 or 20 seconds. Then after that it just BLOWS.

Wikipedia page

Someone knowledgeable about this issue should update the wikipedia page about sandvine. [wikipedia.org]

The way it's written now, everyone should use Sandvine - it sounds like wonderful software.

applications for testing ISPs?

There's a lot of guesswork here about what providers may or may not be doing; are there any applications for actually testing ISPs? Such testing apps would discover traffic shaping, port filtering, connectivity, and other traffic modifications by the ISP. Something like a bandwidth tester on steroids.

When Google calls Comcast

*Comcast phone ringing at head office*

Comcast Secretary: Hello, thank you for calling Com-

Google Big Cheese: This is Google Inc. calling, I want to talk to whoever's in charge. Now.

Comcast Secretary: I don't know who you think you are but-

Google: Go visit google.com right now.

*secretary visits google.com, google recognizes the comcast head office IP range and serves up a pdf of a lawsuit document (Comcast as defendant) instead of the google homepage*

Secretary: Oh my, one moment please I'll transfer you.

Comcast Big Boss: What? I'm busy lining my socks with money and throwing darts at cust