How ISPs May Quietly Kill VoIP

ravenII writes "PBS's i'Cringley's informative piece gives an eye-opening look at the anticompetitive behavior of some ISPs who are showing up late to the VoIP game. This is not something that could be easily mandated, and the beauty of this approach is that they're not explicitly doing anything to the 3rd party service applications. They're just identifying and tagging their own services, which is within their rights."

Re:Verizon

[Paska]

Read the damn article, this is different to the case you mention. They are not blocking access to other VoIP providers. However they are tagging (or will?) be tagging their own VoIP traffic which will force all other VoIP provider's traffic to run as non-guaranteed traffic and thus, could lead to dropouts or all round crap service.

Encryption is the simple answer...

[datastalker]

...and just as you can tunnel just about any traffic you want through port 443 assuming you know what you're doing, you can encrypt traffic between networks. Granted, that will make things more difficult at first, but it will allow people to get around things like this.

Packet shaping

[saskboy]

I for one welcome our VOIP packet shaping telco overlords.

But seriously, this has been a known threat for a while, at least it is a threat to every other P2P service on the Web. Universities routinely packet shape their networks, filtering out P2P filesharing programs, or giving them such a low priority it's as if you're using dial-up when using Kazaa lite.

www.theswitchboard.ca looks like the gold nugget in that article.

Re:Encryption is the simple answer...

[m0rningstar]

No, it won't. You just de-prioritise ANY traffic other than your VoIP traffic.

And without some form of prioritisation across a public network, VoIP becomes a flaky proposition at best. You have a 250ms round trip latency budget, and encryption adds to the serialisation delay on both ends and impacts this. Plus any out of order packet delivery or jitter will further impact voice quality, along with compression.

And people expect their phone to work. All the time. Early adopters will tolerate the impact, but the money is in the commoditisation of the service and deploying it to everyone -- and everyone will not be willing to deal with a flakey phone.

That makes it worse...

[katharsis83]

Actually, adding on another layer of encryption makes the problem worse. From the article summary:

"...the beauty of this approach is that they're not explicitly doing anything to the 3rd party service applications. They're just identifying and tagging their own services, which is within their rights."

The service providers are prioritizing THEIR VoIP traffic; so unless you can encrypt and then mask your VoIP service provider's packets to look like the ISP's, all encryption will do is increase the latency for voice - remember encryption/decryption requires time. The ISP doesn't explicitily delay Vonage's packets, for example, it simply upgrades the QoS priority of their own packets; this conveniently screws over 3rd-party providers like Vonage while not getting the ISP's in legal hot water.

Encryption can protect your 3rd party call from evesdropping, but can only increase latency under this new sneaky scheme.

Re:Verizon

[Anonymous Coward]

there is nothing to fear. you can always make voip traffic look like gamer traffic on any port. small, bursty packets send for long periods of time. what are they going to do? block gamers? yeah, that's smart marketing.

Re:Here Come the Commies...

[Nimrangul]

Yeah, you're right, I'm from Ontario. You prarie folk, especially Saskatchewan, are traditionally ahead of the rest of Canada on the political scale of things; though you often end up the butt of jokes for it (for being "way out there").

Hmmm, this reference to commies made me think of something that made me chuckle... In the past in America the Reds were the evil outsiders, now they are the good ol' boys back home.

Bah, strange things enter a man's mind at 11:30.

Re:Tag your own packets?

[The Vulture]

Well, you've got the idea correct, but it has nothing to do with how DSL can run aside POTS. DSL can run along POTS because one uses the low frequencies, and the other uses the high frequencies. If you have DSL on your phone line, and don't have the filter, you'll likely hear a hissing, that's the DSL signal.

However, should you get a combination cable modem/MTA (the VoIP box) from you cable operator (i.e. Comcast), it works like this:
* The DOCSIS 1.1 specification calls for a nifty little feature called "service flows". Service flows have their own QoS, and can be triggered by a variety of criteria, including TCP/UDP port numbers, Ethernet frame type, etc.
* From there, the cable operators will provision two (or more) service flows for the cable modem. One would be for the voice, which would receive the highest priority possible (but with a lower bandwidth), and the other one(s) would be best-effort, with a higher bandwidth allowed.

Cable operators can also use this to throttle any arbitrary connection (i.e. P2P), and in fact, have done so in the past, I would imagine.

A "side effect" of this would be that Vonage boxes would considered as best-effort, simply because they don't get classified into the voice flow by the software of the modem. This is because they won't meet the characteristics of the voice flow.

-- Joe

Re:This indeed disproves the myth of capitalism

[Anonymous Coward]

you're thinking of "chaos" not "anarchy"... do some reading.

From Cringely's 17 March Column

[jonbrewer]

On the same topic this week, Cringely speculates...

"there are other dirty tricks available to broadband ISPs. Telecom New Zealand, for example, is reportedly planning to alter TCP packet interleaving to discourage VoIP. By bunching all voice packets in the first half of each second, half a second of dead air would be added to every conversation, changing latency in a way that would drive grandmothers everywhere back to their old phone companies. This is because phone conversations happen effectively in real time and so are very sensitive to problems of latency. Where one-way video and audio can use buffering to overcome almost any interleaving issue, it is a deal-breaker for voice."

This has certainly pissed off a few Kiwis, as seen on the NZNOG list: http://list.waikato.ac.nz/pipermail/nznog/2005-Mar ch/thread.html/ [waikato.ac.nz]

but of course ...

[porky_pig_jr]

if ISP X has an agreement with ISP Y to pass the traffic through itself ('transit AS'), without any special considerations, it will do just that, as a best effort. Tagging? Of course, X will ignore any tags created by Y. X would be crazy to do otherwise. I used to work for ISP (which had AS 1. makes a good trivia question, eh?) so that's pretty much the rules of the game. This is incidently the main reason why QoS on the Internet (with a capital 'I') is practically non-existant. Since the backbone is privatized and fragmented, there is no real cooperation, only competition. I do what's optimal for my AS, and to hell with the global perspective (a 'hot potato' routing would make a good example). In such an environment I'm surprised VoIP works at all. In principle it ought to be less reliable than two tin cans connected with a wire. At least that wire is a point-to-point conneciton, not going through the hostile AS.

That's not how it works.

[billstewart]

Sure, a few ISPs may try to play games, but that's not the game they'll play - some third-world monopoly or recently-ex-monopoly telcos will block VOIP entirely, and some cable modem companies will do stupid things because they're incapable of not doing any stupid thing they can think of, but it's really few.

Here's how it works technically:

• Upstream bandwidth from customer to the ISP is often limited due to asymmetric technologies, and it's up to the customer's hardware to put the time-critical packets on the wire first and keep their upstream MTU sizes small enough (e.g. 1500 byte packets take too long at 128kbps.)
• Some ISPs have DSL concentrator networks that are oversubscribed upstream, but not many - the technology's symmetric, and consumer bandwidth is mostly downstream. VOIP takes up very low bandwidth - typically about 30kbps (8kbps G.729a plus RTP/UDP/IP headers). If you're not getting enough upstream bandwidth, you may need to buy a bigger pipe.
• Once you're past the feeder networks onto open backbone, there's plenty of room, at least on any telco-sized ISP (mom&pop ISPs may have congestion problems, but they're not the ones Cringely is accusing of being Evil.) Most of the Tier 1 providers use a lot of OC48s, especially telcos who own their own fiber plants - it's cheaper to waste bandwidth than to use lots of mux equipment to limit it, at least in the network core.
• Downstream feeder networks can be congested, but they're not usually that bad, and again, VOIP uses very little bandwidth.
• The big problem is dumping the traffic onto the recipient's egress line. If the recipient is trying to run BitTorrent and VOIP at the same time, without any QoS markings, their quality will suffer - but most people have fatter pipes downstream than upstream, and they'll just have to pause BitTorrent/ftp/etc. while talking unless their CPE is smart enough to throttle outbound requests.
• QoS markings can help prioritize that egress traffic, so the VOIP packets get to exit before data packets do. As ISPs add QoS to their available services, they'll obviously include it with any of their VOIP offers, and they might or might not charge extra for it as a separate offer.

Basically, if you're satisfied with VOIP quality now, it's not going to get worse as new technology gets deployed, except technology that encourages you to consume more bandwidth at home without buying a bigger pipe, and it might get better.

Re:Capitalism isn't a myth. It's gotten us this fa

[Anonymous Coward]

The main inventions that were required for television were made in Germany before 1900: Paul Nipkow invented the world's first electromechanical television system, the "Nipkow Disc". Karl Braun invented the cathode ray tube, which to this day is the basis of most televisions.

Bell patented the telephone after working with Antonio Meucci, an Italian who had invented the telephone 15 years earlier.

A light bulb very similar to that built by Edison was created by Joseph Wilson Swan in England one year before Edison made his "invention". Edison bought Swan's patent.

The first electronic computer was designed and built by German mathematician Konrad Zuse, in Germany.

The transistor was invented by three American physicists, John Bardeen, William B. Shockley, and Walter Brattain.

The pacemaker was invented by electrical engineer Wilson Greatbatch in America.

George de Mestral, the inventor of Velcro, is from Switzerland.

The (unpowered) airplane was invented by Sir George Cayley in England, or the German engineer Otto Lilienthal, depending on the achievement which you want to count as first flight. The first powered flight was performed by the Wright brothers in America.

Rockets were invented by the Chinese. The foundation of modern rocket science is attributed to Robert Hutchings Goddard in America.

The first working jet engine was constructed by Hans von Ohain in Germany.

RADAR was invented by Robert Watson-Watt in England.

One important reason why capitalism wins over communism is that it has less obvious failure modes. Capitalism's failures are generally attributed to "natural disasters" like wars while the failures of Communism are generally attributed to Communism.

When you can't switch to a competing ISP you may realize that capitalism isn't the same as free market economy and monopolies do leave you no choice.

Scare Tactic BS

[Anonymous Coward]

The author obviously does not know much regarding how the "Intarweb" works... Sure, Comcast can CoS all the packets they want, but as soon as they're off of their network it makes absolutely ZERO service improvement to their customer. I'm not sure about those engineers he's referring to but if this is what they informed him of they've been sleeping for the past 10 years on the basics of IP comm. If your service provider is giving you, oh, say 1Mbit down and 512Kbit up and *you* classify your traffic locally it gives you pretty much the same net effect as Comcast doing this one level up only because, more than likely, your VoIP call will travel beyond Comcasts autonomous network and, as previously stated, does you no more good. SO, unless there is a collaborative effort between huge ISP's who control the greater portion of backbones and major hicaps then "I, Cringely" is just being paid to stir up bull shit topics on /. and other geek attracting sites. Bounce this off of anyone with any decent networking background (CCNP, CCIE, M-Series, T-Series, E-Series, etc...) and they will tell you the exact same thing. I'd delve into more details regarding specifics but I think the majority of the /. populous get the point.

Anonymous CCIE

*Cringely's* not being fair - or accurate

[billstewart]

Yes, there are some ISPs that want to kill every application that generates upstream traffic, so that any consumer in their right mind would buy services from somebody else. And there are third-world telecom monopolies or ex-monopolies that use their position to strangle new competition. (In the US, the worst offenders are mostly cable TV companies, but even in the bad old days of Excite@Home banning anything server-like, they understood that the main reason people bought their service was to file-share pirated music. And in the Pacific Rim, unfortunately Australia's telecom industry has a third-world attitude toward data users.)

But fundamentally, the things Cringely's complaining about aren't accurate, because he doesn't understand the technology or the resulting economics. Yes, telcos are dealing with the threat of VOIP, and it's making their heads explode, and VOIP is much *much* harder to integrate with an old-fashioned telephone infrastructure than to run as a pure-VOIP business. (The technology's difficult, making it scale is difficult, different parts are centralized or decentralized, all the assumptions about who hands money to whom are different, the regulatory infrastructure doesn't match well at all, etc.) And the telcos are making sure that their data networks will support any VOIP services they develop with as close as they can get to traditional telco voice quality, and they're not sure how to deal with the fact that cellphones have convinced the public to accept lower-quality calls and newer codecs with much higher frequencies can support speakerphones much better.

Some big ISPs happen to be owned by telcos, or by telco-wannabees like the cable TV companies. Most of them are working on adding CoS capabilities to their backbones, but that's the least critical part of the network because most of them own their own fiber plants, and it's cheaper for them to burn more wavelengths on their fibers than to add fancy engineering capabilities to their routers or to hire fancy engineers to run them. It's the friendly mom&pop ISPs (that Cringely's not worried about) who are most likely to have backbone congestion issues that need CoS support to prioritize VOIP over best-effort data applications, because they're running at a different scale and don't generally own their own fiber networks.

The places that CoS matters most are the skinny parts of the network - the ingress from the customer's premises to the ISP's POP, and the egress from the ISP's POP to the customer's premises. The ingress direction is really a customer hardware and management problem, making sure that VOIP packets get on the wire before data packets, but service providers (including Vonage) typically handle that by forcing the customer's data through the same box that converts traditional-phone signals to VOIP, and software-based providers like Skype handle that inside the user's PC. This doesn't require the ISP to do anything, though it's sometimes cheaper to build those capabilities into the DSL/cable modem.

The egress direction can benefit from CoS marking, or from other fair-queuing systems that share bandwidth between remote sites or protocol types, or even from dumber systems that prioritize UDP over TCP. In a symmetric environment, like most business T1 connections, this is the most critical part of the system, because data applications can drown out voice unless there's some QoS approach. But most consumer connections are asymmetric, with much faster downstream connections than upstream, so there's less of a problem. Also, in most home applications, if downstream bandwidth is the bottleneck, it's usually because of some application like downloading music that can be turned off or slowed down during phone calls, which isn't a practical approach in most multi-person offices.

Cringely's arguments are especially bogus because the impact of backbone QoS / CoS features on network performance is much smaller than the impact of slow upstream connections in ADSL and Cable Modems. A 12

Re:Robert X. Cringely is a sick, sick man

[JudeanPeople'sFront]

He-he, actually "vulgar" comes from Bulgar and used to mean "common people", "citizen". The Vulgate Bible was written in the commonly spoken language, the vulgar language. Germanic languages have probably gotten "volk" and "folk" from this word, also.

There were mass settlements of Bulgars in the Apennines at various times trough the Middle Ages. There are a lot of Bulgarian toponyms in Italy [tribal.abv.bg], for example Monte Bulgheria [cilentonet.it]. Bulgarian derived family names are common in some places - Bulgari and Borgii are famous examples. The Latin language itself was "vulgarized" due to a large number of non-native speakers (mostly Germanic people, like the Langobards).

"Vulgar" acquired a negative meaning later: plain, plebeian, unrefined, coarse and rude. It is quite a common linguistic phenomenon. Actually, in "Genealogy of Morals" Nietzsche claims that the words for "bad" in all languages have evolved likewise.

Way off-topic, I know. Whatever.

911 is a mess anyway

[billstewart]

The current 911 and E911 systems in the US (for you non-US folks, that's Emergency calling or 999 or whatever) are designed with heavy dependence on a bunch of technical assumptions that weren't always valid for the traditional telco infrastructure and are less valid now. They didn't really like PBXs, and in some sense VOIP is like PBXs for everybody, and they certainly don't like mobile phones, though the control-freak FBI types have managed to bully the wireless companies into building location-tracking capabilities without making them actually useable to the owner of the phone. For instance, you can stick your Vonage or AT&T CallVantage phone in your suitcase and take it on a business trip with you - but if you call 911, you really want the fire engine to show up at your hotel or the Starbucks you're in, not back at your house when you're not there.

Cringely's speculations about providers tinkering with QoS are bogus - I've heard other clueless people ranting about how awful it is that some ISPs might start offering higher quality service for more money (the bastards!) At most, his arguments are really that some ISPs might fail to provide higher-quality service for people who don't pay extra for it, and that this might not be good enough quality in spite of the fact that many people like it today. And if that's the case, and their basic service isn't good enough, then either you're going to get a different ISP, or you're going to pay more for better service, or you're going to keep your old-fashioned phone, and of course, if you can afford broadband and VOIP service, you can also afford a cellphone (at least a pre-paid 7-11 phone for emergencies), which will even work when bad weather makes your cable modem go down.

Furthermore, Cringely focuses on QoS in the backbone, but the real impact isn't there, where the network's fat enough, but at the skinny edges. The ISPs have no control over your outbound traffic - what if you're trying to call 911 and somebody is downloading that music video file you're sharing? That has a much bigger impact on VOIP performance than anything a backbone provider is going to do. Or what happens if that music you're downloading starts getting better performance because the server is less busy - QoS could help that direction a bit, but if your ISP uses one standard for QoS markings and the 911 Center's ISP uses a different standard (there are lots), then the QoS isn't going to work the way they expected anyway.

Rogers Cap doesn't affect VOIP

[billstewart]

VOIP uses trivial amounts of bandwidth. I assume that's 60GB per month? So that's 2GB per day. Compressed voice uses 8kbps, so 1KB/sec; when you turn it into VOIP it's about 3KB/sec. 86400 seconds/day means you can leave your phone on for 24 hours a day using about 250MB/day - a mere eighth of your bandwidth.

While there are ISPs that are far worse about it - Telstra, for instance, or some of the US cable companies who think that they need to catch up with Telstra's unwillingness to let people actually use data for anything - a 2GB/day cap is still annoying. Basically, it means that you can't do file-sharing without being very selective in what you leave running for how long - so you can download that latest Knoppix release and share out a couple of copies, but you can't leave your entire set of Linux and *BSD distros open all month.

Re:It's going to be bad, in theory

[Pollardito]

there's two points that he made that are useful here :

1. they're not rate limiting VOIP traffic in particular. he should have said that all internet traffic is 11 and that all internet traffic would be limited to 10, so that it'd be clear that there is no discrimination. VOIP is one of the more latency sensitive applications, so it will suffer first and most if all non-preferred traffic is limited
2. they don't have to reduce anything to throttle, they can do it via inaction. ever increasing subscription to VOIP (and internet services in general) means that they only have to freeze bandwidth to effectively throttle future traffic