Become a fan of Slashdot on Facebook


Forgot your password?

Comment: Private Links != Paid Priority (Score 0, Flamebait) 258

by Drakonblayde (#48391525) Attached to: Comcast Kisses-Up To Obama, Publicly Agrees On Net Neutrality

Full Disclosure: I am a network ops engineer for Comcast.

Anyone who believes that buying private links into a providers network is the same as your traffic getting paid priority knows jack shit about network ops. In the case of Comcast, Netflix traffic gets no special priority once it's on the internal network. The direct links simply lets them bypass the naturally occurring bottlenecks that occur at internet peering points.

Now I'm sure a bunch of people (who are not network engineers) are going to argue over the wording and philosophy as to whether or not buying paid links into a providers network constitutes priority or not. It's not. In network operations, priority is a very specific concept. It means that you treat one class of traffic better than others, usually to the detriment of other classes of traffic. As an example, e911 voice traffic has the *highest* priority on the Comcast network.

Comcast does not treat Netflix traffic any better than anyone else's traffic. Nor is it treated any worse. It is forwarded as Best Effort within the Comcast network.

The only difference that buying direct links in meant was that they got to skip the congestion in the peering points. Comcast has alot more bandwidth internally and once traffic makes it into the network, congestion is not usually a problem (things do break, redundant links become saturated, etc. It's a big network, but in normal operation mode, congestion doesn't exist). What little prioritization we do has alot more to do with latency than with congestion (ie, your phone call is more important than your massive porn transfer, since voice is alot more sensitive to delay than bulk data transfer).

Comment: I think the part that scares me.... (Score 2) 149

by Drakonblayde (#48142591) Attached to: ISPs Violating Net Neutrality To Block Encryption

Is that techdirt did virtually no research on the issue, they just passed along what Golden Frog said in their filing.

Which brings me to the *really* scary part.

A company which provides VPN service should reasonably expect to have a clue when it comes to network operations.

Not only did this company not have the chops to figure out that 'someone may have incorrectly configured a firewall!', oh no. They decided to compound their inadequacy by including it in a filing to the god damn FCC.

So many levels of failure involved in this.

Comment: Re: Alternative explanation (Score 1) 398

Actually, when Sub2 joins, his system can start buffering the show 10-minutes in, and while doing so, send a request to resend the first 10 minutes of the video. Netflix servers save bandwidth, and both users watch the video they wanted, when they wanted.

That's not how Multicast works. The source has absolutely no idea who the members of the stream are. Only the edge routers know that. IGMP joins do not get forwarded up the multicast tree, only PIM joins. There would have to be some additional intelligence built into the app that would tell the stream to start adding additional data for the buffered client to the stream. And then you *still* have the problem that all that additional buffering data? Gets delivered to *EVERY SINGLE MEMBER OF THE STREAM*. So you end up saturating and transiting data that not everyone needs. This is a large part of the reason why no one uses PIM Dense mode, the way it works, alot of unnecessary traffic gets flooded.

So sure you could have the late joining client join another stream to get the first 10 minutes they missed, staying in the old stream while they buffer so that when they're caught up they can just drop the catchup stream and stay in the original multicast. Or you could unicast the first 10 minutes to them.

Or, you could just unicast the entire damn thing, which is alot less complex operationally, and doesn't require your upstream providers and their customers to essentially reconfigure their entire networks.

Now, that's just the engineering side. Let's look at the business side. Doing that would require effort from software developers, network engineers, and network operators. All to improve Netflix's experience. For the providers that Netflix's customer base connects to, there's no incentive, especially since they likely already have complex multicast implemetations of their own (which will usually be the case for any provider that hosts video). They will expend alot of man hours, possibly some opex and capex for absolutely no return. If they're publicly traded, this will make shareholders unhappy.

For the transit providers, they have even less incentive to make this go. If it did decrease Netflix's bandwidth usage, it would directly effect their revenue, unless Netflix was stupid enough to agree to pay the same amount of money for less bandwidth used.

Comment: Re:Alternative explanation (Score 4, Insightful) 398

I suspect that whats going on is that Netflix put the majority of their traffic on Level3 and Level3 is trying to charge Verizon an exorbitant rate for enough bandwidth to handle that peer. Verizon said "No" and told Netflix to go with another peer. So Verizon has plenty of bandwidth, Netflix has plenty of bandwidth... it's where those peers are located that's the problem.

Ok, but you're wrong.

Level3 has admitted they have settlement free peering with Verizon. Level3 does not pay Verizon anything. Verizon does not pay Level3 anything.

Netflix pays Level3. This is why Level3 gives a shit about this situation.

What's going on is that Verizon is trying to cut out the middleman. Verizon wants Netflix to pay them to get traffic into their network instead of paying Level3 to deliver traffic into the Verizon network. Why? Because they don't make any money from Level3.

Naturally, Level3 is all in a huff about Verizon trying to fuck with their revenue stream.

Comment: Re: Alternative explanation (Score 2) 398

There are at least three underlying problems for the congestion issue - one is the DMCA and related copyright laws that prevent any sort of sane caching, the general fear of multicast that everybody on the Internet still seems to have (half a million unicast streams of the same show is insane - where are the global warming people on this?), and the grants of monopolies and/or prohibitions on competition that prevent local competition.

I agree with you that half a million unicast streams can be nuts, when it comes to on demand content, multicast is a non-starter (we'll ignore the fact that multicast is sorely lacking in security features and would require some serious re-engineering on many networks to work... there's a very big reason why inter-domain multicast routing is not seriously employed, and it has nothing to do with fear).

Think about how this traffic flows -

Sub1 wants show A and starts playing it on Netflix.

10 minutes later, Sub2 wants show A as well.

What happens if this stream is multicast? Well, Sub2 gets the show 10 minutes later, assuming he's joining the same multicast group as Sub1. Sub2 is not happy, he wanted to watch the entire show.

How to get around this? Well, ok, so Netflix could just mux the feed for Sub2 inside the feed for Sub1, and presumably, the client would be able to tell which parts of the feed were for which sub. However, the problem is that the data for Sub2 would still be delivered to sub1, sub1 would just throw it away and pay attention to it's own data. However the data for both feeds have to transit the same backbone, which drives capacity usage up. This is also unsustainable, as eventually, as more subscribers joined, the feed would grow so large that it would saturate the downstream of all the subscribers receiving it and eventually lead to packet loss.... which would lead to loss of video, stuttering, etc. All the same shit thats going on now.

Ok, so muxing different streams into the same feed for the same show isn't going to work.

So Sub2 could just start getting the feed over a different multicast group, that would solve the ever growing feed problem!

Except that if you do that, there is functionally no difference between sending the traffic to a multicast destination and a unicast destination.

So sure, while sending half a billion unicast streams seems insane, especially since alot of those will be watching the same show, the fact is that a very small percentage are going to be watching the same show at the exact point in the show. For the most part, they will all be at different parts. Multicast is a solution when the data for the given event is live, or when it's linear (ie, this is the point we're at, and there's no going back). On Demand services does not fit either of those profiles.

Multicast has it's place in the on demand world, but only on the backend. It's wonderful for things like distributing a new asset to all of the streaming sources, or for filling caching servers that will the streaming boxes will need to pull from, but multicast is simply not a workable or superior implementation when it comes to delivery of content to subscribers.

Comment: Re: Alternative explanation (Score 1) 398

You do not do networking for a living very well then.

The problem isn't how Verizon is egressing traffic to Netflix, it's in how Verizon is allowing traffic from Netflix to ingress to their network. Once the Netflix traffic makes it into their network, I have no doubt it is being delivered over the most efficient route.... which means jack shit when the ingress point is such a tight bottleneck. Inbound traffic engineering is a very different animal from outbound traffic engineering.

You're not incorrect in that the fault lies with Verizon, but I seriously doubt you actually understand why or how.

Comment: Re:Could be a different route involved for the VPN (Score 2, Informative) 398

You do not understand how BGP works.

The problem isn't the data that verizon is sending to netflix, it's the data netflix is sending to verizon. Verizon messing with routing policy on netflix's announced prefix's wouldn't have an effect on verizon's streaming speeds. The traffic flows from Netflix to Verizon.

Therefore, in order to influence streaming speeds, Verizon would have to change their routing policy on how they announce their own routes in order to influence which links netflix traffic can come in on. The problem is, there's no way within BGP for Verizon to say I want Netflix traffic to only come in over these specific links. It would influence *all* traffic from that peer. Routing policy is destination based, not source based, and not source-destination based. By simply announcing the routes are more preferable over Level 3 saturated links, that forces traffic Level3 delivers for those prefixes to come in over those links.

Sure, Level3 could do some traffic engineering of their own and ignore or mutate some parts of Verizon's route announcements, and force that traffic in over unsaturated links Verizon may have with Level 3 (if there are any), but Level3 is a middleman. Doing so would take them out of their middleman status and put them firmly on Netflix's side. Verizon's likely response would be to immediately de-peer Level3.

The only folks who can effectively change how the traffic reaches Verizon's network is Netflix. They determine their outbound routing policy, but only up until their own border. Once it transits to another AS, it will be forwarded according to the upstream AS's routing policy. If Netflix wants to avoid saturated Verizon-L3 interconnects, the only thing they can do is not send traffic to Level3 for Verizon prefixes. They could easily modify their inbound route policy to send traffic for Verizon's prefixes via another peer. This is something that Level3 does not want, because it effects their revenue, hence their seeming to take sides with Netflix on the matter. It's one thing for Level3 to have an opinion on what Verizon is doing, that doesn't really effect operations. The second you change operations to try and force that opinion, well, you're likely to invoke the Law of Unintended Consequences.

Now, for whatever reason, Netflix has decided to go ahead and keep sending Verizon traffic to Level3. The reality is that if Verizon has decided to be douchebags about this, then they can do the same thing for whatever peer the traffic is ingressing through. Maybe all of Netflix's other peers ultimately transit to Verizon via Level3 anyway, which would make any change of forwarding policy moot.

About the only way for Netflix to solve this is to go ahead and cut out the middleman and just pay Verizon directly for interconnects into their network. This is what Verizon wants: another revenue source for traffic their going to deliver anyway. This is what Level 3 does not want: When you cut out the middleman, the middleman makes no money.

Netflix has already done it with Comcast and AT&T, so it's not surprising that Verizon wants in on this action as well, and will continue to be douchebags about it.

In the meantime, savvy customers can come up with their own solutions in order to avoid having netflix traffic destined for them coming in over saturated links. VPN and tunneling are two perfectly valid solutions.

Comment: Re:Advantages? (Score 1) 146

by Drakonblayde (#47534141) Attached to: Comcast Carrying 1Tbit/s of IPv6 Internet Traffic

NAT provides "security" because it is actually impossible to hack a computer behind a NATing router, without A) hacking into the router (in which case a firewall doesnt matter), or B) having the end user poke a hole / port forward through the NAT (which they could do with a firewall).

Oh that's not true at all. Just because you can't access the IP assigned directly to the computer from outside of the NAT hardly means you can't communicate with the device. NAT doesn't imply packet inspection or anything of that nature.

Far too many people confuse NAT as a security product because it's often paired with limited firewall capability in consumer grade routers. If you can do sufficient packet analysis (like, say, compromising an outside host that the NAT'd box talks to), you can communicate with the device. Malicious packets are translated inward just fine.

If the end user of a NAT box does something stupid, like install software that (innocently or not) leaves the host exposed behind the NAT, the NAT might as well not be there.

NAT is an ugly hack for address conservation, nothing more. Those who trumpet it as a security benefit are severely lacking in their TCP/IP skillset.

Comment: Re:TFA is a lie (Score 1) 337

by Drakonblayde (#47210555) Attached to: Cisco Opposes Net Neutrality

I got here late, but TFA is a lie. Stating the obvious (voice and HTTP are not "equal" to the client nor provider), doesn't make an official Cisco stance against Net Neutrality. In fact, most Net Neutrality proposals (every one I've seen officially submitted in Congress), would have allowed for such action. No Net Neutrality has yet prevented reasonable traffic grooming. It's designed to prevent Comcast from running a VoIP service with premium QoS and deliberately lowering the QoS of all other competing services. To keep all competing services at the same level is "neutral".

Net Neutrality is not "traffic neutral" It's "provider neutral" at least so far in every bill I've read. And that's the best way. Why force every packet to be the same when we know they are inherently not?

I wish I could mod you up. You have a proper understanding of what net neutrality is about, rather than what it's been perverted into.

Comment: Re:Title is a bit sensationalist... (Score 1) 337

by Drakonblayde (#47210543) Attached to: Cisco Opposes Net Neutrality

There is a difference between intra-domain and inter-domain prioritization and the operational futility of the latter.

inter-domain prioritization is hardly futile. ISP's don't own the entire world, nor is the entire world directly connected to one network. Customers use applications that are time sensitive and not owned by the provider. Customers expect that, if they want to view a video, for example, that the video actually plays and isn't choppy, or doesn't stop to buffer every 5 seconds. This is a crapload more important than how fast your Google search results load.

In this case they are warranted. Cisco's statements cannot possibly be applied to the real world without picking winners and losers.

In what way? Cisco is not saying Comcast should prioritize Netflix over Hulu, or vice versa. Cisco is saying that, yeah, ISP's should be able to prioritize Hulu and Netflix over, say, Facebook.

Let me put it this way - by insisting you treat everything the same, you're also picking winners and losers. Services and applications which need priority access (ie, very low latency and/or jitter) in order to work correctly or reliably are losers in the 'all should be equal' philosophy.

Or, let's illustrate this a little more colorfully - since the interent is often compared to a highway, that analogy will fit. Let's say you've got a gunshot victim in the back of an ambulance and he needs to get to a trauma center immediately. Do we expect the ambulance to follow the posted speed limits?

Or let's go another way - I have a limited amount of money. Tomorrow, I have to make the decision to pay the rent, or to pay my internet bill so I can make that nights World of Warcraft raid.

We already recognize the intrinsic need for priority in our everyday lives in order to get things done.

Some things are more important than others. The same concept applies to network operations, and trying to deny that is what's operationally futile.

The last person that quit or was fired will be held responsible for everything that goes wrong -- until the next person quits or is fired.