How Do Businesses Scale Their Bandwidth Needs?

onebadmutha asks: "I'm technology admin for a very rapidly growing company. We've gone from a fractional T-1, to supplementing that with a snappy DSL line, and now we're running out of reasonable options. I've looked at routers that load-balance, but do so horribly. I've considered splitting up my network users to use several incoming DSL lines, only to be confronted with intranet accessibility issues. None of these provide the kind of redundancy and control that I'd like, and certainly not with a nice pleasant UI that doesn't cause me great grief. I've looked at Open Source router distros (like routerOS, and others) and I've looked at using the full gamut of Microsoft madness. How do other businesses solve this problem of scaling bandwidth needs, without completely unlimited budgets for redundant OC-48 runs?"

Dark fiber Ethernet service, or fractional DS3

[Anonymous Coward]

That's how real tech companies do it. If you can get Yipes, Cogent, AboveNet, or some other dark fiber provider to serve you Ethernet service, that's the cheapest way to get a lot of bandwidth (10-100Mb/s range). If you can't, then you get a fractional DS3. Most real providers will let you dial the bandwidth up and down reasonably, so you could start out with a 5-10Mb/s circuit and grow from there.

Bonding T1s and DSL is neat and all, but if your business actually depends on the Internet working, go with one really good fat pipe and then maybe a thin one (T1 or so) as a backup. Don't mess with complex setups. Complex = new ways to fail.

What exactly do you need?

[dereference]

Your "requirements" seem to be all over the map. If you want redundancy, that's one thing. If you want simply to scale, that's quite another thing. If you want partitioning, that's yet a different problem.

Then, ask yourself what kind of traffic you are handling. If you're looking at users surfing the web, you probably needn't be overly concerned with load balancing; if you're receiving tons of inbound traffic to your servers, on the other hand, not only do you need load balancing, but you probably also need to seriously consider co-location solutions for your servers.

The adminstrative traffic is typically a much lower priority in most companies. I don't know how many users you're talking about, or what they're doing, but most small companies just live with a single (full) T1 until they absolutely need to bond another T1 (where "need" is subject, but should be kept in check, especially given that last bit about not having unlimited funding).

I guess this is not much of an answer, but these are all important questions you need to be asking yourself well before seeking specific answers. I'm not sure where you're coming from, and I don't mean to accuse you of anything, but taking the approach that you'll know the right answer when you see it is usually flawed from the start.

Whoa, slow down there

[misleb]

Slow down there, chief. Exactly what kind of company would be going from fractional T-1, to DSL, to... an OC-48? (I sssume you were exaggerating on the OC-48)

Couple questions:

1) How many employees are we talking about here?
2) What are they doing on the internet that is so demanding?
3) Are you running any web/streaming servers onsite?
4) Have you gone to any lengths to diagnose exactly what your bottleneck might be?
5) Are you sure you don't just have a couple of hogs downloading porn all day?

I know 200+ employee companies that get by with a single T-1 just fine. I'm a little suspicious of your bandwidth needs.

But if you really meed that much bandwidth for web browsing (I doubt you do), the next step would be a DS-3 circuit at about 45Mbit. But that can be pretty costly for the circuit alone. It would, however, allow you to scale because you'd probably be paying for the bandwidth used and not the full 45Mbit. If you are in a building with other companies who have similar needs, you may be able to split the cost of the circuit and share it.

Also, depending on your location, you may be able to setup a wireless (not WiFi) deal with someone. Something with real gear, of course. Not just a couple Linksys' with Pringle can antennaes.

-matthew
 

What are you using it for?

[georgewilliamherbert]

Is this internet access for desktop users? People from outside coming in to your corporate website? VPN connections to other offices? How many users? Are you attempting to syncronize any data across the link? In real time, or overnight?

The possible set of right answers depends a lot on what you're doing with it.

Policy based routing plus any number of DSL lines will work for splitting up desktop web access.

Inbound traffic for the corporate website is pretty much the antithesis of that... outbound traffic is the target, and that ends up being T-1 optimized for small sites and bonded T-1s or faster links for bigger ones.

VPNs can be symmetrical or asymmetrical. Your mileage may vary.

Ban BitTorrent = problem solved

[patio11]

You've got a variety of options for banning bitorrent (that is your problem, right? You have done traffic analysis before coming to Slashdot, right?). This is in an escalating hierarchy of how invasive you'll have to be. 1) Tell your employees that bandwidth costs have gone up, that you know BT to be the source of the problem, and that you trust them to do what is necessary. 1.5) Ban BT by policy, threaten severe sanctions up to and including dismissal for skirting the ban. 2) Block the standard BT ports. 3) Filter out BT packets. 4) Install computer forensics software and look for evidence of BT use (pretty much has to be combined with 1.5).

More than bandwidth

[misleb]

Such a line can easily be brought to it's knees by simply saturating the upstream. ADSL does not work well in business environments with many users. I'd take a full T1 over that 7M/768k DSL line for a business any day.

-matthew

Re:Speakeasy Bonded T1?

[mnmn]

Forget speakeasy bonded T1, you can bond your standard DSL lines through an OpenBSD firewall using CARP. Read also about VRRP and (HSRP and GLBP) for cisco solutions. They add not only redundancy but also load balancing, and recovery is real fast as opposed to something like RIP2. You can also use OSPF but careful, OpenOSPFD and zebra dont provide load balancing and redundancy of default routes. IOS does.

I say spend your budget on additional lines instead of cisco smartnet.

Weigh your options

[aelbric]

In short, there are several commercially available choices that may be available depending on latency, bandwidth, price, reliability, and availability.

1) Classic T-1, 1.5Mbps
2) IMA (Inverse Multiplexing over ATM) - Essentially bonded T-1s up to about 6 Mbps before the cost of the routers becomes prohibitive
3) Ethernet Switching - 10Mbps and higher
4) DS-3 and higher - 45 Mbps and up

If you need high availability, option 1 is ruled out. IMA is good for speed and availability, but increases complexity. Ethernet switching is fast, but redundancy will cost you and it will require additional CPE devices for security and traffic monitoring. DS-3s and up are reliable and fast, but the cost of high availability (e.g. dual-entrance facilities, multiple providers) is astronomical.

Set yourself up a matrix of each of the key metrics that make a difference to you. Talk to all your possible providers and populate your matrix with their service responses. Read their SLAs very carefully. Understand how they calculate their measurements. A 99.98% availability can be insufficient depending on how they calculate it. Weight their responses based on your business requirements and then choose the option that best suits your needs.

If all else fails, bring in a telecommunications expert for a couple hours to help you analyze your options.

Re:More than bandwidth

[Wdomburg]

I wouldn't make that kind of determination without evaluating the existing and projected traffic and use patterns. Considering this was supplemental bandwidth, it's almost certainly being used for internet access and not critical services. The upstream requirements are likely well below what a business class ADSL line provides. If the problem isn't upstream and you go for a plain T1, you'll bring it to its knees saturating the downstream. :)

Reliability may be an issue of course. Depends on how much the userbase depends on real-time access to the internet. If most of the users rely on intranet resources, no biggie. They can live without the web for a little while. On the otherhand, if the company relies heavily on externally hosted applications like CRMs or mail, you likely should have redundant connections, period.

You forgot the part...

[Atario]

...where morale drops through the floor and people start looking for new jobs.

Nobody likes living under a fascist big-brother network policy. But, hey, you put those lousy "freeloaders" in their place, huh? That's all that matters, after all...

And YOU forgot...

[JoeD]

... that it's their network, their rules.

Some non-work net use is inevitable (like me making this post). But when people are using their workplace's network connection for non-work activities to the extent that it's impacting the performance of the rest of the network, then something has to change.

For most businesses, there is simply no business reason to allow people to download music and/or stream video to or from the office. It's just like the telephone. Most places don't mind people making personal calls, but they ask that they be reasonable about it, because you're supposed to do your socializing at home, on your own time. It's the same with the office net connection. Nobody cares if you use it to order a book from Amazon, or read the daily news, or browse Slashdot. But if you start hitting iTunes or Youtube, or start doing lots of Ebaying, or share a torrent of last night's "American Idol", then you might be crossing a line.

Re:Dark fiber Ethernet service, or fractional DS3

[baptiste]

i can't belive he asked slashdot.. there are plenty of forums out there deticated to this type of stuff.

Why is it every time someone asks a legitimate IT question on /. the poster is ridiculed with the above statement. Every. Single. Time. I for one think /. is a GREAT place to ask questions like these. Unless you've been the 'jack of all trades' IT guy at a small company, you have no idea what it is like. You're expected to know EVERYthing. Sure - there are forums all over the place dedicated to this specialty or that specialty. And if he was a network admin only, he likely would read those forums every day.

I think /. is a great place to ask questions like these. Sure you have trolls and ACs who sometime suggest silly solutions. But you also have a LOT of hardened geeks and IT types who have been around the block a few times who make good suggestions. Already here I've seen 3 or 4 solid solutions that he can now consider and do more research on to see which fits his company best.

Asking /. a question is not a sign of a n00b or bad IT person. What better place than one of the biggest techie readerships on the Internet to ask questions. I find many Ask Slashdot threads to be very informative, filed away for 'future use'

At least you followed up the standard 'I can't believe he asked /.' with an actual, you know, answer.

OK, move along nothign to see here. I had to waste a little Karma anyway.

Re:You forgot the part...

[ednopantz]

>fascist?

Lay off the bong hits kid. Grownups understand that they aren't supposed to be torrenting all day on the boss's network connection. Anyone who quits because they won't be allowed to torrent porn all day does the boss a favor.