How Much Does Your Work Depend on the Internet? 322
malord asks: "I work for a small company that has recently had problems finding a stable internet connection. It started when we moved our office in order to upgrade our connection speed. We decided to go with cable internet through Comcast, since they offered the best speed for the price and told us that it would be available before we moved. Unfortunately, Comcast did not provide any service for two months after we moved, so we piggy backed on an existing (slow and unreliable) wireless account with another company in the meantime. When Comcast finally came around, the service that was provided was far from adequate with a consistent 30% packet loss and multiple disconnects everyday, which was confirmed through Comcast's tech support. Throughout this process, we have realized that having a reliable internet connection is more important than having a phone line and almost as necessary as electricity. What would you do if your internet was suddenly like dial-up for weeks at a time? How much money would your workplace lose if it was out for an hour or an entire day?"
How timely! (Score:5, Interesting)
You may think: hey, that's not bad. You only lost one day - really less than a full work day. Oh, but that's where the pain comes in. I run all our services in house: Goodlink (a Blackberry-like system), Exchange 2003, DNS, everything. Plus, while the lines were down, anyone who called our office heard five rings and was then disconnected. The loss in customer service is irreparable to one major client, and three unbelievably important emails were lost forever - the kind where the intended recipients weren't really in a position to say "Hey, can you resend that for me?" We'll never know exactly how many emails were lost. In a world that works 24/7, business never stops, and an important email that comes in at 3 AM is just as critical as the important email that comes in at 9 AM sharp.
Direct answer to your question: Our T1 line is beyond essential to the daily operation of the organization. It's absolutely mission critical that we're connected at all times, without interruption or major packet loss.
Lost forever? (Score:5, Insightful)
Re: (Score:2, Informative)
I'm mildly sceptical of the need for secondary MXs, especially ones you don't manage yourself.
Re: (Score:2)
Re: (Score:2)
Our webhost, who offered us secondary MX for "free", changed their hostnames for SMTP servers. Apparently they updated DNS records of those whose DNS records they manage.... and didn't bother to tell anyone else. Now I have to test the secondary MX services every few months just to make sure they still work.
Re: (Score:2)
Re:Lost forever? (Score:5, Informative)
Re:How timely! (Score:5, Insightful)
Re:How timely! (Score:5, Interesting)
Hang on... (Score:5, Funny)
Re:Hang on... (Score:5, Funny)
Re:How timely! (Score:5, Insightful)
Re:How timely! (Score:4, Insightful)
runs on extremely tight deadlines? Where clients don't
always double-check that their emailed order was received
but expect a job to be done/shipped/delivered the same day?
It's not exactly feasible for most companies to offsite
one or more employees just to maintain a constant internet
presence.
Web hosting isn't the only vital internet service.
How did that get marked "insightful?" (Score:3, Insightful)
Re:How timely! (Score:5, Insightful)
Re: (Score:2)
I'd like to commend you on your response. You hit all the right points. Many people noted that he could have had multiple T1s running with failover, but I think that's only one of many techniques that should be in your arsenal. The fact is, there are many ways to protect yourself from internet outages, and buying 20 T1s isn't the most efficient (or effective) method. Multiple connections, an extra POTS line, mirroring/colocation-- you have many techniques available, and you should use as many as possibl
Re: (Score:2, Insightful)
And you only have 1 T1?
If it really was that mission critical you'd have a second dual-diverse line.
Amateurs.
Re: (Score:2, Interesting)
Re: (Score:3, Interesting)
Uh huh. Meanwhile, he made a good point. If you need guaranteed services, you have to realize what it takes. Hosting web services in house has some mythical attraction that i've never grasped. Get this, host at a colo that has multiple very fast very reliable incoming connections, and you then only worry about your internal people.
Re: (Score:3, Funny)
Re:How timely! (Score:5, Insightful)
South, the only person likely to read that as snide is you.
He's completely correct - if connectivity is as critical to your business as you're trying to make out then you should have at least N+1 redundancy not just for your comms links, but for your core servers like mail and web (if you're hosting your own).
You work for a non-profit business. That doesn't mean you work for a no-money business! Make a business case to your management *now* to get a redundant link so you're not a repeat victim. Don't wait one, two or six months to do this, do it now while the pain is still fresh in their memories! You may not be planning to change providers any time soon, but do you honestly think you'll always have completely unimpeded 100% uptime?
If you fail to do anything about this then you're no better than the noob at home who thinks his RAID array is enough for backups and then complains about losing his multi-terabyte porn collection when he's defrag'd after "accidentally" deleting it.
Redundancy for home (Score:2)
I currently have microwave fixed wireless and i'm considering getting a cheap dsl or cable package as a backup. I work from home and since i'm paid hourly it costs me dearly when my connectivity drops off.
Re: (Score:2)
Re: (Score:2)
"Does anyone here keep multiple internet connections at home?"
If you have a cell phone, you have a net connection for that "emergency email that just has to go out", etc.
If you have a laptop with wireless, a bit of wardriving or a stop at a Starbucks gives you another alternative.
So yeah, a lot of people have multiple home connection options.
Emails were lost? (Score:2)
So... Why do you have only one of them? It all comes down to money, did you lose more than the T1 costs when you lost the ma
Re:How timely! (Score:5, Funny)
I think that would apply to just about everyone on the planet (and many animals from what I've seen at the zoo). Why single out tech support?
Three backbones (Score:2)
If we're down, we know for sure that we have bigger problems than an outage: namely, it's time to crawl into a bomb shelter and wait things out.
Re:How timely! (Score:4, Informative)
We specifically use EasyDNS [easydns.com]'s DNS service which includes the backup MX service. We use their DNS Plus service which only costs about $40/year, and allows us to use their CLUSTER of backup MX servers (How cool is that!?)! Its also available on their DNS-only service (~$20/yr). I don't work for EasyDNS (just a happy customer). You can also get the same service from lots [zoneedit.com] of other [no-ip.com] places [dyndns.com] as well.
Realistically, I think you need to use an external DNS service to do this for network outages (since other mail servers will need access to your domain's MX records to find to the backup MX servers). For us, this meant we needed to use a different DNS server inside our local network. The external dns points people to our mail server's public IP. The internal dns points to our internal ips.
Another note, we use PFSense [pfsense.com] as our firewall (great product!). Recently, I think I saw support for NAT Reflection was added (allowing internal machines to contact internal servers using a public IP address), which might negate the need for the "split" dns described above. Haven't tried that yet, though.
Re: (Score:3, Insightful)
If it was unbelieveably important, a courier should have been hired to deliver a CD or hard drive. That's what professionals do. Email is designed to be a "best effort" service. It's perfectly fine for routine day-to-day communications, but not for "unbelievably important" things. If it's really that bad, the people who sent the material by email should be reprimanded or fired.
Re: (Score:3, Informative)
Thankfully, our server only handles ~5000 emails a day, and that leaves about 30 a day I review. I know that I'm in a special situation, where at a larger organization I wouldn't be able to do that. But it works.
Newsletter subscription request (Score:2, Funny)
Wow, what an amazing conclusion. Next thing you'll be explaining that lower contention and higher service levels are why business class DSL is sold at premium. Please, keep us informed of your awesome discoveries.
Don't use a consumer-grade service for buisness! (Score:5, Informative)
Mistake #1.
You're a business. There's no reason a business should be using anything less than SDSL. It costs more for a reason - it's reliable.
quoth http://www.speakeasy.net/business/dsl/ [speakeasy.net]
> Symmetrical dedicated line DSL with throughput SLAs, rigorous uptime and repair time.
That means they guarantee it'll be fast, it'll work, and if it doesn't, they'll fix it fast.
If a couple hundred per month for internet is too much for your internet-dependent business it sounds like you've got bigger issues than packet loss.
Speakeasy (Score:4, Informative)
I can't recommend them highly enough. Pick-up-after-a-few-rings, by-a-person-who-can-talk-dBs-and-DNS grade service, 24/7.
And that's on their residential product.
Here's a question for you (Score:4, Insightful)
1: Buy an SDSL business service from one supplier, with SLAs, rigorous uptimes and repair times.
or...
2: Buy cheap ADSL services from two or more suppliers but forget the SLA, uptime and repair time guarantees?
I strongly suspect that (2) is the cheaper and more robust system.
Re: (Score:2)
If you just want to surf the web, and you don't mind a little wierdness and in-office NATing, then your solution is the right one, but for services offered to the world, you either become a network admin
Re: (Score:2)
15 second TTL for your mail handler DNS should handle failover quite nicely; you only need once mailserver in-house and N DSL lines.
As has been said, iptables and masquerading to fix the outgoing route is the simplest solution, and update your zone file to point to a working route for your email server.
The cost of coping with N incompetent suppliers could well be high, and beware if they use the same "wholsesale" provider for the underlying service.
We have 2 DSL lines
Hmm, externally viewable services (Score:2)
Re: (Score:2, Insightful)
You low SLA providers are sure a
Re:Here's a question for you (Score:5, Insightful)
Except for the fact that both the ADSL lines go over the same copper bundles from the same CO from the same LEC regardless of whether or not they come from the same ISP. Most businesses feel that $600/mo for Internet service isn't worth the price until they realize how they've built themselves into a corner.
A sad fact is that most small and medium businesses will go through this pain and suffering at least once each year for several years before learning better and, in the end, spend more and lose more money trying to do Internet on the cheap than paying up-front to do it the right way. And while it seems sensible and cost-effective to host mission-critical services in-house, the reality is that if they are truly mission-critical and you can't afford proper redundancy, than those services are best off being hosted or deployed at a co-lo center that does provide N+1 redundancy and 24x7 business-class service and support.
Lessons learned:I've been out of the small/medium business consulting market for a few years now. But when I was consulting I encouraged customers to host or co-lo all mission critical applications and use terminal services (Windows remote desktop) or Citrix for access. The hosting or co-lo center provides all the redundancy and 24x7 service and support, you just pay the bill. The cost was not unreasonably more expensive than hosting these apps in-house when you consider downtime, maintenance and ongoing consulting fees to keep things going.
That's just off the top of my head. I could go on, but then I'd have to send someone a bill.Re:Don't use a consumer-grade service for buisness (Score:4, Insightful)
Re: (Score:2)
I'm just playing devil's advocate here, but I find it amusing to do the math. Also from their site:
Do you realize that works out to 9 hours of down time per year? Or 10 minutes of downtime per week? Or 2 minutes per business day? While 99.9% uptime sounds good, you have to ask yourself if th
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re:Don't use a consumer-grade service for buisness (Score:3, Informative)
Personally, I can't wait until congress finally legislates Net Neutrality out of existance, so everyone can truly find out how sweet we have things right now (or actually, how sweet we had things in the 1990's).
Re: (Score:2)
Do you hire yourself out as Tech Support? I mean being able to troubleshoot a connection from halfway across the country without even having to examine anything surely makes you some pretty good money.
Re: (Score:2)
All the Time (Score:3, Funny)
'Course I have a backup connectin through the bluetooth connection on my cellphone and T-Mobile's unlimited data service. Which leaves me in the perfect position to score with the hot sales babes if our provder's border routers ever go down. Aww right!
My connection's been dropping randomly (Score:2)
If you depend on the internet and your internet connection is fubar then by definition you are fubar. If your supplier is telling you to go f*ck yourself by denying there's a problem or refusing to fix it then change suppliers. They ar
100% (Score:3, Insightful)
Re:100% (Score:5, Funny)
If my Internet connection goes down, then I have to go to work.
Irreversable Damage (Score:5, Interesting)
Using two ISP's is a relatively untapped resource today, much like mirroring hard disk drives in a RAID array was a few years ago. Today, nobody will build a server without at least one redundant drive. I believe Internet connections should be the same way. How often do businesses complain of "sorry, our network/Internet is down" and lose customers? Do a Google search on a "Dual-WAN" router and see there are a few products around. I love my HotBrick LB-2 router that I use at home. There are about half a dozen people that can easily stress a standard RoadRunner connection. Using my friend's DSL connection going to the same house, it both load-balances and has failover capability. I don't even think twice before unplugging my cable modem. Without any downtime, the router will use the DSL line to pick up the slack.
Is it affordable? Well, that's the same question people were asking about mirrored hard disk drives years ago. The question becomes, is it nessesary? I'm not willing to move into a house that doesn't have the availablility of having two ISP's.
Aj
Diverse networking is normal. (Score:4, Informative)
Um... It's pretty much been standard practice since day one. It's how the Internet provides robust routing. All businesses relying on their network should be doing it. Diverse home networks? Depends how important your porn supply is to you.
Re: (Score:2)
What about email?!?
Re: (Score:2)
Re: (Score:2)
Using two ISP's is a relatively untapped resource today
Not among business of much size. Multiple T1s with some sort of load-balancing is pretty standard practice, really. Providing services to the outside world over those connections can be a bit tricky, but outgoing connections are easy as pie.
is this the right place for this question? (Score:5, Insightful)
Pretty much all of it. But then, look at the crowd you're asking.
Re:is this the right place for this question? (Score:5, Insightful)
None at all (Score:5, Funny)
Hooray for state parks!
if service is out (Score:2)
I work from home half the time... (Score:2, Insightful)
What I would do (Score:2)
I remember when we moved our offices it turned out our current dsl provider didn't serve the area, a T1 fro Verizon was hideously expensive for too little bandiwdth. We ended up going with Cox, we've got a 10mpbs link to them, and then two 2mbps feeds to our other offices. Works very well.
However when they had New England Line drop the runs from the MDF to our point of presence, they used stranded connectors
Same here (Score:5, Interesting)
Answer to the question (Score:5, Funny)
T1 (Score:2)
Redundant feeds (Score:5, Informative)
My computer feels worthless without the Net (Score:2)
Which reminds me, if the internet is down, is there a good firefox extension or other thing that saves almost everything that one surfs? Scrapbook is nice, but requires manual use from what I have seen. The cache doesn't cut it either.
Re: (Score:2)
There's always an alternative. (Score:2)
Verizon EvDO comes to mind. Sprint offers a similar service. There are sometimes local LMDS providers. Cable, DSL, ISDN, T1 (it's not always as expensive as you think).
An EvDO or similar data card is typically around $60/month, and can be used by travelers to boot. Every business should have a backup when affordable, and this one is....
No service for two months?! (Score:2)
Internet is important to me (Score:2)
anonymous coward (Score:2)
You can pick up your final check on Tuesday.
99% (Score:2)
Josh
Not much (Score:2, Interesting)
I'll give it another 10-
Just realizing this? (Score:2)
My father is currently an executive director for a billion dollar/year company, managing the disaster plan and more. They do increasingly more bussiness via telephone and internet, though its probably still mostly "storefront." Regardless, if the internal network went down (it spans much of New York City) or the net connection died, they stand to lose over USD $1,000,000/day (on an off day).
Home based design and programming (Score:2, Interesting)
I've lost power -- 2 hours this week, one friday for 24 hrs into saturday, etc. on several occasions. Losing power is disasterous.
I have lost internet without losing power, but far less frequently. There's only so long the cable modem stays up on a UPS
Without power, my laptop battery goes from 2-4 hours. I can still usually code and design for a bit, wrap up
bah that's nothing (Score:4, Informative)
we're right on the border to where Medicom and Comcast seperate.. and verizon is simply a joke.. I've actually contacted the President of Verizon for Delaware's district, to no avail.. One of those typical, "I'll get back to you on that" phone calls.. For us to get DSL, would require them to spend a few thousand dollars in running new lines underground, as well as special hardware for the fucking FIBER FED PG BOX literally a hundred yards from our office.. Cable companies have also said, that they'll need to dig underground, costing thousands, just to lay some cable to our little warehouse..
I've thought of every possible solution, and they are either too cost worthy, or they simply won't work.. we can't afford to have downtime, and dialup is better than nothing at all.. but I did do the math, and we lose a maximum of a 1000 hours every year in productivity due to waiting for pages to load, uploading high res images for products, and the bulk submission of hundreds of ebay items.. ahh well.. i've definitely gotten used to it, but it makes me wonder how much more money we could make, if we just had a faster internet connection.. I certainly understand that even a crappy satellite investment could help us out big time.. but my bosses are still struggling to pay the monthly bills, so its really out of the question until someone like Verizon, Mediacom, or Comcast can offer a decent $30-70 a month internet connection..
Re: (Score:3, Insightful)
Very important.. (Score:2)
This afternoon a crappy netgear router blew up (you get what you pay for..) and we lost our internet connection a couple of hours before the end of the day.. Perfect timing on a friday, early pub
How much we rely on it? (Score:2, Interesting)
Currently, I'm in the process of setting up a new company. We will rely on the internet even more, since we will deve
Verizon FIOS (Score:2, Interesting)
We've been using the business class of Verizon's FTTP service for a few months. Their entry level is $100/month, and they give you a solid connection, no nonsense, and 5 public IP addresses. They do the install and everything.
We've been pretty happy with it, but recently Verizon seems to have been doing maintenance, and connection speed has gone downhill. This is not typical, however.
An added advantage of using Verizon for the connection is that they also provide residential connections. This allows tho
In the same Boat (Score:2)
I telecommute as the editor of a fairly lage magazine (circ. ~ 100k). Net outages or low bandwidth are killers to us (I up- down-load hundreds of megs of text and photos daily). Because I am in a rural area, I hve NO choice of ISPs. Were Comcast one of them, I would NOT choose them because of theor lacadasical attutude toward outgoing spm from zombied machines on their network.
Comcast sucks.
Probability of failure (Score:2)
Two providers; A and B. Provider A has a 10% chance of failure, provider B a 5% chance of failure. If you have a single line then you want provider B and have a 5% chance of losing your line. If however, you use both providers at the same time and don't rely on just one of them. You now have only a 0.5% chance of losing your internet connection, assuming both lines are completely independant. The nice thing is that IP was designed from the ground up to take
Don't. Want. To. Think. About. It. (Score:4, Funny)
I'm a Web Developer (Score:2)
Re:I'm a Web Developer (Score:4, Informative)
It's more like my sanity (Score:2)
I changed strategies on that recently (Score:4, Interesting)
Now, the stuff clients see is 100% reliable (I have a failover server). The cable modem for my own use works fine -- in fact has been more and more reliable as the cables companies are now trying to compete for phone service and discovering people don't tollerate phone outages nearly so well as cable tv outages.
The Horror... (Score:4, Funny)
Well... (Score:3, Funny)
If Comcast confirmed the problem (Score:3, Interesting)
Lose? (Score:5, Funny)
For some of us, we'd probably make massive *gains* in productivity.
- MugginsM
Possible fix for malord's Comcast problem (Score:4, Informative)
If Comcast had any brains they'd keep a whole bunch of these in every Comcast service guy's truck and train their people to read the cable modem's signal status page. It'd be a helluva lot cheaper than repeated truck rolls to the same very annoyed customer. Better yet, they'd replace more of their aging copper with fiber before FiOS poaches all their best customers (alas, I'm in SBC/AT&T territory), but that's another rant entirely. Overall I'm reasonably happy with Comcast in my area but I'm still jealous of folks who can get FiOS.
$130 000/hr (Score:4, Interesting)
Servers are clustered.
Spare desktops are available.
Floor switches are redundant (and on separate power feeds).
Internet service is redundant (through two major carriers).
People have backups who know their job.
All service contracts have specific performance requirements.
If Comcast isn't meeting their stated performance, then they'd better FIX IT NOW! It's their job, after all. Mind you, if they haven't guaranteed anything to you, then they don't have to worry about any more penalty than losing you as a customer.
Get the SLA it in writing, hold them to it, and if they fail, legal action may be neccessary as a last resort.
The military does... (Score:4, Interesting)
Now this doesn't mean we can't do the job at all. It just means we have to switch back to the old paper and mail methods. This is significantly slower obviously, but it works.
It's interesting to me how the military doesn't do this for money, but rather for this idea that a Soldier's life is at stake. So does that mean that these companies that abandon paper methods don't take their work as seriously as the Army? Or just that the risk of saving money by abandoning these methods is worth it in the long run?
Does a day without net really matter? Or as the parent post mentioned, do months really matter?
Why do people go for the cheapest? (Score:3, Insightful)
absolutely essential in medicine (Score:3, Informative)
I'm expected to know/do something about virtually anything that walks in the door, including industrial toxin exposures, any/all medication overdoses, even "my child ate this weird plant" complaints. I can access pill databases, get radiology reports and images, look up MSDS, and even have a few botany sites bookmarked for exactly that kind of weird stuff.
Standard ER stuff I can do with my eyes closed, but reference materials online are absolutely essential for the bizarre ones, and it's why I have redundant internet connections (one of which I set up and maintain myself).
I'd be far less effective without it.
Extremely important... (Score:3, Informative)
It cost me my entire life (Score:4, Interesting)
Guess what I found? They barely have internet here. I had to pay $600 a month for a T1 from Bell South and then found out the infrastructure and/or local workers could not make it run reliably. I had SLAs which were totally ignored. Monthly credits were usually close to the cost. And finally the line went down completely for over a month. I was forced to switch back to dial up, and if I got a 2400 baud connection I felt (feel) special.
Needless to say, I have lost all of my clients, all of my work, they take my car in a month. Moving is not an option for unrelated reasons, but the bottom line is there are still places in the country where internet can not be reasonably obtained (i put satellite in the unreasonable group - it sucks - try VPNing with a dish) and it cost me everything. No this is not made up.
MOD PARENT UP (Score:2)