KPNQWest Admins Keep Bankrupt Network Running 202
sebastianw writes "Some of the network administrators from KPNQWest, although they have been (apparently) ordered to shutdown the network, took over control of the KPNQWest NOC. They are trying to keep the network running and keep customers up, regardless of KPNQWest's insolvency. The company warned on Thursday that they would be forced to shut down KPNQwest's entire European data network on Monday unless its customers paid their bills in full immediately." There's a related story on the Register, talking about the possible effect on UK internet access. If anyone needs to hire some network engineers...
At least it's efficient (Score:4, Insightful)
It appears to be less than the number they thought.
Re:At least it's efficient (Score:5, Informative)
Though when things go bad, we have other people on call-- it can get hectic.
So if all goes well, they could hold out for quite some time (until something important breaks).
Re:At least it's efficient (Score:2)
MTBF is far too short for crummy, overpriced little hardware boxes, and diagnostics still aren't automatic.
With the trillions of dollars that have flowed through the industry, network equipment should be less trouble than a refrigerator by now, but certain (CSCO) companies (CSCO) have had a vested interest (CSCO) in keeping the system unstable (CSCO) and difficult to maintain (CSCO).
--Blair
Re:At least it's efficient (Score:5, Interesting)
Re:At least it's efficient (Score:2, Informative)
Oh, and I've seen an ethernet module in a cat5000 suddenly start turning every packet into a broadcast packet. Oh, that was a day in hell.
But, on the whole, the hardware is impressive and stable. The software provided is a very different story.
[See Also: http://www.cisco.com/warp/customer/770/fn18164.sh
Re:At least it's efficient (Score:1)
Since their hardware crashed if Code Red tried to infect it?
I see an "intentionally" in there. How do you intentionally plan to be vunerable to a virus that hadn't been written yet?
Re:At least it's efficient (Score:2)
Most products today suck. At least computing and network gear has an excuse - widget 2002 is 50 to 200% faster than widget 2001.
ostiguy
Operated by a man and a dog (Score:2)
As you and others have said, if nothing goes bad, it doesn't take much maintenance to keep running, though it's adding new service that needs resources and causes mistakes that break things.
Re:Operated by a man and a dog (Score:1)
Yeah, but you should SEE how many people it takes to fix things when it goes wrong. We had a guy in a pickup truck driving a new mainframe processor assembly from Boston to northern Maine one day just to get things up again...
Here's to network stability in Europe for the next week.
Re:At least it's efficient (Score:2)
I'm sorry, but this kinda comment chafes my ass. Where I worked (I was laid off recently) we had a satellite fiber-optic cable production plant in Canada. My manager once stated in a meeting the place 'was running with only 17 staff...' completely neglecting his IS department in the room that remotely managed, upgraded, troubleshot, etc etc...
'You can keep most networks running with one person' does not jive with 'we have people on call..' the on-call person(s) cannot be discounted so easily. Otherwise, why have them in the first place?
Re:At least it's efficient (Score:3, Interesting)
I'm not saying they likely couldn't trim some fat, but a staff reduction of more than 2/3 cuts too deep.
Re:At least it's efficient (Score:1)
Re:At least it's efficient (Score:3, Insightful)
Re:At least it's efficient (Score:1, Funny)
Re:At least it's efficient (Score:2, Funny)
(yes, usually that's upgrades, but it is a standing joke that things only break when we're here)
Not Linux, Cisco and Fiberthings and HVAC (Score:5, Insightful)
There are other parts of KPNQwest that do run on general-purpose computers, such as the administrative and billing databases, network management systems, etc. Probably some are running Windows or Linux, and others are probably running Solaris or HPUX.
Re:depends on the architecture, really (Score:1)
Just been on their webcam in the NOC (Score:2, Redundant)
You can see the problem, none of them are wearing shirts and ties. If they were wearing shirts and ties none of this would have happened.
I'm being more serious than you might think.
Re:Just been on their webcam in the NOC (Score:1, Funny)
Now I'm gonna have to find a way explain "ebonelive" to my girlfriend when she finds it in my recent files list later tonight.
Re:Just been on their webcam in the NOC (Score:2, Funny)
Re:Just been on their webcam in the NOC (Score:1)
Honestly, I still don't see what the big deal is with that chair. I got to sit on one for a few days. It _is_ preety comfortable, but honestly, a $50 chair isn't all that bad either...
Maan
Re:Just been on their webcam in the NOC (Score:1)
But we're state sponsored, so it's the taxpayers' money. Not sure how economical it would be if we were a corporate entity.
You fool! (Score:5, Funny)
Just think of what could happen if your tie got caught in the bitstream!
*ZOINK* there goes access to Poland!
Wedding rings? *ZAAM* Oh damn, I think we just transported Roger's ring finger into the shared computer in Ms. Kingston's 3rd Grade class in Manchester, UK.
You gotta be careful when working around heavy data processing machinery, man! They aren't safe or simple, like a table saw or anything!
Re:Just been on their webcam in the NOC (Score:1)
Female, pretty (Score:1)
What's a nice girl like her doing in a place like that? (cleaning up/out, obviously, which is perfectly fine in this gender equal world, not to say that women are identical to men, nor superior, nor inferior, but on a level that is as high, though not higher, and certainly not lower, than that of the men who are browsing the internet in the background.)
Anyway, I'd like some nice person there to knock that orange cap on the ground for me, about a meter further away from the camera.
Hmm.
What do you mean, 'customer service' does not equal 'fan service'?!
Anyway, kudos on the operation to everyone there.
Re:Just been on their webcam in the NOC (Score:1)
Ah, hell. He's probably just reading about himself on
Re:Just been on their webcam in the NOC (Score:5, Funny)
Plus they can use the Internet access to do job hunting and play really fast QIII. (What you actually thought that they were maintaining the network?)
Actually I think it's great to see such dedication.
fat chance... (Score:1)
Fat chance that will ever happen. People in the US can't even their bill payment service bills on time. =)
How? (Score:3, Interesting)
Re:How? (Score:5, Informative)
As a KPNQwest customer myself, I'd like to say that they've provided a great service over the years. This is absolutely the best ISP ever, no real outages, great customer service (as you might have noticed). It's a real shame it turned out like this.
Re:How? (Score:2)
Must be the flat panel monitors and Aeron chairs [195.158.250.186].
noc admins (Score:5, Interesting)
Re:noc admins (Score:3, Interesting)
QUADROUPLE KUDOS TO EVERYONE THERE!
Re:noc admins (Score:3, Informative)
Re:noc admins (Score:2, Insightful)
If I was a admin for that company, I'd be like fuck it.
Re:noc admins (Score:5, Insightful)
We're all working very hard to make sure that disruption to our customers stays limited to the absolute minimum and all techies in this company have now shown to management that they can make their stand in the darkest hour of this company.
Needless to say I'm proud to work with these guys.
Re:noc admins (Score:1)
Screenbert - Thou wenching pottle-deep whey-face!
I think they just found out... (Score:4, Funny)
Re:I think they just found out... (Score:2)
Burning and melting hardware and then the distinctive odor of shit in their pants.
;-)
Re:I think they just found out... (Score:1)
in the end, it was Slashdot that ended them.... (Score:5, Funny)
bbh
Re:Slashdotting.. Blah.. Blah.. (Score:1)
However, I find that being a bit more poetic than "Shut up you Troll" works better. ;-)
Potential european outtage map (Score:3, Informative)
Its good these guys keep on working even though they havent been paid for a long time (some in Holland since may). If KPNQwests network goes down well.. here is their map [kpnqwest.com]. Too bad most of the news around this is in english.
Re:Potential european outtage map (Score:1)
There is an emergency plan in place that would keep them running until the end of the month if customers pay their june fees in advance.
If you have a lot of money and want to have a state of the art network with REALLY dedicated people now is the time to buy!!
I haven't been paid either (Score:2)
Oh. I bet you didn't mean bimonthly payroll...
The Ultimate Playground (Score:4, Interesting)
Call it the ultimate training lab. Make any change you want - nobody can really bitch much about it, since they're not paying for it anyway. The only motives to keep things running are to, well, keep things running...
Internet COULD be free (Score:2, Interesting)
Take some powerful countries, like France, Britain Japan, allocating funds for high speed backbones whose bandwidth is provided and distributed by volunteer organizations or other NGOs in the country, distributing mainly by the districts population. Now take laws that say a telcom company should allow DSL for free thru its network by its phoneline customers. Also take government policies that update the speed of the bones every three or so years to the several adjacent countries.
Naturally other nations would follow suit seeing the economic improvement such communications allow. Government run backbones would be kept up to speed and private competitors would be hard pressed to provide higher speeds and lower costs. Once the threshold number of countries have this, the countries that started it would be under pressure to continue providing the service, thus a positive feedback can be attained that sustains Free Internet. Apparently all we need is a few smart politicians in a few powerful countries.
Re:Internet COULD be free (Score:1, Insightful)
Re:Internet COULD be free (Score:1)
for 34 cents i can send pages across the country in about 3 days. theyre usually cheaper than ups/fedex. 50 cents for a postal money order vs my banks $3 money orders.
Re:Internet COULD be free (Score:2)
There is a law that says that NO ONE can compete with the Post Office offering 1st class mail. There used to be a law that said no one can offer next day service, and when that law was overturned, look at how many companies popped up, and are employing people off the government dole.
Socialists bug me.
Re:Internet COULD be free (Score:2)
Let's have a cite for that. Otherwise I say you're blowing smoke. The only thing that could even vaguely be termed a subsidy would be existing depreciated assets that were not funded from revenues at the time of purchase. That's not an ongoing subsidy and the only "cost" is the opportunity cost of not retroactively remunerating the treasury.
There's a specific reason for that. The USPS is able to provide flat-rate service nationwide because money-losing rural routes are subsidized by highly efficient metropolitan deliveries (i.e., all the mail within New York City that costs the same to send as a letter from the Aleutians to the Bayou).
This situation is maintained as a policy choice; the government has (wisely) decided that a baseline communications infrastructure is essential to running a country.
Open it up to the market, and companies will move in to the profitable areas, ignoring the rest. FedEx can't make money delivering 35-cent letters once a week to Zeke in his mountain cabin. Nobody can. But if we want to keep Zeke within the fabric of our society, we have to accommodate his choice to live there.
Tinpot anarchists amuse me.
Re:Internet COULD be free (Score:2)
Apparently the schools aren't as efficient as the Post Office. Pay attention much in math class? Ever hear of something called inflation?
The price of a stamp has remained pretty much exactly constant over the past 30 years.
In 1971, a first class stamp cost $0.08. Plug that into the Inflation Calculator [westegg.com] and see what you get.
Re:Internet COULD be free (Score:1)
Re:Internet COULD be free (Score:3, Insightful)
And as an added bonus, the gov'ts will not have to fight with ISP's in order to get carnivore, echelon, and whatever else on the network.
Re:Internet COULD be free (Score:1)
Naturally other nations would follow suit seeing the economic improvement such communications allow.
Economic improvement? WTF?! DSL companies keep going bankrupt charging money for their service, if its 'free', does that mean the government is gonna go bankrupt?
Though if the telco's would back the fark off, that would help. Anyways, A freely connected internet would NOT be good. Your killing LOADS of income for alot of companies (small and big). Competition is bad enough the way it is. We dont need that crap out there.
Re:Internet COULD be free (Score:2)
My tax dollars going to "free" broadband does NOT make it free.
What you meant to say is "Internet COULD be socialized"...
Webcam! (Score:1)
Re:Webcam! (Score:5, Funny)
It's the lack of ties around their necks. They have oxygen going to their brains. Makes people friendly and creative!
Re:Webcam! (Score:2, Funny)
It's the lack of ties around their necks. They have oxygen going to their brains. Makes people friendly and creative!
Shhh! Don't tell my boss or he'll start wanting me to go business casual! Then he'll raise the creative and productive bar.
You just can't let secrets like this out into the general public, man! What were you thinking?!
:lol:
Re:Webcam! (Score:2)
Re:Webcam! (Score:1)
Shows what he knows... (Score:1)
Same beer as the regular kind but less of it, at a higher price. Longnecks, pfft.
How bad will it be? (Score:1)
I'm sure I read somewhere that during the Gulf War the Allies had problems trying to disrupt Iraqi communications as the Iraqis were using a TCP/IP-based network similar in configuration to the Internet.
If the plug is eventually pulled on the KPNQWest network how drastic will the effect be?
Re:How bad will it be? (Score:5, Informative)
Take a look at this map [kpnqwest.com] and you'll see the problem...
Re:How bad will it be? (Score:2)
Re:How bad will it be? (Score:1)
Just like in a big city, sure you can get around without the major highway but without it, it takes 2 - 3 times longer to get there (Unless your in LA).
You also have the possibility of overloading the other links and then almost no one in Europe could get online.
Re:How bad will it be? (Score:1)
Re:How bad will it be? (Score:1)
Re:How bad will it be? (Score:2)
Yes, but first you have to get to "the Internet" itself. Taking down KPNQWest's backbone will take down any customer who is not multi-homed; that is, those who do not have transit through other providers besides KPNQWest.
Oh, and for those customer who are multihomed, they're going to see surges in latency & packet loss as their other providers try to keep up with the demand from all their, uh, "peers." =)
Question (Score:1)
The story so far... (Score:5, Interesting)
I can pretty much assure that the UK network will keep running, definitely one of the main AS numbers of the UK (5571) has alternate transit so do most in-country networks. If anything it's the Eurorings that are at risks but even they are supported right now from Brussels AND The Hague (both IP NOCs are operational right now).
The actions of the last few days make me proud to be part of KQ, Ebone and Netcom and I really hope that we can all survive this crisis, and if we don't, it won't be from the lack of trying or courage.
just another Netcom/Ebone/KQ techie
Re:The story so far... (Score:2)
Of course not...it will be from the shite-heads in management making all the lame-ass decisions that got you so fucked in the first place.
As always.
Re:The story so far... (Score:2)
Do you mean to say that a competent management would have recognized the decreasing demand and sacked half the work force long before it got this bad?
Re:The story so far... (Score:2)
But, let me ask you this: Would it be better for half the employees to be out of work than all of them?
Re:The story so far... (Score:2)
That's the theory. All I can say is never go into management, unless you REALLY like the money. During an economic downturn it's the shittiest job I can imagine.
But, let me ask you this: Would it be better for half the employees to be out of work than all of them?
Depends on which half you ask. If you mean, which is better for the company, the answer is self-evident.
Re:The story so far... (Score:2)
We are actually growing right now and have been for the last few years...in spite of an economic downturn. This is for two reasons:
We all work hard, and nobody is above the small stuff.
Re:The story so far... (Score:2)
I'm more concerned that you used the phrase all-hands afternoon team-building exercise without cracking up.
Al.Payments (Score:3, Interesting)
I find it interesting that they seem to be having trouble with users paying them. The most serious problem I face is customers not paying for the services they receive.
Just shut down one site 2 months ago which was overdue by 6 months... Sortly thereafter got a call berating me for their website not being up. I pointed out that they were past due and stated the site would not go back up until they paid their bill... Couple of days later I had a check.
It supprised me since the site has always been a problem with paying late. I've finally decided to take a no nonsense approach to billing rather than trying to work with customers. I've had too many customers just walk away without paying their bills and thus I have been shedding customers during the last couple of months.
It's definately a lot easier now, without maintaining a bunch of sites that don't pay.
Re:Payments (Score:1)
Someone is going to become the next Standard Oil by scooping up all of these destressed telecomunications networks nursing them back to life over the next decade. Our kids will revile them as much as we do to Microsoft.
Re:Payments (Score:2)
I ended up getting fired by my boss for having too low a subscriber base. They didn't care that the people weren't paying... they just wanted the subscriber numbers to look good for advertising revenue.
Worst job I ever had! But I'd cut those non-payers again without a second thought if I was in the same position again.
Bottom Line: In my book, you don't pay... you don't play.
Finnish KPNQwest standing by to re-route (Score:5, Informative)
On the Finnish company's page they state they're ready to re-route traffic and do what it takes. KPNQwest Norway got bought by Catch Communications and this might be the fate of the others, too. Seems Nokia is already on board as a customer.
http://www.kpnqwest.fi/yritys/uutiset/uutinen20
Anyone got more on this?
J
Re:Finnish KPNQwest standing by to re-route (Score:2, Interesting)
KQ portugal and italy are going standalone as well I believe.
We're now all waiting to see who will pick up the network so we can buy cheap transit from our former parent
Seriously though, even though we're going our separate ways, right now we're still one company (apart from KQ NO that got sold already) and we're still running the network, even if we have to do it without the assurance of being paid.
Re:Finnish KPNQwest standing by to re-route (Score:2)
Two thumbs up! I would buy you a beer if I could.
Re:oops, scratch that Nokia. (Score:1)
Then again, might have been accidentally correct, but whatever
J
Re:Finnish KPNQwest standing by to re-route (Score:1)
http://www.kpnqwest.ee/uudised.php?ID=65 (in Estonian).
It states that the customers will not be plugged out and they're setting up the necessary backup routes.
It is also stated that some HQ supported services (such as roaming) will probably discontinued or reorganized.
I hope some companies repay the admin with jobs (Score:1)
Hiring network admins (Score:2)
However, i do admire their dedication to their jobs. It's either customer loyalty or "Hell no. Not again. HELL NO." Mentality of losing a job.
:)
Re:Hiring network admins (Score:2)
Is this where the line ends? I just got laid off on Friday =)
However, i do admire their dedication to their jobs. It's either customer loyalty or "Hell no. Not again. HELL NO." Mentality of losing a job.
Yah, well. The market sucks for admins in general right now, and at least while they volunteer to stay on the job, their skills are kept fresh. Plus, think of all the free soda they would miss! =)
No, seriously, system admins can build their own boxes at home (or buy cheap discarded Suns) during their downtime, and keep practicing and learning new stuff. It's a bit harder for network engineers - most of us can't buy multiple Ciscos big enough to run BGP and ISIS on, much less any Junipers, etc. Staying with the toys until they are made to go home is a decent plan.
Besides, we don't know the full labor situation there; there was mention of the government and labor unions in one of the articles, and so they may be retaining some legal status by staying there.
Re:Hiring network admins (Score:2)
Do the dying dot-com thing and take home the switches and rouers you need. Replace them with 3COMs & Linsys's (Linksii?) from Best Buy.
No company, but still running? (Score:1)
The ones still there seem to be a dedicated bunch of people, and I wouldn't be surprised if they managed to build something from what they have now.
Although reality would probably rear its ugly head, and they would be forced to shut down anyway. Everything is nice in theory, I guess...
watch the live stream on linux using xine (Score:1)
make sure you have the necessary win32codecs installed, though.
Re:watch the live stream on linux using xine (Score:2, Informative)
KPNQwest's bankruptcy is already causing problems (Score:2)
We were leasing our lines from KPNQwest Norway, so that might explain some of it, but I suspect anyone who's trying to connect to anything through KPNQwest's backbone will be experiencing intermittent problems if things aren't resolved soon.
AT&T story (Score:2)
Most of the people just left the NOC (Score:2)
Two people left in the NOC now (Score:2)
Dawn is breaking there. Soon we'll see if a day shift shows up.
On running a homogenous network (Score:3, Interesting)
As a senior systems engineer from a similar organization (Carrier1 (FALCO!)) I can say there were no issue running a multi unix environment, and I've never had any issue with it at any of my previous companies (nor have any of the engineers I've worked with).
At Carrier1 had FreeBSD, Red Hat & Debian Linux, Solaris 9 & 9, HP-UX, even GNU/Hurd and Mac OS X (well, on *my* system
The only problem I've ever had is the fairly trivial (?!) one of getting the command flags right - stuff like the 'ps','route','ipchains, 'ipfw' and 'ifconfig' commands syntax being different, the different flags for package management tools, that sort of thing.
I quickly came to realise that it's not possible to remember all the flags for all programs and remember the best way to do something on a particular system if you are busy all the time, things just seem to seep out. This happens if you are spending lots of time programming or in meetings or working on large projects - in which case you might not touch one type of system for months (until there is a problem with it), at which point you find your self quickly reading man pages and referring to Google a lot. All you need to do is remeber what's improrant, especially things you'll need for troubleshooting, and not worry about the rest - it's enough to know about tool's like Solaris 'ndd' and Linux's 'mknod' and what they do, if you need to remeber exactly how to use them in a given instance you can refer to man pages, O'Reilly Books or Google (which I often find the fastest).
Staying current, reading Freshmeat everyday, installing and configuring new Unixes and new & un-familer packages regularly, being on mailing lists and reading Slashdot are good ways to stay up to date - the more you know the less likely you are to run into something completely unexpected. If your resourceful (which you should be as a Systems Engineer) the only real problems arise went you don't even know where to start, everything else is a piece of cake.
Basically, if you really know unix (and are not just a Red Hat Linux or Solaris flunky who has convinced themselves they are Gurus while they still run Windows 2000 day to day) then you won't have any problems.
Oh, and making lame excuses like 'well I need Windows for work stuff' and 'they won't let me run Unix on my desktop' DO NOT wash - they are just that - excuses for lameness.
I have been for job interviews and been introduced to guys who called themselves (literally!) 'Unix Gods', yet they had only ever used Solaris - if you have any of those you are in deep shit right now. [ Needless to say I ran a mile! ]
Most people fall somewhere in the middle of those two, you'll probably only have one or two decent guys, if your lucky, though if you need to ask you are very possibly in trouble already!
YMMV.
Re:On running a homogenous network - Mis-Post (Score:2)
Cheers.
I'll get my coat....
dedication and commitment (Score:2)
The professionalism of doctors and engineers is one of the things that help to leaven the more purely cost/profit approach of their managers.
Danny.
Not as heroic as it sounds? (Score:2)
So it doesn't sound like they locked the doors and kept the pipe going as much as they believe in their company and want to keep working there.
Hang in there guys