Slashdot Log In
The High Tech Sweatshop
from the sometimes-no-amount-of-money-is-enough- dept.
The High Tech Sweatshop
Its 4:30 am on a Friday and I just finished the last Mountain Dew. We ran out of coffee hours ago, the remains of it now black sludge at the bottom of the pot. The buildings air conditioning went off sometime the previous night and its up to almost 90 degrees in the server room. The two volunteer hackers on the staff went home after 12 hours, leaving me and the sysadmin...
This is a normal day for me.
I'm a systems engineer in the client services division of a network security software company. Basically what that means is that when networks break, I fix them.
I am 22 years old, I make a large multiple of the national average salary, and if I cashed in my stock options I could buy a very nice house. I'm also sixty pounds overweight, I sleep an average of four hours a night, and I have several ulcers. I usually spend about 60 hours a week at the office, but I'm on call 24 hours a day seven days a week. If I was honest with myself Id probably say I worked about one hundred hours last week. This is a normal life for someone working in this industry.
We live in a world today that runs on information. And people want all of it now. When was the last time you actually wrote out a personal letter to someone, on paper, in pen? Why bother when E-mail is so much faster and easier? But what goes on behind the scenes when you hit the "send" button? There are thousands of people out there just like me who have titles like "Network engineer" and "Systems administrator". We keep that information flowing, and we get paid what seems like a lot of money to do it. If you've been in the market for a good network admin lately you know what I mean. The market is pushing the salary into the 100k+ plus range for someone with the necessary experience to handle even a relatively small network, never mind what the really large companies like State Farm insurance or Wells Fargo bank have.
I started work on this problem with the sysadmin on Thursday before the close of business, getting things set up, preparing for the changes etc... The company was switching internet service providers that night because the previous one hadn't provided the level of service they needed. This entailed changing the IP addresses, and DNS configurations of every machine in the building, running three different operating systems, probably two hundred machines all told, then setting up the servers, routers, and switches necessary to get it all running. It's a big job, but with six people working on it we figured we could get it done before start of business the next day. Normally you would do this kind of thing over a weekend, but the ISP could either do the changeover tonight, or wait till next week, and we needed to be online before Monday.
Getting back to what happens when you press the send button. You expect the computer to send the message, and that the person it was sent to will receive it. What happens to the message then is an incredibly complex series of storage, sending, routing, switching, redirecting, forwarding and retrieving, that is all over in a fraction of a second, or at most a few minutes. But you don't care how or why it gets there, only that it does, and this is all you should care about. After all you don't have to know how your cars engine works in order to drive it right. But someone has to know in case it breaks. And when your email breaks you expect someone to fix it. It doesn't matter what time it is, or where the message is being sent, you want it to get there now.
Its now 8 am and the network is still down. We've managed to isolate a routing problem and are in the process of fixing it. The ISP gave us the wrong IP addresses and now we have to go back and redo all two hundred machines in the building. The router was crashing and we couldn't figure out why. Two hours on the phone with the vendors support, and three levels of support engineer later we fix it. People are starting to come in to work and ask why they can't get their email. The changeover process takes us about three hours and finally everyone has the right IP, but things still aren't working right. A bunch of people use DHCP for their laptops and the DHCP people cant get out to the net. The CEO of the company is one of those people...
So what do we do? Well we hire people to take care of the network. And we give them benefits and pay like any normal employee. We also give them pagers, cell phones, a direct phone lines to their houses so that any time, any where, we can get them, because the network could go down, and we DEPEND on that network, and those people. This is where things go skew from the normal business model.
All compensation is basically in exchange for time. The only thing humans have to give is their time. When I pay you a salary it is in exchange for me being able to use your abilities for a certain period of time every year. The assumption is that the more experienced or knowledgeable you are the more your time is worth. This works fine when you are being paid a wage, but salaried employees aren't. They exist under the polite fiction that all their work can be done in a forty hour period every week, no matter how much work there is. We all know this isn't the case of course. And when it comes to Systems administrators and network engineers that polite fiction isn't so polite. In exchange for high salaries and large stock options the company owns you all day and all night, every day and every night. You are "Mission critical". High salaries become an illusion because when it gets down to it your hourly rate isn't much better than the assistant manager of the local Pep Boys.
I finally went home at 1 that afternoon. I couldn't stay awake any more and if I didn't leave right then I wouldn't have been able to drive home. The funny thing is I felt guilty for leaving. Things still weren't working quite right, and I felt like I should have stayed until they were. Even funnier is that I volunteered for this. The only part of the job that I actually had to do was to change a few IP addresses and configure the firewall, but I thought I'd lend a hand, and I couldn't do the firewall till everything else was working anyway. My wife hadn't seen me in two and a half days, and I could barely give her a kiss when I walked through the door and collapsed on my bed. The SysAdmin was fired a few hours after I left. Back to work Monday morning.
4am Friday here too... (Score:3)
We (way up here in Highland NY) get a phone call 2pm and are told "I can't get on the AS/400" so the usual 30 minutes of attempting to figure out what really is wrong with the user over the phone.
We then finally pull from them that the "security" guys were just cutting(!) some "phone lines."
Well as it turns out that "phone line" was our Cat 5 line from the upstairs office facility to the downstairs production floor. There goes 35 connections!
So if we leave now (2ish) we will get there at 5 and they will have left. So we just finish up the usual 12 hour day here in Highland and then run home and grab a bite to eat, change into "sweatshop" clothing (from khaki's to shorts) and head down to Garfield. 2 hour trip, one way. We get there, all the lights are off, can't find the panel. Time to light the way with a flashlight.
Anyway, its about 105 degrees with about 80 percent humidity and about 11 hours of work running 800 feet of pure hell through concrete, brick, wood, and what seemed to be like butter: suspended ceilings
Did I mention the whole MIS dept is me and one other guy? For 300 users (most god-awful PCs) and 3 locations (all of which are 2+ hours away from each other.
Just another lovely day in MIS. Remember folks: the MIS folks don't exsist untill someone can't send email!
Re:email (Score:3)
they haven't gotten any email for the last two hours.
echo "yes, the net is working" |mail -s FYI phb@hell.com
In the crontab, every 2 minutes. For more fun, grab a few random spams and echo those instead.
Re:This guy is full of it. (Score:5)
As y'all know, I've done a lot of work on the NT platform, and in my experience about 80% of NT problems can be traced to poor systems administration (about 15% more are caused by deploying it into inappropriate roles, and about 5% because of flaws in NT). Why is such a large proportion due to this cause? It's because NT looks like Win95 on the surface, a simple, domestic OS, and it's very easy for people to bluff their way into sysadmin roles on the NT platform - there are people calling themselves Domain Administrators who I wouldn't trust to look after a digital watch, much less an enterprise computing resource! And there's no way to find out until a recovery situation for most companies, as they lack the skills for a truly rigorous hiring process. This isn't a criticism - after all, that's why people get hired, to bring a skill into the company in the first place!
I've never worked with Netware, but I gather the Novell folk found themselves in a similar situation in the early 90's. A bunch of people who could manage the basics were placed in positions of responsibility, and when the situation arose that required deadly skills, they just weren't capable. And everyone suffered for this: the corporates didn't have the network support they needed, the operators were humiliated and fired, and the industry as a whole was blamed. However, the CNA/CNE programme went a long way to weeding out the incompetent, and the MCP programme is starting to have an impact in the quality of NT staff.
Any kid can download linux and teach themselves, which is a good thing when viewed abstractly, but it will definitely result in a lot more people on the market who, whether intentionally or not, grossly exagerate and misrepresent their own skills. This can only be a bad thing, it will bring ill-repute on the sysadmin profession.
Sounds like he's creating his own problems (Score:3)
As far as salaried hours and time spent and the "polite" 40 hour weeks -- he's not demanding it so he's getting pushed around.
If he were smart he'd cash in his stock options and find another job where they'd not push him around like that. I know this is easier said than done becuase I feel for the guy. I don't like leaving until things are working. I hate seeing something only partly working. I've pushed myself like he has.
I, however, have wised up.
No longer do I work more than 50 hours a week (normally 40). No longer do I take on the world as my own personal responsibility. I have a wife and a child and another on the way. I have my own worries and there isn't an amount of money in the world which would rearrange those top priorities. I make decent coin (less than he claims anyway) and yes I could be making more somewhere where they demand 80 hour weeks and 24 hour on call, but I refuse to do that becuase of my family. My health and my family are not worth it.
There are emergencies, yes. There are times when I do have to run into the shop at 5am to fix something. But those times are few and far between. I get a healthy amount of sleep at night. I play with my children at home. I wear a pager, yes, but it hardly goes off because my network doesn't die when someone plugs in a new computer or trips over a power cable.
If companies require 24 hour 7 day a week tech assistance, then they need to hire multiple techs and have one pager that is circulated between them. "Ok, Ben, this is your week for 24/7" If the network is up and down that much, the network is designed poorly.
Lastly, why the HELL are 200 machines NOT on DHCP?!? If we change ISPs I change one config file and IPs, gateways and DNS are updated for everyone. I change another file and all our web clients are updated. Sounds like his network falls into the "poorly designed" category.
I really do feel for the guy, but there is no reason to push himself / allow others to push him like that. If he's half as qualified as he says, he can get a job ANYWHERE and sleep at night.
Waaa waaa waaa (Score:4)
In the end though, it boils down to one thing. If you don't like it, quit. As you said, you're making multiples of the national average income for someone your age. You could always go sell clothes at The Gap or something. Or take one of those several hundred thousand other open IT jobs at companies that have sufficient technical resources and skills in house not to end up in that sort of a situation. (And a properly designed network architecture shouldn't have nearly the issues in that sort of a switch over... but I'll get to that)
There is a tendancy for people in the industry -- particularly people who are in positions significantly beyond their realistic abilities (I'm not saying this is your case, but A case) -- for people not to stick up for themselves. If you don't like working late hours, don't. Half the time people think they have to, their management really isn't saying that, they're just assuming it. If management IS saying it, then say no. If they fire you, they fire you. If you really have any skills, you'll get another job without any problems, and if you don't, maybe thats what you should concern yourself with.
On the area of mass IP migration, I hope this story serves as a warning to anyone else working in those situations. Its not difficult to engineer your network systems to handle this cleanly. Generate your DNS entries out of a database. Generate a DHCPD configuration file that assigns internal-only IP's for each server, Also out of the database, do the same thing with your server configuration, and IP configuration. Simple scripts to do that. (And you're not using NT for real work are you? You probably could do it with NT anyway, just takes a bit more hacking)
A few days before the switchover, change your SOA's for a near-immediate changeover. Run a query against the database to regenerate your various configuration files, and bring down and back up the networks on the servers. On most systems you won't even need a reboot, and you'll have a few seconds downtime.
I've done provider switchovers at companies with dozens of servers and hundreds of clients no-sweat with less than an hour downtime. If you don't have any other downtimes, you're still doing better than EBay
The mythical MCSE litmus test... (Score:3)
Certifications and degrees do not prove knowledge. I've seen all too many people wandering around with certifications and/or degrees who couldn't config their heads out of a paper bag. Meanwhile, I've met a great many home-grown technicians and engineers who could rewrite your OS from the ground up, even though they had never set foot in a college classroom or even picked up an A+ study guide.
At my last job, we had an 18-year old desktop support technician who knew more about the NT domain at the company than our MCSE-certified Systems Engineer did. Sure, the MCSE could quote Microsoft recommendations all day, but didn't have a single bit of real-world experience behind an NT domain. He couldn't handle the "make due with what we have" philosophy of our small company. He knew what we should have (according to Microsoft) to do what we wanted to do. He didn't know how to operate in the real world of tight budgets and obsolete hardware.
At my current job, I have a coworker who recently went from no certifications to an MCSE in just over 2 months. He's now certified to administer software that he's never even seen outside of shrinkwrap. Today he's also a Microsoft Certified Trainer, and plans to make his fortune training the future MCSEs to pass their tests without ever having to actually touch NT Server.
The biggest problem with certifications is that they hide the qualities that employers really should be looking for behind this "Microsoft says I'm qualified" facade. If you have an MCSE, most employers assume that means you know how to administer an NT domain. Unfortunately, that's not always the case. If you don't have an MCSE, you have to prove that you can admin that domain.
If you have a CS degree, most employers assume that means you spent at least 4 years dedicated to learning your subject matter. The fact is that most college students today spend their 4+ years trying to get a degree while learning as little as possible.
This is not to say that certifications and degrees are worthless. Far from it. While they do NOT prove knowledge, they DO prove dedication. If you're not willing to spend the few months necessary to get your MCSE, how can your employer know that you're willing to spend the time necessary to make his network operate properly?
The point, however, is that certifications alone do not make one great. When it all comes down to the wire, knowledge is what really counts.
The making of a BOfH (Score:3)
But that's not the whole story. When someone fscks up - like giving out the wrong IP addresses, in Morrigan's case - it can cause a lot of headaches. So you blow up at them. I'm known in my Company for exploding over the phone at Telco staff when they give me stupid answers to simple questions: not just because I look down on them, but because their incompetence causes more unneccessary work for me.
Is it any wonder BOfHs are what they are?
"Experts" aren't allowed to have lives :) (Score:3)
No matter what their actual job description is, the person who knows how to fix things when they break will be continually called upon to do so. By everyone. And I do mean continually.
My work-study job was predominantly secretarial/admin. assistant type stuff for a woman who does teacher in-service programs. She's quite intelligent, but knows very little about computers. As part of my "job" on a few different occasions, I got to go to her house and fix her personal computer. Generally, what turned out to be wrong was something fairly simple that was obvious to me, but not to her. (Her idiot ISP had misled her into thinking that 9600 baud was an appropriate speed to attempt to run Netscape on. OOOOOOOPS! Other similar problems had occurred as well.)
There were also the calls from my mother about "how do you get this to work?" She didn't want to ask my father, who knows more about computers than I do, because she didn't want the long technical explanation that she wouldn't understand anyhow, she just wanted it to WORK. I still get those calls.
Even at the other jobs I've held prior to this one, I've been the computer-savvy one and on several occasions had to spend a good piece of my day: explaining that the Budweiser Frogs virus is a hoax (and putting up with a very rude co-worker who said she'd "trust a company vice-president over a bunch of snotty college kids any day." Um, maybe the college kids actually understand computers? No, never! *sigh*), teaching people how to send e-mail, "fixing" various "bugs" directly trace-able to misunderstandings of how the program works, and answering various "how'd you do that?" questions when I had done something like change the type size on icons or the background colors in the CICS screen.
Way to keep me from getting my work done. *grin* And this is 99% end-user stuff. My father, who knows a great deal about how to set up networks, etc. (all self-taught) was really not allowed to have a life. Still isn't, sometimes.
Sometimes I'd go with him. He'd set me up to play games (when I was younger) or get into my Internet account (when I was older), and he'd work on fixing whatever the latest thing to break was. Invariably, we were there for at least an hour later than we were supposed to be. Either the problem would be more complex than he had thought, or someone would see him in the building and start hitting him with questions because they (with good reason) did not trust the actual computer services folks.
But, as far as the "real" tech folk (who have been marginal at best, dangerously incompetent at worst) go at the community college he is a professor at, he's quite unpopular. Things like stumbling across a gaping security hole in the system, pointing it out, and getting reprimanded for trying to poke holes in security. What fun.
The problem is that the faster computers and information get, the more demanding people will become that they STAY that way. And until the industry as a whole has the sense to scream "STOP IT!" in some form or another, this is going to continue. And it is going to continue to get worse. Mad as Mom and I used to get at Dad about this, and much as the stress started to take a toll on his health, I don't think that he was regularly getting only four hours of sleep and/or working 100-hour weeks. (60-80, probably, but not 100.)
I keep hoping that this problem will fade once more people get at least a basic understanding of how systems work, but we have a long way to go before that happens.
OK, this is getting a bit much: (Score:3)
Classic example of what bothers me about a lot of computer experts, right there. Admittedly, it is irritating to deal with folks who don't understand computers when they just won't listen to you.
However, scattered through this and other
Here's a clue, folks. I used to work for an orthodontic school. The department chair is 77 years old, and one of the foremost experts in his particular field (treatment of facial birth defects, especially cleft lip and palate). He's been teaching since the 1950s, written books and journal articles galore, and knows less about computers than I did at the age of five.
Is this man an "idiot"? I hardly think so. People who don't have a high level of proficiency with computers are not stupid, generally speaking. They are either: (a) sufficiently old to have been using typewriters or pen-and-ink for most of their lives, and thus a bit set in their ways; (b) experts in other fields (dentistry, music, early childhood education, what-have-you) who devote a lot of time to their area of expertise and don't have enough left over to become computer gurus; or (c) honestly trying to learn and frustrated by arrogant "experts" who focus on what they are doing wrong and act as if the end-user is wasting the expert's time, not to mention by technology that becomes obsolete practically before it hits the market.
I know it gets frustrating to keep explaining "simple" concepts to someone who doesn't have the same intuitive understanding of computers that most
Hell, by some people's standards, I'm an "idiot." I'm a technical writer, not a programmer.
*steps off soapbox*
Re:student women's groups, GLB groups (Score:3)
Good. Employers like you are exactly the sort I wouldn't want to work for anyhow.
Ah, how quickly you assume what I left off of my resume. One item was work with the campus GLB group, yes. But the other two were a more generalized student-activist group (working out of a state university in NY, which had a wonderful system that could very well get killed by Pataki and co.) and the newly-started-up gaming club.
Ever had to explain to a potential employer that no, you're really not a satanist even though you play AD&D?
reality checks (Score:4)
You are not curing cancer.
You are not saving the world from mass destruction
go home.
Too often, I find that we sysadmins shoot ourselves in the foot by trying too hard to meet a user's requests. We get a project request, bust our ass to complete the project in record time, and please the user community immensely. This is all fine and good until we get another project request, and now we're expected to complete it in record time as well because "you did it once before, why can't you do it again?" Usually, in the first instance, it was not necessary for us to complete our project so quickly. Our users would have probably been happy if it was finished a week or two later, but we delivered if only to demonstrate that we could. But then we've doomed ourselves, because now the user expects miracles to happen; s/he actually makes plans based on the fact that miracles occur on a regular basis. And we chastise them for their naivete, even if we set them up for it in the first place by working hard when we really shouldn't.
Why do we work so hard? Part of it is to keep the high-paying job, but it's mostly because we take some sort of masochistic pride in burning the midnight oil longer than anyone else; working on some component that has been deemed mission critical by someone who has grown too lazy to know how to conduct business with an abacus. And we call this martyr syndrome professionalism.
But in the end, for most of us who work for corporate or academic institutions, what have we accomplished when we finally go home? Some people can receive an e-mail about "How to make $$$$ FAST" in ten seconds instead of ten minutes. Some people can make more money in less time. Some people never notice that anything changed. Their lives go on.
I'm not saying that we should be fat and lazy, but we shouldn't be burning ourselves out when we don't have to. Yes, there will always be projects and network outages and an ever-increasing pile of work that we need to tunnel out of, but no it doesn't have to all be done today. Any project that requires any sort of planning should be done without anticipating anything like overtime. If overtime is required, it has to be for a good reason. Too often, we bitch about having unreasonable project deliverable dates, but that's usually because we just don't know well enough to push back.