Slashdot is powered by your submissions, so send in your scoop


Forgot your password?

Comment Re:It's not just open source projects (Score 1) 140

It's a good point but following some sort of vaguely sensible process and best practices (as nebulous as that can be) doesn't cost any money. Like, it doesn't cost any money to create role based accounts for service/license management, bring up systems in a clean repeatable way, implement monitoring etc. Most of my career has been around open source software which is free licensing fee-wise. I think you need qualified engineers regardless if it's OSS or not so the argument of "we need vendor support!" never really flies for me, outside of hardware which does fail.

The purchasing stuff I agree is a big minefield. I think the best we can do as engineers is give them the whole picture. Present the options, the risks, and let them choose. Be as clear as possible and get them sign off on the risks if they choose crappy solutions to save money. Then you can hold them accountable when stuff melts down the way you said it would. If you're lucky, over time you can develop a reasonable working relationship with the business side of the house and there can be mutual trust. Makes life much easier. It's easy to see certain groups as "the enemy" (I'm certainly guilty, depending on which hat I'm wearing on that particular day) but the truth is Hanlon's Razor is a real thing and most people want to at least do a reasonably decent job. Most of the real problems are people related not technology. Sometimes a simple "hey, we're going back and forth on this a lot; I don't fully understand where you're coming from. Want to grab a coffee and you can explain it?" goes a long way.. vs "they're dumb/bad/the enemy and I will 'win' this battle". Hell, they might even be bad but you still gotta find some way to work it out; massive hostility doesn't make for enjoyable workplaces or lives - considering how much time is spent working.

Comment Re:It's not just open source projects (Score 1) 140

> How does that sort of shit even happen? Who the hell promoted this person? Why?

A lot of the problem is the business side almost always knows nothing about the tech side or (bigger problem) who to trust and who is full of it. They're generally the sorts who can barely keep their own workstation under control. So you run into a few problems.

1) There's a shortage of qualified tech people who have been around long enough to avoid some of the common problems
2) The people ultimately responsible for hiring and business continuity don't know the difference between a good techie and someone who just fabricated most of their resume
3) Incompetent people tend to prefer to hire other incompetent people
4) Qualified people with experience can usually sniff out the hellholes during the interview process and GTFO

Specific to this company, they had extremely weak management. They had a profitable product that required very little actual engineering.. it was a mobile service but the app actually was self contained on the handset.. the only reason they hit our webservice was for billing and account related activities.

They hired the clown via contract from a consulting company to be their Windows/IT guy. Then their service was expanded to multiple carriers and they needed a real datacenter; because this guy was already there, they asked him if he had the ability to build out the DC. So suddenly this low tier windows guy has an opportunity to be an ops/infrastructure manager and add all sorts of fun stuff to his resume; he jumped on it. None of the people had any grownup level ops experience to evaluate his skillset or vet his plans/ideas.

After they had the DB meltdown I mentioned earlier they realized they needed a real ops person. The startup I was at was dying and a friend I worked with before (a business analyst SQL person) suggested me. They also wanted to put their product in teh clowdz; this was 2008-2009, the startup I was at was a pretty early AWS pioneer. (Pre-EBS, pre-ELB.. it was the wild west!).

Anyway I got in and it was a mess, I fought to help them clean stuff up and to avoid problems before they happened. The devs appreciated my efforts but due to the aforementioned weak management, the monkey dude was actually technically a manager? and I was just an engineer? Or something. So while technically I was "in charge" of production he still kept his claws in enough that he'd fight with me over every decision. I finally gave up and got a new gig when it became clear the situation wasn't going to change. Basically it came down to - I'm used to cleaning up peoples messes but I'm not going to fight for the right to clean up other peoples messes.

Ironically when I put in notice they actually offered to fire that dude on the spot if I'd stay. That made me doubly sure I made the right decision to leave; you should never put an employee in that situation to make that kinda choice. He was let go ~ 3 months later after they hired a replacement for me (they actually paid me contract to interview people for them). They're still around (and profitable!) despite themselves; most of the original people are gone due to a series of acquisitions.

Anyway I have a bunch of stories.. I've been doing internet startups for the last 18ish years. I like some aspects of them but sometimes I go park at a big company for a while to recharge. Maybe someday I'll write a book. I actually collect some of the funnier/more absurd stuff that goes on at the various companies. Mostly for my own personal amusement (and because other people wouldn't believe it otherwise). I anonymize them though.

A lot of the kids coming into syseng/devops/dev aren't necessarily bad, they just are inexperienced and have to learn the same lessons. That's the part that wears on me sometimes.. but if people are willing to take advice/mentoring from old timers(heh.. not even 40) then it's all good.

There has been a few times in my career I worked with lots of smart people (smarter than me) and it was a blast. Unfortunately that never seems to last all that long.

Comment Re:It's not just open source projects (Score 4, Interesting) 140

I guess. Lately I think the real difference between a seasoned engineer and a "senior" engineer is just taking that extra 10% of time to do things vaguely sensibly up front. There is no such thing as temporary. At the least, set things up so refactoring them later doesn't require a total redo.

This guy in particular was just in way over his head but one of those sorts who is paranoid about admitting that or asking for help. In fact he was hostile to help. Not really part of my original point but he'd do stuff like;

- ignored my advice to not tie production services into corporate domain (if you ever get sold/acquired etc, you understand)
- ignored my advice to not create a "split domain" with the corporate domain (eg the windows domain was, the windows dns servers thought they were SOAs but that same domain had actual internet resolvers with different records)
- refused to entertain the notion that linux was production ready (this was in 2009) and forced solaris as a standard. On x86. As vmware VMs.
- refused to take any help or assistance in installing the base OS despite being a windows guy with zero unix knowledge. We ended up with stuff like DB servers that had 2x swap as ram.. and they had 128G ram..
- For some odd reason was very hostile to the notion of service/host monitoring.. like.. not just against nagios but _anything_

The list goes on and on.

He was just really promoted way above his experience level as happens in startups; they hired me probably 8 months after him, when production databases had been wiped and backups hadn't been successful for months (back to the no monitoring thing).

It took a bunch of years to fully undo all the crap he had put in place. I danced a jig when I closed the lights on the datacenter he had built (we migrated). Did I mention in that datacenter, he setup "redundant" switches and firewalls for the servers.. but had all the internet drops coming down into one single unmanaged 1G cheapie netgear entry level switch?

If he had allowed me to help I bet he'd still be working there. I have no problems mentoring people as long as they're not asshats. Last I heard he was in law school after a stint in real estate..

Comment It's not just open source projects (Score 5, Informative) 140

We almost lost our production domain. The original dummkopf who set things up registered it all under his own name and individual email instead of using a role based account. He then was fired for unrelated incompetence. Fast forward to the domain renewal coming up.. charge went to his personal CC.. he disputed the charges.. we would have lost it except by pure dumb luck I was in the middle of a DNS migration project and was auditing/cleaning up the registrar details. It was as last minute as you'd want; expiration was within 12h.

One of my pet peeves - people who register for services or get licenses tied to their individual accounts.

Comment Impartial referees (Score 3, Insightful) 100

Pretty much any set of algos is going to be easily defeated by humans trolling and no system is going to be anything near perfect. My thoughts;

1) Create a small set of simple, concise rules that are inviolate
2) Have a system so people can mark submissions as good (no rules broken/useful) or bad(rules broken)
3) Have your referees do nothing but determine if that submission is breaking one of your rules
4) Based your user trust as a derivative as how the user voted compared to what the referee votes

The theory is any controversial submission is going to get flagged & referees attention. Their job is limited in scope to just determining if the post breaks the site rules or not, nothing to do with quality / content / opinion. If users are trying to game the system their votes are going to conflict with the referees so their user trust is going to go down, whereas if people agree their trust is going to go up.

Eventually you'll have a group of users that you can generally trust to do the right thing so you can weight their actions accordingly.

Obviously there are some weaknesses;

- Referees are pretty much god (that's why the scope of their power is extremely narrow and simple)
- You can end up with hive mind (though you can combat that if enough trusted users conflict with other trusted users). I'd argue it's a way better protection than pure crowdsourcing ala reddit where the demographics crush submissions into hivemind

Just tossing that out there off the top of my head. It's not something to replace automated reputation management, just something augment it and limit some of the abuse.

Comment Re:Issue is more complicated (Score 4, Insightful) 928

Would you treat coworkers like that? In meatspace? If someone called me a fuckwad in a way that wasn't obviously jokey/ironic it would be a huge problem to me. Being nice and civil is a much better way to get things done. Like, grumpy/curmudgeonly is kinda par for the course but that's totally different.

I used to work with a really toxic dev that for whatever reason our management dept protected. He was extremely misogynistic and refused to work with any women. One time in a meeting he said something basically like "it's in the spec, you stupid bitch" (as an aside, it wasn't in the spec ;p). He used to get away with yelling at people etc. Then I got sucked into the daily meetings and said basically "there is no call to raise your voice in a meeting at work. if you have a personal problem with me, we can go outside and talk about it". Never had a problem with him after that - unsurprisingly most misogynists and bullies are in reality cowards.

Comment Grandstanding nonsense. (Score 1) 488

Does anyone believe that the NSA shared any details or scope about their hardware purchase? Or any of their vendors?

"Hi we're building out datacenters so we can do some probably illegal data snooping, can you help us?" I'm sure an agency that is cloaked in as much secrecy as possible goes around sharing that sort of information. Especially without gag orders.

Not only is she a massive tool she's obviously full of it. She needs a gag, it's an order.

Comment Great - except for one thing (Score 2) 133

The business side is why the company exists. When they add feature creep etc, it's generally because they don't really know what the customer wants and are trying to see what lands. They don't understand the cost of changing the design / refactoring. They tend to not even really understand how to tell if a time estimate is BS or not.

It comes down to trust and working relationships. The business side almost never bridges the tech side - it's up to tech folks to bridge that gap and help them understand. Often times they simply won't care.

Sucky but that's the way the industry generally works. There are a few bright spots but they're few and far between. However the attitude of "I'm going to be a lone hero and push this out!" is just setting yourself up for more frustration and failure. There's a quote - "in writing, you must kill your darlings". Same thing applies to softdev, be prepared to write elegant code you are proud of, only to have it rot away and disappear. Your options are basically;

1) stay professional, do your job, collect your paycheck
2) try to find some startup with ideals like fogcreek (when it comes to valuing individual developers)
3) simplify your lifestyle and financial requirements and write code for your own projects, do a little contracting or take a job in a different field

Comment Re:I'm spending 60% of my monthly income on rent (Score 1) 940

The 3x rent as salary thing is fairly new and common with newer buildings. I've been a renter for the last 15 years in a few different large cities; it used to be largely based on your credit score and rental history. They would work with you if you had a strong rental history but marginal credit.

In my current apartment (which is fairly upscale/new) they did the 3x rent/income, a credit check for any late payments and that's it. They didn't care about my 15 year rental history with details.

Comment Really only two main reasons I can think of.. (Score 1) 118

1) Many hardware vendors, especially storage, will give you trouble or refuse to support their gear if you're running an OS they don't bless

2) It can make business continuity easier; you'll find more people who claim RHEL/CentOS support than Debian (though Ubuntu is pretty wide-spread these days).

The rest are pretty minor. Some C level tools want to feel like they're magically protected by vendor support.

The reality is I've pretty much always had to do my own diagnostics, RCA and come up with a resolution. If there's no patch in a RHEL blessed kernel but there is in mainline, I'd just as soon pull it in and patch it myself, which is way easier to do IMO using something like Debian, or write a patch myself if necessary.

So I'd say if you're having to deal with a lot of hardware support it might be worth it just to get less pushback but if you're mostly cloud based or just have a few machines go with whatever you think rocks (unless you suck, but you would never realize that :P)

At one shop we all wanted to use Debian but the vendor only "supported" suse/rhel etc.. so we went so far as to modify uname and whatnot to pretend we were a redhat shop ;)

Slashdot Top Deals

When a fellow says, "It ain't the money but the principle of the thing," it's the money. -- Kim Hubbard