Please create an account to participate in the Slashdot moderation system

 



Forgot your password?
typodupeerror

Comment Re:Well... (Score 1) 377 377

I probably went "scp -r directory new_server:direcory/"

Yeah, the devs had full run of that place. In fact, a LOT of people at that place had root access. To the point where I would go around the place when I'd stay late and pull the damn post-its with the root password off peoples' cube walls.

And yeah, I remember it. Because, first, it WAS a defining moment. And second, I kinda remember a LOT of stuff.

Comment Re:Well... (Score 1) 377 377

> I'm gonna have to go against the chorus and lay the blame at your feet, honestly.

That pretty much HAS been the chorus.

And I never said I wasn't (at least) partially to blame - I definitely had a blind-spot.

Also, had a new sysadmin been hired, he'd have no reason to turn the old machine on. Other people WERE aware of what was going on, including the people who would've trained a new guy. What's more, that machine was leased and would have been returned within a week or two, so he couldn't have repurposed it. And he couldn't have pulled any storage from it because that was part of the lease. And even if he could, it was a Sun box and the new one was an AIX box, so stuff wouldn't just run.

And here's another thing... say the new guy gets hired, never touches that machine because he's been told it's being returned in a few days. And then one of the devs turns it on and a week's work gets erased. They would've still blamed the new sysadmin even though he had nothing to do with it.

If you want funny, I actually knew the guy who ended up replacing me through a local Linux user's group. I know he was plagued by the same kind of crap. He tried to update the remote connections to use ssh rather than telnet and almost got fired for THAT.

This was not a good SysAdmin environment.

Comment Re:Well... (Score 1) 377 377

As I've mentioned, this was about 20 years ago, so I can't really remember it 100%.

However, this was one of those shops that started with about 10 employees, and even though by then it was 200+, it still operated as if it was a small, small company. The head devs were part of the original 10, and they were like gods. They had full access to EVERYTHING. Including root access to all the servers. They were basically allowed to do whatever they wanted.

If something went wrong where they and someone else was involved, it was never their fault.

Comment Re:Well... (Score 4, Interesting) 377 377

You know the old saying, "make something idiot-proof and someone will come up with a better idiot."

They'd have plugged it back in. Again, the guy physically went into the server room and pushed a button.

I certainly should've disabled the cron job or, better yet (as pointed out by AC down there) have known what rsync actually was and used that - I know I said I did in the original post but in retrospect I couldn't have as it wouldn't have overwritten everything. This was about 20 years ago...

Comment Re:Well... (Score 2) 377 377

They weren't supposed to, but the head developers were like gods at that place. They had the root passowrds and I wasn't allowed to restrict them in any way.

It stemmed from them being among the original 10 people when the company started, and even though the place was now a 200+ employee organisation, in some ways they still ran it like 10-person operation.

I did vocally complain about this. They quite often went in and overrode stuff I did.

Comment Re:Well... (Score 1) 377 377

Good call... this was about 20 years ago, and it's not likely that I used rsync (not sure I knew how to do that back then).

My memories of the event are not... perfect. But it's likely that I just used scp to dump entire directories. Couldn't have been using rsync because, as you say, it wouldn't have one as much damage.

Comment Well... (Score 4, Interesting) 377 377

I don't know what monetary cost they assigned to this, but this is the one I got in the most trouble for.

Frankly, it was something I got blamed for. I guess I can take partial responsibility. You guys tell me.

I was the only UNIX guy at this place. We were moving our Main Internal Server to a newer machine. I had set up a cron job to rsync all user data nightly, so that when we transition over the rsync would be faster.

So, the big day comes. I come in on a weekend, do the final rsync, change some DNS entries, shut down old machine, bring new machine up. No problem.

Next day everyone is working happily, everything is working smoothly, no worries.

Or so I thought. Turns out the main developer wanted something off the old server, so he turned it back on to copy his files... and then left it up.

So, during the night, the thing automatically rsyncs and overwrites an entire day's work for about 80 people.

Definitely partially my fault for not disabling the cron job, but I was the only one who got in any kind of trouble at all for this (to the extent of almost losing my job, and frankly that was the catalyst for me leaving that place).

Old mail has arrived.

Working...