I picked up a large base of C code of a guy who would only code when he was drunk. It was the biggest mess I ever saw, except a small part that was commented, "I did this sober because it had to be fixed yesterday..."

In his case mind altering substances helped, but he had no business coding to begin with.

I had the misfortune of meeting one of his team mates who was consulting for the firm to "bring us up to speed." 60's throwback begins to describe him, which explained the other mass of rambling code I had to deal with. You could tell, by the names of functions and variables, when he had the munchies/giggles and it got dark when he was paranoid. Humorous to go through, terrible to maintain. Oh yea, did I mention this software was operating networks of ATMs, as in peoples money?!?

Yea, it's called Naquadah. Our ancient ancestors strip mined Sol of all of it! Now we are stuck with oil, and attempting to go green :(

They were on a pretty good UPS system connected to a GFI breaker. The room was climate controlled so unless something very weird happened I don't think electrical or environmental were an issue.

I couldn't agree more... But what's best often meets real world. It was a skunkworks project with no budget. It was amazing we got things working the way we did and the results got the attention it needed and then the resources were allocated.

The server was the old 737(?) pin first gen amd64 bit system. 64 bit Gentoo linux with software raid running the 5 SATA 80GB Seagate HDs. 2Gb of ram for a DB of 150GB of which 80% of the data was accessed on a daily basis... It was CRAZY project put together with the lowest of budget that achieved results good enough to actually get resources allocated rather than "it's good! keep it up!"

Gotta love the reactions on /.

That and the fab process is so precise that a fault is replicated so precisely that after 90 days of 24/7 operation they all failed within 24 hours, 4 failing in 8 hours. So it was engineered bad luck!

Anyway I glad those days of system admin are behind me, I'm with my passion now which is HPC C++ development. Those experiences stuck with me and give me much more respect for the admin of the HW I now use. It's funny and sad to watch their expressions when I talk to them intelligently and with respect. It's like they've never had that happen before.

[Sarcasm]Nothing like 20/20 hindsight... If I had done anything like trying to rebuild the array it would have fallen apart... Oh wait... If I had followed what you suggested I would have been SCREWED.[/Sarcasm]

I made a decision based on what on the information on hand.. The rebuild would have take more than a few hours, 80GB disk was SLOW, i.e. first gen SATA. By executing the DB dump I was hitting less than 1/2 the disk capacity on read than 100% disk capacity on a write. It would be significantly faster to retrieve the data than to rebuild. That time window was critical, 2 hours of read vs 4+ hours of write. I also knew I had all the data on hand and all the scripts tested monthly for rebuilding the entire DB on a different server. The decision was easy! Grab the DB data now, redeploy on another system and address the issue on the spot. The system ended up being down 3 hours rather than 24+.

Secondly The failure was abrupt with no SMART messages, I couldn't trust the others to not have the same non-reporting issues. I made a choice on the spot on how to proceed knowing full well I may have signed my own 24h torture warrant. Fortunately I didn't have the worst case happen and I learned a critical lesson.

A bit more information...

+- 30 minutes on each one
First disk failed...
2 hours later second disk failed...
2 hours later third disk failed.
2 hours later 4th disk failed
16 hours later 5th disk failed.

Never paranoid enough when dealing with data! I had a RAID 5 (5 disks) of Seagate 80GB SATA disks; 4 failed within an 8 hour window, the 5th failed within 24 hours of the first; this was 3 months after purchase. It was a HUGE PITA. First drive failed and I started an immediate DB dump to an NFS mount. 20GB and 2 hours later the second disk failed and RAID was dead. I ran the other three disks just to see what would happen...

I will NEVER, EVER run two storage medium (Spinning platter, SSD, ...) from the same lot in the same RAID ever again. I was saved by 20 minutes, in the above situation, from 24 hours of hell.

They aren't that stupid, they just choose to be! There was a /. article a few months back that showed that giving evidence that contradicted someone's beliefs had the effect of reinforcing their beliefs. That on top of that you have many that just don't care, don't understand, or just want to be distracted. They exist on both sides.

Based on what is readily available, linking the ice melt in the north to global warming is incorrect. This does not mean there is no global warming, I personally believe the earth is still warming from the mini-ice age that just ended http://en.wikipedia.org/wiki/Little_Ice_Age.

Below are reports on what is going on. Both state facts that can be shown to draw separate conclusions. The really interesting thing is we are past the 2nd standard deviation for antarctic ice growth, which is exceeding the amount of ice lost so we are in a net positive. Just try to explain this to the average Joe and watch them lose interest really fast! Use a car analogy and you still don't get anywhere. Once evidence is shown that seems to conflict most humans ignore it because understanding the complexity exceeds the effort to survive the next week.

Earth Loses Its 'Air Conditioner': Arctic Ice Cap Shrinks to Record Low Level
http://www.pbs.org/newshour/bb/science/july-dec12/icemelt_09-20.html
http://nsidc.org/news/press/2012_seaiceminimum.html

Polar sea ice could set ANOTHER record this year
Exceptionally large amounts of it down south right now
http://www.theregister.co.uk/2012/09/21/arctic_antarctic_sea_ice_record/

[sarcasm] Scientists first observed global warming in 1895. Then in 1920 they said it was global cooling. Then in 1935 they said there was global warming, but then in 1975 they said it was the verge of a new Ice Age but then it became global warming again. But that is all old news. Let's stop talking about discredited work... [/sarcasm]

From generation to generation people have heard so much about global warming and global cooling that they don't believe what is being said now. "Back in my day the world was cooling and the US was going to be covered in ice in 50 years!" Kids grew up hearing that and those kids now have heard from their (grand)parents the opposite of what is being said now. So science was wrong before it is wrong now so give me my iPhone 7SSS!

Also we have become numb to almost everything due to the massive bombardment by the media of anything and everything. From the most important, the Kardashians, to the least, Global Warming. Oh yea and something about our embassies being attacked, somewhere in a desert...

The fault lies with us, as a population, not wanting to deal with what isn't going to affect us in the next week or two (oddly the time between most paychecks.) The masses are incurably ignorant. In any group large enough, most are idiots! So we continue to consume a scarce resource in moving about back and forth to the mall and think that consuming 2x as much to produce the equivalent in "bio-fuel" which is then consumed to go to the mall is "green."

The blind following the blind following the def.

I'm just in a bad mood today so take that into account.