Catch up on stories from the past week (and beyond) at the Slashdot story archive

 



Forgot your password?
typodupeerror

Comment Re:If it's working for them (Score 1) 209

Brian from Backblaze here.

> What's the typical drive temperature in Backblaze's cases in their environment?

Short answer: the coolest drives are 21.92 Celcius and the hottest drive was 30.54 degrees.

I wrote this up above in response to a temperature question, copy and pasted here. The raw data dump from Backblaze includes drive temperatures as reported by "smartctl". You can find a complete set of historical data of all drive temperatures in the Backblaze datacenter here: https://www.backblaze.com/b2/h...

We analyzed the failures correlated with temperature in this blog post in 2014: https://www.backblaze.com/blog...

In a conversation with some of the Facebook Open Storage people, they said hard drives have increased failure rates at extremely high temperatures (somewhere up near 40 degrees Celcius) but our drives never get anywhere NEAR the temperatures required to correlate with failures. We monitor every drive for temperature, taking readings once every 2 minutes, and we have had situations where the drive temperatures caused our internal warning alerts to go off (well below those catastrophic levels Facebook saw failures at). When we go to investigate, the most common cause of rising pod drive temperature is that some of our fans in that pod have died. We used to have 6 gigantic fans to keep it cool, but we reduced it to 3 with no increase in drive temperature. If one of the fans dies it doesn't get warm enough to set off any alerts, but if 2 out of 3 fans die it can't move enough air to keep the pod within reasonable operating temperatures. We don't monitor the fans directly, but drive temperature has been such a good proxy for it we don't feel any pressing need to figure out how to monitor the fans.

Comment Re:High failure rate (Score 2) 209

Brian from Backblaze here.

> I think their pods only have GigE interfaces

Originally (up until 3 years ago) that was true, but all new pods have 10 GbE interfaces, and 100% of the pods in our "Backblaze 20 pod Vaults" have 10 GbE interfaces. And there are some really strange (and wonderful) performance twists on using 20 pods to store each file: when you fetch a 1 MByte file from a vault, we need 17 pods to respond each supplying only 60k bytes to reassemble the complete file from the Reed Solomon. So the actual bandwidth when fetching just one medium size file can reach more like 170 Gbit/sec theoretical bandwidth. However, if you tried to fetch ALL the files from a pod all at once, the raw 7200 RPM drive performance is our current limiting factor.

Here is a link to a blog post on the 20 pod Backblaze Vault architecture: https://www.backblaze.com/blog...

Here is a link to the Reed Solomon encoding we open sourced that we use on the 20 pod Vaults: https://www.backblaze.com/blog...

Comment Re:High failure rate (Score 2) 209

Brian from Backblaze here.

> I also wonder if we'll ever get numbers from Backblaze on things like the actual temperature ... power these drives lived through.

The raw data dump includes drive temperatures as reported by "smartctl". You can find a dump here: https://www.backblaze.com/b2/h...

We analyzed the failures correlated with temperature in this blog post in 2014: https://www.backblaze.com/blog...

In a conversation with some of the Facebook Open Storage people, they said hard drives have increased failure rates at extremely high temperatures but our drives never get anywhere NEAR the temperatures required to cause failures. We monitor every drive for temperature, taking readings once every 2 minutes, and we have had situations where the drive temperatures caused our internal warning alerts to go off (well below those catastrophic levels Facebook saw failures at). When we go to investigate, the most common cause of rising pod drive temperature is that some of our fans in that pod have died. We used to have 6 gigantic fans to keep it cool, but we reduced it to 3 with no increase in drive temperature. If one of the fans dies it doesn't get warm enough to set off any alerts, but if 2 out of 3 fans die it can't move enough air to keep the pod within reasonable operating temperatures. We don't monitor the fans directly, but drive temperature has been such a good proxy for it we don't feel any pressing need to figure out how to monitor the fans.

Comment Re:High failure rate (Score 1) 209

Brian from Backblaze here.

> Perhaps they don't keep the temperature as cool as they should in order to save a few bucks?

The colocation datacenter is SunGard in Rancho Cordova California and there are other tenants. I assume the temperature of the datacenter is industry standard? But even better, in the raw data dump it includes all the temperatures of all the hard drives, so you (or anybody) could check the correlation. We looked into it in 2014 and didn't find much correlation between temperature and hard drive failure as long as we kept the temperature of any one hard drive well below a tipping point (which we do). Here is the blog article and stats behind our analysis: https://www.backblaze.com/blog...

Comment Re:Reliability (Score 1) 209

> Protection against data loss is done with backups, not RAID.

RAID helps against data loss for some causes of data loss (like hard drives going bad).

However, RAID doesn't protect against human error or software bugs - if you tell a RAID system to delete a file it is deleted - RAID does not mean you can roll back time. If you have a "backup" from a few days ago, if you realize you just destroyed some data with user error, you can use the backup to recover most of the data you just lost.

Just to be absolutely clear - Backblaze does not use RAID inside each pod anymore, we use our own Reed-Solomon encoding across 20 drives in 20 different pods in 20 separate locations inside the datacenter. We open sourced the Reed-Solmon we use here: https://www.backblaze.com/blog... and you can read about how we organize the 20 different pods into a "Backblaze Vault" here: https://www.backblaze.com/blog...

Comment Outsourcing is so 2006 - robots are 2016 (Score 1) 482

Is it heart breaking to program a robot to do your job? Because that happens much more than H1B replacements.

http://www.wsj.com/articles/meet-the-new-generation-of-robots-for-manufacturing-1433300884

"[The new robots] are nimbler, lighter and work better with humans. They might even help bring manufacturing back to the U.S...."

Comment Re:Unsurprising (Score 5, Insightful) 441

> will not surrender control to a computer

I think the kaur (the user you were responding to) is wrong, buses and airplanes have windows you can open to watch the interesting and colorful world go by - in addition to window shades if you want to watch Netflix - it will be your choice. But you are also wrong, you already surrender control to a computer when it lands the commercial aircraft you are riding in. You even surrender control to your ABS brakes (occasionally) in your car which make better and faster decisions than you can about which ONE of your four car wheels to brake 10 times a second.

I see a bright happy future where I am actively enjoying the scenery and actively suggesting to the car where to go, but the car will "kick in" and avoid running over a small child or deer in the road faster than my human reflexes could manage. In my 35 years of driving (every day commuter here) I still managed to let my attention waiver once and got in a minor accident (my fault).The average driver gets in 3 or 4 accidents, so I think I'm still "above average" in my driving, but some day a computer will be able to do better than I can in avoiding accidents. I look forward to the help.

Comment Re:TIme flies (Score 1) 97

I think you guys hit all my favorites except: Threshold. Anybody else with gray hair remember it?

I directly attribute my current career as a software developer to the time in my mid teens that I played (and loved) those Apple ][ video games. I'd rather be lucky than good, and those games coming out just at the right moment in my life luckily pulled me into a ridiculously lucrative career doing something that I love - working with computers.

Comment Re:Is cheaper really better? (Score 1) 130

> By chance do you guys sell the hardware for the storage boxes?

You are in luck! Backblaze does NOT sell the hardware, but we give the design away entirely for free (and others sell it unassembled or assembled for a tiny markup). You can review the latest design here including downloading schematics and specs and parts lists to assemble your own: https://www.backblaze.com/blog...

It sounds like you only want one, and you may not want to worry about assembling it yourself, so you should definitely check out: http://www.45drives.com/ who will sell you a completely assembled storage pod without drives, or may still even sell you a "kit" of the parts that you have to build yourself to save some money.

Backblaze doesn't get anything at all from this, so you might ask why it is all this way. Two things: first of all, we aren't in the business of making and selling hardware, we sell raw storage as a service (our B2 product line) and also we sell online backup. It doesn't HURT Backblaze to release the designs and we get a little free press and good will about it and people hear our name and might want to purchase the OTHER products we actually charge money for. Also, the very nice people at "45 drives" helped us when we were starting out by prototyping our sheet metal and helping with industry cad drawings and such (we were mostly software people, don't know much about manufacturing) so we simply want good things for them. Finally, Backblaze benefits by a larger ecosystem of people using this design. Some of the past improvements have been contributions from OTHER companies and people improving our original design and giving back the improvements.

Comment Re:Interesting (Score 2) 415

> if robots are cheaper manufacturing can be brought back home now?

It has been happening for the last few years. More and more manufacturing is coming back to the USA for exactly that reason, but for the same reason there aren't any manufacturing JOBS coming back:

http://www.governing.com/gov-i... "Manufacturing Is Coming Back. Factory Jobs Aren’t."

Comment Re:ST3000DM001? In a DATA CENTER? (Score 5, Informative) 130

> What ... is this company doing using consumer hard drives in a ... data center? .... they will fall out of an array every time there's a URE

Brian from Backblaze here. You assume we use RAID (inside of one computer), which is incorrect. We wrote our own layer where any one piece of data is Reed Solomon encoding across 20 different computers in 20 different locations in our datacenter (which is using some of the excellent ideas from RAID and ditching some of the parts that don't work well in our particular application). Our encoding happens to be 17 data drives plus 3 parity. We can make our own decisions about what to do with timeouts. When doing reads, we ask all 20 computers for their piece, and THE FIRST 17 THAT RETURN are used to calculate the answer. Now if one of the computers does not respond at all we send a data center tech to replace it. But if it was just momentarily slow a few times a day we let it be (we don't eject it from the Reed Solomon Group).

> These drives are only meant to be powered on a few hours a day and consumer workload duty cycles

I think a really interesting study would be to power a few thousand drives up once per day for an hour and shut them down. Compare it to a control group of the same drives left on so their temperature did not fluctuate. See which ones last longer without failure. I honestly don't have the answer. (Really, I don't.) What I do know is that Backblaze has left 61,590 hard drives continuously spinning, most of these are often labeled as "consumer drives", and that the vast majority of drives last so long that we copy the data off onto massively more dense drives (like copying all the data off a 1 TByte drive into an 8 TByte drive) not because the 1 TByte fails, but because it ECONOMICALLY MAKES SENSE. An 8 TByte drive takes less electricity per TByte, takes 1/8th the rack space rental, etc. So Backblaze honestly wouldn't care if the "Enterprise Drives" lasted 10x as long in our environment-> we would STILL replace them at the same moment.

Comment Re:Is cheaper really better? (Score 5, Informative) 130

Brian from Backblaze here. This is exactly correct. We have redundancy across multiple computers in multiple locations in our datacenter, so losing one drive is usually a calm, non critical event that we take up to 24 hours to replace at our leisure during business hours.

If you are interested in details of our redundancy, here is a blog post about our "Vaults": https://www.backblaze.com/blog...

Summary of article: Backblaze uses Reed-Solomon coding across 20 computers in 20 locations in our datacenter. It is a 17 data drive plus 3 parity configuration, so we can lose any 3 entire pods in 3 separate racks in our datacenter and the data is still completely intact and available.

Comment Re:Is cheaper really better? (Score 5, Informative) 130

> Does it really pay off in the long-run to buy lower quality drives?

Disclaimer: Brian from Backblaze here. We use a fairly small, simple spreadsheet to answer that exact question. If Drive A is the same size as Drive B but fails 1% more often, then we might choose the drive that fails at a higher rate if is 2% cheaper, and if it is 10% cheaper it is a slam dunk. Make sense?

You ask about warranty. We enter the warranty information into the simple spreadsheet. If a warranty is 5 years long, then replacement drives are free during that time. If the failure rate is 1% per year, then that warranty is worth exactly 5% to us. If a drive with no warranty at all is 10% cheaper, then it is cheaper. If the drive with no warranty is 2% cheaper then we purchase the drive with the warranty.

In reality, the simple spreadsheet has a few more categories. For example, an 8 TByte Hard Drive takes half the datacenter space rental as two 4 TByte drives and the 8 TByte drive takes about half the electricity of the two 4 TByte drives. So if they were the same price we would obviously choose the 8 TByte drive. But they aren't the same price, so the additional cost of the 8 TByte drive has to be recovered over three years of reduced cabinet space rental costs and reduced electricity costs. We purchase drives once per month, so we get 20 bids from our cheapest suppliers, and right now SOME months Backblaze ends up purchasing the 8 TByte drives because they will pay for themselves within 3 years, and some months we go back to the 4 TByte drives because they are so ridiculously cheap it would take 7 years for the 8 TByte drives to pay for themselves.

Comment Re:I weep for the airline industry. (Score 1) 655

This is a video that breaks down the cost of an $80 airline ticket, but I summarize the video below:

https://www.youtube.com/watch?...

It's 10 minutes long, but it breaks down the cost of an $80 airplane ticket as:

$2.50 - fuel (airplanes get a per person fuel efficiency of 104.7 miles per gallon)
$1.50 - crew costs (2 pilots, 4 flight attendants)
$13.50 - airport fee (takeoff fee, landing fee - include using gates, luggage, etc)
$15.60 - taxes (domestic passenger ticket, FAA $4 fee, TSA has $5.60 9/11 tax)
$11.50 - pay for the cost of the airplane amoratized across the flights it will take
$14.00 - airplane maintenance
$10.00 - employees at airline (janitor, benefits, salaries for United employees, etc)
$0.25 - insurance for the airplane
$1.25 - misc (hotel costs for crew, liability insurance, etc)
$10.00 - profit
------------------
Total: $80 for one way airplane ticket

The two things that struck me about that is: 1) fuel was a really REALLY small component, and 2) taxes are the largest single part of the ticket.

Slashdot Top Deals

Any sufficiently advanced technology is indistinguishable from a rigged demo.

Working...