> Protection against data loss is done with backups, not RAID.

RAID helps against data loss for some causes of data loss (like hard drives going bad).

However, RAID doesn't protect against human error or software bugs - if you tell a RAID system to delete a file it is deleted - RAID does not mean you can roll back time. If you have a "backup" from a few days ago, if you realize you just destroyed some data with user error, you can use the backup to recover most of the data you just lost.

Just to be absolutely clear - Backblaze does not use RAID inside each pod anymore, we use our own Reed-Solomon encoding across 20 drives in 20 different pods in 20 separate locations inside the datacenter. We open sourced the Reed-Solmon we use here: https://www.backblaze.com/blog... and you can read about how we organize the 20 different pods into a "Backblaze Vault" here: https://www.backblaze.com/blog...

Is it heart breaking to program a robot to do your job? Because that happens much more than H1B replacements.

http://www.wsj.com/articles/meet-the-new-generation-of-robots-for-manufacturing-1433300884

"[The new robots] are nimbler, lighter and work better with humans. They might even help bring manufacturing back to the U.S...."

> will not surrender control to a computer

I think the kaur (the user you were responding to) is wrong, buses and airplanes have windows you can open to watch the interesting and colorful world go by - in addition to window shades if you want to watch Netflix - it will be your choice. But you are also wrong, you already surrender control to a computer when it lands the commercial aircraft you are riding in. You even surrender control to your ABS brakes (occasionally) in your car which make better and faster decisions than you can about which ONE of your four car wheels to brake 10 times a second.

I see a bright happy future where I am actively enjoying the scenery and actively suggesting to the car where to go, but the car will "kick in" and avoid running over a small child or deer in the road faster than my human reflexes could manage. In my 35 years of driving (every day commuter here) I still managed to let my attention waiver once and got in a minor accident (my fault).The average driver gets in 3 or 4 accidents, so I think I'm still "above average" in my driving, but some day a computer will be able to do better than I can in avoiding accidents. I look forward to the help.

I think you guys hit all my favorites except: Threshold. Anybody else with gray hair remember it?

I directly attribute my current career as a software developer to the time in my mid teens that I played (and loved) those Apple ][ video games. I'd rather be lucky than good, and those games coming out just at the right moment in my life luckily pulled me into a ridiculously lucrative career doing something that I love - working with computers.

> By chance do you guys sell the hardware for the storage boxes?

You are in luck! Backblaze does NOT sell the hardware, but we give the design away entirely for free (and others sell it unassembled or assembled for a tiny markup). You can review the latest design here including downloading schematics and specs and parts lists to assemble your own: https://www.backblaze.com/blog...

It sounds like you only want one, and you may not want to worry about assembling it yourself, so you should definitely check out: http://www.45drives.com/ who will sell you a completely assembled storage pod without drives, or may still even sell you a "kit" of the parts that you have to build yourself to save some money.

Backblaze doesn't get anything at all from this, so you might ask why it is all this way. Two things: first of all, we aren't in the business of making and selling hardware, we sell raw storage as a service (our B2 product line) and also we sell online backup. It doesn't HURT Backblaze to release the designs and we get a little free press and good will about it and people hear our name and might want to purchase the OTHER products we actually charge money for. Also, the very nice people at "45 drives" helped us when we were starting out by prototyping our sheet metal and helping with industry cad drawings and such (we were mostly software people, don't know much about manufacturing) so we simply want good things for them. Finally, Backblaze benefits by a larger ecosystem of people using this design. Some of the past improvements have been contributions from OTHER companies and people improving our original design and giving back the improvements.

> if robots are cheaper manufacturing can be brought back home now?

It has been happening for the last few years. More and more manufacturing is coming back to the USA for exactly that reason, but for the same reason there aren't any manufacturing JOBS coming back:

http://www.governing.com/gov-i... "Manufacturing Is Coming Back. Factory Jobs Aren’t."

> What ... is this company doing using consumer hard drives in a ... data center? .... they will fall out of an array every time there's a URE

Brian from Backblaze here. You assume we use RAID (inside of one computer), which is incorrect. We wrote our own layer where any one piece of data is Reed Solomon encoding across 20 different computers in 20 different locations in our datacenter (which is using some of the excellent ideas from RAID and ditching some of the parts that don't work well in our particular application). Our encoding happens to be 17 data drives plus 3 parity. We can make our own decisions about what to do with timeouts. When doing reads, we ask all 20 computers for their piece, and THE FIRST 17 THAT RETURN are used to calculate the answer. Now if one of the computers does not respond at all we send a data center tech to replace it. But if it was just momentarily slow a few times a day we let it be (we don't eject it from the Reed Solomon Group).

> These drives are only meant to be powered on a few hours a day and consumer workload duty cycles

I think a really interesting study would be to power a few thousand drives up once per day for an hour and shut them down. Compare it to a control group of the same drives left on so their temperature did not fluctuate. See which ones last longer without failure. I honestly don't have the answer. (Really, I don't.) What I do know is that Backblaze has left 61,590 hard drives continuously spinning, most of these are often labeled as "consumer drives", and that the vast majority of drives last so long that we copy the data off onto massively more dense drives (like copying all the data off a 1 TByte drive into an 8 TByte drive) not because the 1 TByte fails, but because it ECONOMICALLY MAKES SENSE. An 8 TByte drive takes less electricity per TByte, takes 1/8th the rack space rental, etc. So Backblaze honestly wouldn't care if the "Enterprise Drives" lasted 10x as long in our environment-> we would STILL replace them at the same moment.

Brian from Backblaze here. This is exactly correct. We have redundancy across multiple computers in multiple locations in our datacenter, so losing one drive is usually a calm, non critical event that we take up to 24 hours to replace at our leisure during business hours.

If you are interested in details of our redundancy, here is a blog post about our "Vaults": https://www.backblaze.com/blog...

Summary of article: Backblaze uses Reed-Solomon coding across 20 computers in 20 locations in our datacenter. It is a 17 data drive plus 3 parity configuration, so we can lose any 3 entire pods in 3 separate racks in our datacenter and the data is still completely intact and available.

> Does it really pay off in the long-run to buy lower quality drives?

Disclaimer: Brian from Backblaze here. We use a fairly small, simple spreadsheet to answer that exact question. If Drive A is the same size as Drive B but fails 1% more often, then we might choose the drive that fails at a higher rate if is 2% cheaper, and if it is 10% cheaper it is a slam dunk. Make sense?

You ask about warranty. We enter the warranty information into the simple spreadsheet. If a warranty is 5 years long, then replacement drives are free during that time. If the failure rate is 1% per year, then that warranty is worth exactly 5% to us. If a drive with no warranty at all is 10% cheaper, then it is cheaper. If the drive with no warranty is 2% cheaper then we purchase the drive with the warranty.

In reality, the simple spreadsheet has a few more categories. For example, an 8 TByte Hard Drive takes half the datacenter space rental as two 4 TByte drives and the 8 TByte drive takes about half the electricity of the two 4 TByte drives. So if they were the same price we would obviously choose the 8 TByte drive. But they aren't the same price, so the additional cost of the 8 TByte drive has to be recovered over three years of reduced cabinet space rental costs and reduced electricity costs. We purchase drives once per month, so we get 20 bids from our cheapest suppliers, and right now SOME months Backblaze ends up purchasing the 8 TByte drives because they will pay for themselves within 3 years, and some months we go back to the 4 TByte drives because they are so ridiculously cheap it would take 7 years for the 8 TByte drives to pay for themselves.

This is a video that breaks down the cost of an $80 airline ticket, but I summarize the video below:

https://www.youtube.com/watch?...

It's 10 minutes long, but it breaks down the cost of an $80 airplane ticket as:

$2.50 - fuel (airplanes get a per person fuel efficiency of 104.7 miles per gallon)
$1.50 - crew costs (2 pilots, 4 flight attendants)
$13.50 - airport fee (takeoff fee, landing fee - include using gates, luggage, etc)
$15.60 - taxes (domestic passenger ticket, FAA $4 fee, TSA has $5.60 9/11 tax)
$11.50 - pay for the cost of the airplane amoratized across the flights it will take
$14.00 - airplane maintenance
$10.00 - employees at airline (janitor, benefits, salaries for United employees, etc)
$0.25 - insurance for the airplane
$1.25 - misc (hotel costs for crew, liability insurance, etc)
$10.00 - profit
------------------
Total: $80 for one way airplane ticket

The two things that struck me about that is: 1) fuel was a really REALLY small component, and 2) taxes are the largest single part of the ticket.

> You also need young people to do the work

Robots. We need robots to do the work. The Japanese figured this out a few years ago:

http://www.japantimes.co.jp/ne...

Brian from Backblaze here.

> Do you factor in the work cost?

Yes. And I think the mods were being unreasonable to vote you down, it is a fine question!

We have enough drives (56,000+ all in one datacenter) so that we need a team of 4 full time employees working inside the datacenter to take care of it. If we purchase a drive with higher replacement rates, we will need to hire more datacenter techs, so it gets entered into the equation. ANOTHER area this comes up is server design: most datacenter servers put the drives mounted up front for fast and easy replacement without having to slide the computer around. Our pods put 45 drives accessed through the lid of the pod which means it takes longer to swap the drive - the pod is shut down, the pod is slid out like a drawer, some screws or (most recently) a tool less lid is detached, the drive is swapped, then repeat backwards to put the pod back in service. We did the math, and we feel there is (significant) cost savings that outweighs the additional effort and time to replace the drives. Front mounted (traditional) is something like 1/3rd the drive density with what we have, which means the datacenter space bill would be 3x larger but we would hire fewer datacenter techs.

Brian from Backblaze here.

Sometimes the "drive failure" is as simple as the little circuit board on the bottom of the hard drive has a component die. This won't be predicted by SMART stats at all. We have chatted very informally with the people at "Drive Savers" ( http://www.drivesaversdatareco... ) and they say one of the early steps in attempting to recover the data from a drive that won't work is to replace the circuit board with the board from an identical hard drive of same make and model.

I have no affiliation with "Drive Savers" but from my interactions with them I trust them as quite a good and valuable service who know their craft. We even used them once in a panic once to get back the minimum number of drives for data integrity in a RAID array (a long time ago before our multi-machine vault architecture). It worked - we got all the data back from the drive!

Brian from Backblaze here.

The individual drives in our datacenter run ext4 (the OS is Debian). We do an extremely simple Reed-Solomon encoding that is 17+3 (17 data drives and 3 parity) but the 20 drives are spread across 20 different computers in 20 different locations in our datacenter. This means we can lose any 3 drives and not lose data at all.

We released the Reed-Solomon source code free (open source but even better) for anybody else to use also. You can read about it in this blog post: https://www.backblaze.com/blog...

Disclaimer: I work at Backblaze.

> very unlike the type of use case you will likely see

Being extremely specific - we (Backblaze) keep the drives powered up and spinning 24 hours a day, 7 days a week. So if you leave your drives powered off most of the time and boot them only sometimes, the failure rates we see may or may not be something like yours?

I'm curious if anybody has any other suggested differences with "what you will see". Most of our drive activity is light weight - we archive data for goodness sake, we write the data once then maybe read it once per month to make sure the data has not been corrupted. We stopped using RAID a while ago, so you can't say you need drives that are designed for RAID, because we don't use RAID (we do a one time Reed-Solomon encoding and send it to different machines in different parts of our datacenter and write it to disk with a SHA1 on this "shard" where that shard lives it's life independently without RAID).

ANOTHER POINT MANY PEOPLE MISS -> you can't just pick the lowest failure rate drive and then skip backups!! *EVERY* drive fails, every single solitary last drive. So you must have a backup if you care about the data, you really really do. And if you have a backup, then you are free to choose a drive that fails at a higher rate if there are other considerations such as it is a much cheaper drive. Hint: Backblaze doesn't always choose the most reliable drive, we look at the total cost of ownership including the amount of power the drive will consume and the drive's failure rate and let a spreadsheet kick out the correct drive for us to purchase this month. It is rarely the most reliable drive.