Amazon's New Storage Service

dlaur writes to tell us that today Amazon announced their Simple Storage Service (S3) allowing users to store unlimited amounts of data at $0.15 per GB paid monthly. From the article: "S3 was purportedly built to support both Amazon's own internal applications and the external users of the Amazon Web Services platform. That should be proper motivation to build a service that's fast and robust enough for mission critical use, yet flexible enough to support any storage task thrown at it."

Encryption

I wonder how long it'll take to build a backup solution that encrypts your data locally with a private key before sending it off to amazon. That way they wouldn't be able to look through it, and at 15c/month/gig it'd be pretty affordable for home backups

Re:Encryption

Current hard disks sell for 40c/gig. If you plan to keep your data for >3 months, it is more economical to use you harddisks. Though your solution has other advantages (data accessible from anywhere, no need to administer yourself your servers, etc).

Re:Encryption

But hard disks present their own challenges.

You should really use some sort of redundant arrangement to make sure that a failure of your backup device wont result in data loss.

You then need to either offsite the drives or keep them in a firesafe, in which case you probably need two sets of them so that you can keep one live and the other somewhere safe.

And of course the amazon solution leaves your data accessiblwe from anywhere.

Re:Encryption

First, hard drives are a very poor choice for backups. In addition to media failure, you also have serious problems with failure of the electronic hardware. There are just too many single points of failure. Tape is far superior.

Second, unless you have an extraordinary fire-proof safe, off site if far superior. Fire-proof safes are really only fire resistant -- it is merely a question of how long it takes for the internal contents to get warm enough to destroy the contents.

Tape v disc comparison

Not with any of the recent tape formats. They're all "serpentine" - That is, a very narrow track (up to 1/512 of the width of the tape) goes from the start to the end of the tape. The head then moves down a fraction, and writes the next track "backwards".

This means that the seek time is reduced by up to 512x. Of course, this isn't free - Tape wear is increased since there are many, many more passes over the tape.

Re:Encryption

Here we go again.

Tape has fewer points of failure than a hard drive?

Oh, please...

Explain to me why the entire industry is moving to disk-to-disk recovery backup with tape relegated to archival backup.

Explain to me why most data is kept on hard drives for day to day use if they are so failure prone.

Optimum backup requires disk-to-disk for quick and absolutely reliable recovery. For security, disk-to-disk over the network to an offsite location allows for fully automated reliable offsite backup, but it is expensive in bandwidth even if you only transmit deltas. For offsite storage where netword bandwidth is not available or too expensive, for long-term archival storage, tape is useful - provided the tapes are maintained properly, stored properly and not overused.

Modern tape systems can be very fast and very large, and can cost less than equivalent capacity disk drives, but the fact of the matter is that industry studies show more problems with tape backups than with disk backups. Between equipment failure and operator error, tape backup is problematical for recovery purposes.

Re:Encryption

Tape has fewer points of failure than a hard drive?

It does, because, unlike hard-drives, the media and the reading mechanism separate components. If your read head drives on your hard-drive, it is difficult and expensive (but not impossible) to retrieve your data.

Explain to me why the entire industry is moving to disk-to-disk recovery backup with tape relegated to archival backup.

Convenience

Explain to me why most data is kept on hard drives for day to day use if they are so failure prone.

Convenience

Hard drives are more prone to failures than tape drives, but that can be alleviated through stuff like RAID. Hard drives are more convenient than tape for all but the most fundamental backup needs (full backup, full restore).I prefer to use hard drives too; but they are more prone to failure than tape. If I had to choose to trust all my data to a single tape or a single hard drive, I'd go tape every time. If I had the capacity to create a redundant array of hard drives, I'd go with hard drives. If I needed offsite storage on a budget, I'd go with tape - it's easier to transport and store than a hard drive array. If I had the money for it, or my needs were simple enough that the solution wasn't that expensive, I'd go for a local hard drive array backup, and a remote network backup.

That last one is, in fact, the backup system I use at home. I have a cheap RAID array, and a script that encrypts my most important files and FTPs them to a friend's computer once a week. My important files are mostly source code and documents I've written myself - it doesn't chew through much bandwidth or storage space.

Re:Encryption

Upload truecrypt files

Open source, cross-platform, creates a strongly encrypted file that the program can mount as a real HD, you can mount it on any platform, does transparent encryption (for example in WinXP, it mounts itself with a drive letter, you can throw stuff in directly just as if it were a real drive, and it encrypts as it goes in)

http://www.truecrypt.org/ [truecrypt.org]

You can do it in say N meg chunks or something, I guess you'd have to create a new truecrypt partition every time, but I don't really know much about it, just tried it out and it seems neat

Re:Encryption

Except that the upload speeds for most home users is pretty bad. You'd spending a lot of time pushing files depending on exactly what you chose to backup.

-matthew

Re:Google redux

Google has NOT been ordered to "hand over all the search data". Google has not been ordered to do anything in fact. The judge has hinted that he intends to give the government a random sampling of the websites Google has indexed, but not individual search terms.

As far as the government wanting a pile of businesses data bases to search for porn or what not, that is simply illegal and would result in a prompt judicial smack down. If they want to know what is on someone's serve, they need to do it the old fashion way, with a search warrant... well, in theory at least. The executive branch these days seems to consider search warrants as being optional.

As tempting as it is to go off topic on the case of the executive branch ignoring warrant laws, I'll just drop a link and declare both parties worthless and pathetic.
http://www.washingtonpost.com/wp-dyn/content/artic le/2006/03/14/AR2006031401519.html [washingtonpost.com]

Re:Google redux

So what?

I don't understand why people don't automatically distrust the security of any data not under their direct control. Data held by anyone else could be (mis)used by someone you don't want using it - be it a government agency, an employee causing trouble, a naughty contractor, a script kiddie who got access to something he shouldn't have access to, or any one of a million other people or groups.

If you have sensitive data, you should be taking steps to ensure the protection and integrity of that data, no matter what you're doing with it. Encryption is the most obvious solution, although it's not the answer for every situation.

API proliferation

It's good to see more sites adding APIs to their web services (Amazon already has other web services [amazon.com], as does Yahoo and Google of course). It's becoming even easier for "mere mortals" to link together new technologies to make innovative new systems, but I wonder if this reliance on third-party systems comes at a cost, perhaps to reliability or security?

How they are implementing it ...

For each Gig of storage you use, Amazon gets a free Gmail account and uses that for storing your data.

With virtually no cost for this storage, they can make a killing charging $0.15/Gig/month.

In other news...

The Department of Homeland Security announced it has started a data hosting service. They encourage backing up of family pictures, journals, and irc chats, among other things. There is no monthly cost, but encrypters need not apply. When questioned on how this is a good use of taxpayers money, they simply replied that they wanted material to test their new indexing algorithms on.

Apply now at database.dhs.gov/personal/suspects/index.php

Re:In other news...

You just *know* somewhere, behind some desk, in a dusty office, a DHS agent is reading this going, "Damn, why didn't I think of that?!"

I could see it now. All they'd need is a front company. Someone the public trusts. They could release this service...and..uh...

Wait a second.

Anyone remember GoogleZon?

This is an interesting case of diversification. Amazon, no longer content to be the middle man e-tailer, is shifting it's weight into Google's territory with a service-based profit model. If this trend continues at Amazon, I have to wonder if Google will make a hostile bid for its newfound competitor.
 
  Here is a Link to EPIC [robinsloan.com], a speculative piece on the future of media, including the GoogleZon segment.

Re:Sign me up

Anyways, back on topic, at $0.15 a gig, it would take a long time before buying a hdd would be more affordable for me. (my hdd is 250g, I use about 100g, 100g = 15$, so after 10 months thats 150$... Still cheaper than this HDD that I didn't even get a year ago, on sale, for 200$)

Um, you should look at hard drive prices of today if you're going to be comparing server prices of today. Even retail prices put a 160 GB hard drive [circuitcity.com] at $120. If you are willing to count the rebate price of that drive (it was at the top of the list; I didn't choose it because of the rebate explicitly), it's $50. That's 80 cents and 31 cents per gig respectively. Even if you count just the 100 GB, with rebate that's only 50 cents/gig. In under 4 months that way you'll break even.

Besides, whatcha gonna do? Run your programs remotely? Run your OS over the internet? I don't think so. You'd need a local mirror anyway, so you'd need that new hard drive.

This service has a lot of use, but from a backup standpoint I do NOT think it's at all a good option. Too expensive and too much hassle transferring that much data to make it worth it. (Are you really going to upload 100 gig? Even at a sustained rate of 150 KB/s upload (quite good from my experience over cable) that'd take over a week.)

Re:Sign me up

I don't need an envelope to send a DVD-R upstairs...

Worthless

lets see... for a year of 200gigs, that's $360 USD.

couldn't I jut buy a new hard drive every year or burn hundreds of DVDs for far far less? not to mention they'd be secure from whatever prying eyes or security holes an online backup provides.

Re:Worthless

But what if that hard drive fails.

Well sure, hard drives are cheap why not just keep two of them.

But then what if your house burns down. You'd better make another set of your backup drives so that you can have one array live and one array off-site.

This seems like an amazingly good price for managed storage, if it's as reliable as amazon claim then there's certainly some data that i'll put on it.

Re:Worthless

you could. But not everyone wants to back up 200GB worth of data.

I have my backups categorized as follows:

High priority (documents, records, etc): 150MB
Medium priority (digital pictures, code, etc): 8GB
Low priority (movies, mp3, etc): 430GB

The first gets backup up nightly to a remote machine, as well as weekly dumps to CDs
The second is rsync'ed nightly to my website (not my machine -- shared hosting)
The third gets a RAID5 array, but that's it

For the first (and maybe the second) category, Amazon would be much more economical than doing it myself onto another, designated, disk.

Could file storage services use this?...

Since this is nothing more than an API to access data, I wonder if this couldn't be used as the backend storage for existing file storage services, instead of paying for servers and bandwidth yourself...

This limits costs to storage actually used (at $0.15/GB which is a very fair rate), and bandwidth actually used. The cost that could add up is the bandwidth, where you'd obviously have to direct users to the amazon URL directly to avoid using bandwidth to get the file then to pass it on too.

Plus, at $0.20/GB of bandwidth, upload/download could get expensive still, with no cap on that cost. For example, 2,000 GB of bandwidth, which is bundled with most low-end dedicated servers nowadays (ie. even the sub-$99/mo. machines), this would cost you $400 from Amazon. That's pretty steep, and may be the limiting factor making it unfeasible for this idea. Interesting nonetheless.

Statistics can be your friend

In general you might think that the cost per gig you pay and the cost per gig that Amazon pays is similar. And you would be right. But to obtain high reliability you will pay more.

Amazon, or anybody who tries to build a large distributed storage service, can spread out the probability of disk errors over a larger set of users than you are able to do. The marginal cost to replace a disk that has failed, on a per user basis, is therefore lower for Amazon.

Moreover, the overhead to manage many disks does not increase linearly to the number of disks. Put another way, their per user cost to manage the disks is lower than you.

The cost equation is less about purchasing the storage than maintaining it through the inevitable failures over time. This makes the gigabyte-based usage cost very fair, since it is proportional to the rate of error. The access cost manages their bandwidth expense.

What I would like from a service like this is a pricing guarantee -- if they maintain the same pricing two years from now, it will be a ripoff given the diminishing cost of storage and bandwidth. It would be nice to have it pegged to some kind of disk/bandwidth industry index.

Plus $0.20 per Gb transferred!!!

While $0.15/Gb/month is reasonable, the poster fails to mention Amazon will also charge $0.20/Gb on transfer. So while you will pay $15/month for your 100 Gb pr0n collection, you will also pay $20 to upload it, and a further $20 to download the whole lot to your cube-buddy's computer.

From TFA: "Apart from the storage fee, you pay $0.20 per gigabyte transferred, but there are no minimum fees and no setup costs, so you pay as you go."

Still, not bad - but the economics for the home user are a little less ideal than first reported.

Re:Plus $0.20 per Gb transferred!!!

When Amazon lets me pay for the storage, and have other registered users pay the bandwidth charges (plus my profit) to access my content, then they'll have an interesting business. Unless Google beats them to it.

As it is, online remote storage with ongoing upload, download, and storage fees hardly seems interesting.

Terms Of Service

8) If your Application is determined (for any reason or no reason at all, in our sole discretion) to be unsuitable for Amazon Web Services, we may suspend your access to Amazon Web Services or terminate this Agreement at any time, without notice.

I am not sure I see the point of using a storage service that has the right to unilaterally terminate my agreement and thus, presumably, destroy anything I have stored.

]{

Can't wait to see...

Users with data like yours also had the following data...

Trust and the State

I find it disturbing that I do not trust the State enough to place my data with a third party provider for fear of my privacy potentially being violated.

Of course, my data is unimportant and the State has no interest in me; but *as such* it should be the case that my data isn't even *potentially* accessable to the State - and yet I rather suspect that it is.

As such, I am actually now being suppressed by the State; the State behaves in such a way that I, to preserve my privacy, have to protect myself.

The State is way, way too big for its own good; it's destroying now the freedoms it was created to protect.

Link to the actual site:

http://aws.amazon.com/s3 [amazon.com]

forget it

My minimal requirements would be Webdav, sftp, and rsync-ssh; SOAP and REST I don't care about.

Oh, and also it should come from a company that isn't running a vast data mining operation.

Re:forget it

Apparently all the objects you create in their 'buckets' are encrypted and secure from everyone. So, fine with that.

I would like to see them implement rsync to get data to them, but as its primarily a data-storage service, and not a backup-service (ie its for your web app to hold and access data, not to dump nightly backups on), I doubt we'll see rsync ever, especially as rsync does require CPU time which I bet they have little of in comparison to the vast amounts of storage space.

Google are apparently working on a simialr storage system, so we'll have to see what they come up with. If you want backups.. bqinternet are very popular, and support rsync, and is roughly the same price as Amazon once you start storing a certain amount.

Backup Buddies?

It seems like this new service would be best for offsite backup of prescious data.

However, it isn't all that cost-effective. A local disk is very cheap comparatively, but (as a friend of mine found out) if someone steals your computer, they steal your backup too.

Are there any services out there which connects people with reasonable connections over long distances to back-up eachothers data? I'd be willing to get a new 80GB drive and make it available via a private FTP server if someone else would do the same for me.

Or are there cheaper offsite solutions than Amazon's?

I think it was Linus that said...

"Only wimps use tape backup: _real_ men just upload their important stuff on ftp, and let the rest of the world mirror it ;)" (1996)

Quickplacer, the fastest robot in the world [suvalleynews.com]

BitTorrent

Built to be flexible so that protocol or functional layers can easily be added. Default download protocol is HTTP. A BitTorrent (TM) protocol interface is provided to lower costs for high-scale distribution. Additional interfaces will be added in the future.

Amazon supports BitTorrent for the storage. Does that mean they run the tracker? Interesting way to save on transfer fees that :)

Very clever.... but?

I read the documentation this afternoon. Very clever use of REST, SOAP and BitTorrent. They provide client libraries for many languages.

I can see why Amazon is providing this; to make money off their excess bandwidth and storage space.

I can't really see why a customer would want to use it though. Why not just use a real web host? Amazon S3 has is no minimum monthly fee, has redundancy built into it, guaranteed availability.

Compared with Dreamhost (say) which has a bundle for almost 10 USD/mo. That deal has 20GB + 1TB transfers. For the same amout on Amazon S3 you only 5 GB + 64GB transfers, and doesn't have FTP nor SSH access, nor your own domain, etc etc.

Maybe we should think of it as an inexpensive web cache, like akamai.

I suspect that even Amazon doesn't know what this will be useful for. The developed it for their own use, then polished up for resell. Now they wait for the applications to appear.

I wonder how Google Drive will compare to this

It looks like Amazon managed to get their storage product out before the rumored Google Drive (TechCrunch article [techcrunch.com], Slashdot article [slashdot.org]). I wonder how Amazon's product will compare to Google's, whenever Google's is released. I'm particularly interested in seeing how Amazon and Google will end up competing with each other in terms of price and transfer speeds.

Let's compare

Let's take a 300GB System that needs to be backed up:

900GB at $0.20/gb = $180.00 transfer fee
Monthly at $0.15/gb = $135.00/month recurring charge
Weekly incremental of 30GB = $6.00/wk = $24/mo recurring charge

So $180.00 + $159/month = $2088 just for the first year, plus whatever you have to pay your ISP for abusive bandwidth charges.

Let's look at it from another perspective:

4 WD3200SD 320GB Raid-Edition SATA Drives: About $600
1 4-Port SATA Raid Card: About $250
Expected Lifetime: 5 years

So, buying a whole other raid-5 array to mirror your 900GB of stuff costs nearly $10K to store for 5 years on Amazon versus $850 to store locally. Hell, even if you were paranoid and replaced one of the hard disks every 3 months, you'd still be at less than half the cost.

I won't even get into which is more secure. If it's not on your site or some place you have physical control over, it is not secure.

Re:Let's compare

Mom & Dad

5GB at $0.20gb = $1.00 transfer fee
Monthly at $0.15gb = $.75/month recurring charge

$0.75 for a year is $9.00 dollars.

Throw some pictures up there, taxes, and other essentials using a third party program that "helps" you gather what really should be stored in case of emergency (can you say this program might be a good idea for someone in the open-source community?)

Far better than what they have now and its safe from fire. Throw a little encryption through that 3rd party app accessing the Amazon storage and it would be secure.

The difference here is that I used numbers I expect of data that should be backed rather than just dumping stuff on the drives because its there. The amount of stuff people just dump on drives for backup is amazing and wasteful.

fees and limits

so, $0.15 per GB/month storage, $0.20 per GB transferred.

questions: how do i put a cap on my storage (and more importantly transfer) so a runaway service doesn't screw me?

NOT a backup solution

From the terms of service...

2) You may make calls at any time that the Amazon Web Services are available, provided that you either: (i) do not exceed 1 call per second per IP address, or send files greater than 40K; or (ii) do not exceed the limits set forth in the Service Terms for a particular Service. If you build and release an Application, the stated limitations apply to each installed copy of the Application.

Re:NOT a backup solution

Oops. Left out the important part of the quote. It should have said:

2) The limitation of 1 call/per second/per IP address set forth in Section 1.A.2 above is not applicable to your use of Amazon S3. You may not, however, store "objects" (as described in the user documentation) that contain more than 5 Gigabytes of data, or own more than 100 "buckets" (as described in the user documentation) at any one time.