Amazon's New Storage Service 237
dlaur writes to tell us that today Amazon announced their Simple Storage Service (S3) allowing users to store unlimited amounts of data at $0.15 per GB paid monthly. From the article: "S3 was purportedly built to support both Amazon's own internal applications and the external users of the Amazon Web Services platform. That should be proper motivation to build a service that's fast and robust enough for mission critical use, yet flexible enough to support any storage task thrown at it."
Encryption (Score:5, Interesting)
Re:Encryption (Score:5, Insightful)
Re:Encryption (Score:4, Insightful)
You should really use some sort of redundant arrangement to make sure that a failure of your backup device wont result in data loss.
You then need to either offsite the drives or keep them in a firesafe, in which case you probably need two sets of them so that you can keep one live and the other somewhere safe.
And of course the amazon solution leaves your data accessiblwe from anywhere.
Re:Encryption (Score:5, Insightful)
Second, unless you have an extraordinary fire-proof safe, off site if far superior. Fire-proof safes are really only fire resistant -- it is merely a question of how long it takes for the internal contents to get warm enough to destroy the contents.
Re:Encryption (Score:2)
I'm not really sure about the fire-safe issue. I know my old univerirsty (Edinburgh) had a catastrophic fire a few yaers ago and they pulled the backup tapes from the firesafe the following day and they were back up and running in no time.
The other problem with tape is that you need discipline. You need to actually go in each day/week
Re:Encryption (Score:2)
-nB
Hard drive-based backup systems (Score:2)
No. The pb with tapes is that their data throughput is slow, they are not random access devices, not as reliable as what you would think, more expensive than hard drives, and currently losing popularity. I interviewed about 2 year ago at a petroleum company which has Petabytes of data to backup weekly and they were precisely migrating to hard drive-based backup systems for these very reasons. The higher rate of failure of hard drives can easily be compensated by making 2 or 3 copies of your backups. The f
Re: (Score:2)
Tape v disc comparison (Score:5, Informative)
This means that the seek time is reduced by up to 512x. Of course, this isn't free - Tape wear is increased since there are many, many more passes over the tape.
Re:Encryption (Score:5, Insightful)
Tape has fewer points of failure than a hard drive?
Oh, please...
Explain to me why the entire industry is moving to disk-to-disk recovery backup with tape relegated to archival backup.
Explain to me why most data is kept on hard drives for day to day use if they are so failure prone.
Optimum backup requires disk-to-disk for quick and absolutely reliable recovery. For security, disk-to-disk over the network to an offsite location allows for fully automated reliable offsite backup, but it is expensive in bandwidth even if you only transmit deltas. For offsite storage where netword bandwidth is not available or too expensive, for long-term archival storage, tape is useful - provided the tapes are maintained properly, stored properly and not overused.
Modern tape systems can be very fast and very large, and can cost less than equivalent capacity disk drives, but the fact of the matter is that industry studies show more problems with tape backups than with disk backups. Between equipment failure and operator error, tape backup is problematical for recovery purposes.
Re:Encryption (Score:5, Insightful)
It does, because, unlike hard-drives, the media and the reading mechanism separate components. If your read head drives on your hard-drive, it is difficult and expensive (but not impossible) to retrieve your data.
Explain to me why the entire industry is moving to disk-to-disk recovery backup with tape relegated to archival backup.
Convenience
Explain to me why most data is kept on hard drives for day to day use if they are so failure prone.
Convenience
Hard drives are more prone to failures than tape drives, but that can be alleviated through stuff like RAID. Hard drives are more convenient than tape for all but the most fundamental backup needs (full backup, full restore).I prefer to use hard drives too; but they are more prone to failure than tape. If I had to choose to trust all my data to a single tape or a single hard drive, I'd go tape every time. If I had the capacity to create a redundant array of hard drives, I'd go with hard drives. If I needed offsite storage on a budget, I'd go with tape - it's easier to transport and store than a hard drive array. If I had the money for it, or my needs were simple enough that the solution wasn't that expensive, I'd go for a local hard drive array backup, and a remote network backup.
That last one is, in fact, the backup system I use at home. I have a cheap RAID array, and a script that encrypts my most important files and FTPs them to a friend's computer once a week. My important files are mostly source code and documents I've written myself - it doesn't chew through much bandwidth or storage space.
Re:Encryption (Score:2)
At a previous job, our typical SAN solution was a fibre RAID (or two if super-redundant) that was backed up to a larger SATA array. A hard drive backup was about half the price of tape, significantly faster, and much more convenient. With RAID 5 and a hot spare (or two) the only downside of backing up to SATA is the lack of portability...but if y
Re:Encryption (Score:2)
There is no such thing as "the general purpose backup problem". There are two problems - backup as a defence against a luser error or minor failure and backup as a defence against a system failure or a catastrophic failure. Or in backup architecture terms you have to deal with operational recovery and disaster recovery.
Tapes suck rotten eggz for operational recovery. With all advances they are much slower then disk based solutions.
Disks suck for disaster recovery. Once you
Re:Encryption (Score:2)
An external IDE case with USB 2.0 and Firewire ports is about $50 at CompuUSA.
Now explain to me how adding 'removeability' to the equation makes disk much more expensive than tape?
Re:Encryption (Score:2)
Re:Encryption (Score:5, Informative)
Open source, cross-platform, creates a strongly encrypted file that the program can mount as a real HD, you can mount it on any platform, does transparent encryption (for example in WinXP, it mounts itself with a drive letter, you can throw stuff in directly just as if it were a real drive, and it encrypts as it goes in)
http://www.truecrypt.org/ [truecrypt.org]
You can do it in say N meg chunks or something, I guess you'd have to create a new truecrypt partition every time, but I don't really know much about it, just tried it out and it seems neat
"Win/Linux" != "Cross Platform" (Score:3, Informative)
Call me back when it runs on OS X.
Also encrypted sparse image (Score:3, Informative)
Actually on a Mac what you can do is make a free-floating encrypted Sparse image. It's the same way that the OS handles FileVault encrypted home folders. It's superior to just making an encrypted DMG, because it's readable and writable like a regular filesystem, plus it can expand and contract depending on what's stored in it.
It doesn't have the steganographic or deniability benefits of Truecrypt, but it's good encrypted storage. (Plus if you're ultra-paranoid you can put it inside your FileVau
No Mac version (Score:2)
Re:Encryption (Score:2)
JOhn
Re:Encryption (Score:2)
Re:Encryption (Score:5, Insightful)
-matthew
Re: (Score:2)
Re:Encryption (Score:3, Insightful)
A good Open Source encrypted back up solution that makes use of this in an intelligent way would be great. I really want to automatically store my photo's and important document outside my house. Houses do burn occaionally, computers get stolen by burglars. No need to loose all your important d
Not as affordable as Google. (Score:2)
For businesses, a tape backup is better and cheaper than S3.
For home users, a DVD+/-R is better and cheaper than S3.
Google's won't be better than the tape/DVD options, but at least it will be cheaper. (free)
Of course, this is all assuming Google are going to offer their storage service for free, but I think that's a safe bet.
Re:Not as affordable as Google. (Score:2)
For home users, a DVD+/-R is better and cheaper than S3. >
Except that tape/DVD backup doesn't, in and of itself, guarantee geographic redundancy. You need to then take those tapes or DVDs and store them in a secure location offsite. That may or may not be expensive, depending on how paranoid your are, or how sensitive your files are. But even if you just take them home and put them under the mattress, it's impossible to automate.
Whereas w
Re:Not as affordable as Google. (Score:2)
Also, TFA goes on to say: "Apart from the storage fee, you pay $0.20 per gigabyte transferred". That would add up quickly to pay for redundant physical storage of your own. It also mentions the various ways
API proliferation (Score:4, Interesting)
How they are implementing it ... (Score:5, Funny)
With virtually no cost for this storage, they can make a killing charging $0.15/Gig/month.
In other news... (Score:5, Funny)
Apply now at database.dhs.gov/personal/suspects/index.php
Re:In other news... (Score:4, Funny)
I could see it now. All they'd need is a front company. Someone the public trusts. They could release this service...and..uh...
Wait a second.
Anyone remember GoogleZon? (Score:5, Interesting)
Here is a Link to EPIC [robinsloan.com], a speculative piece on the future of media, including the GoogleZon segment.
Sign me up (Score:2, Insightful)
Anyways, back on topic, at $0.15 a gig, it would take a long time before buying a hdd would be more affordable for me. (my hdd is 250g, I use about 100g, 100g = 15$, so after 10 months thats 150$... Still cheaper than this HDD that I didn't even get a year ago, on sale, for
Re:Sign me up (Score:5, Insightful)
Um, you should look at hard drive prices of today if you're going to be comparing server prices of today. Even retail prices put a 160 GB hard drive [circuitcity.com] at $120. If you are willing to count the rebate price of that drive (it was at the top of the list; I didn't choose it because of the rebate explicitly), it's $50. That's 80 cents and 31 cents per gig respectively. Even if you count just the 100 GB, with rebate that's only 50 cents/gig. In under 4 months that way you'll break even.
Besides, whatcha gonna do? Run your programs remotely? Run your OS over the internet? I don't think so. You'd need a local mirror anyway, so you'd need that new hard drive.
This service has a lot of use, but from a backup standpoint I do NOT think it's at all a good option. Too expensive and too much hassle transferring that much data to make it worth it. (Are you really going to upload 100 gig? Even at a sustained rate of 150 KB/s upload (quite good from my experience over cable) that'd take over a week.)
Bandwidth (Score:2)
Largely i'd want to back up my documents and my digital photos, and while i probably have about 15 gigs right now, i dont produce more than a couple of hundred megabytes a week. For $3 or $4/month i'd take that over pissing around with my own backups (or probably in addition to).
Re:Sign me up (Score:2, Insightful)
Re:Sign me up (Score:2)
3 words: Off-site backup.
The hassle isn't too great when your place burns down, or lightning strikes your box (and don't think a "surge suppressor" is going to save you), or the cops come and grab all your computer stuff. Some stuff you don't want to lose.
Re:Sign me up (Score:3)
Re:Sign me up (Score:5, Funny)
Re:Sign me up (Score:2)
Worthless (Score:5, Insightful)
couldn't I jut buy a new hard drive every year or burn hundreds of DVDs for far far less? not to mention they'd be secure from whatever prying eyes or security holes an online backup provides.
Re:Worthless (Score:4, Insightful)
Well sure, hard drives are cheap why not just keep two of them.
But then what if your house burns down. You'd better make another set of your backup drives so that you can have one array live and one array off-site.
This seems like an amazingly good price for managed storage, if it's as reliable as amazon claim then there's certainly some data that i'll put on it.
Re:Worthless (Score:2)
Re:Worthless (Score:2)
Re:Worthless (Score:4, Informative)
I have my backups categorized as follows:
High priority (documents, records, etc): 150MB
Medium priority (digital pictures, code, etc): 8GB
Low priority (movies, mp3, etc): 430GB
The first gets backup up nightly to a remote machine, as well as weekly dumps to CDs
The second is rsync'ed nightly to my website (not my machine -- shared hosting)
The third gets a RAID5 array, but that's it
For the first (and maybe the second) category, Amazon would be much more economical than doing it myself onto another, designated, disk.
Re:Worthless (Score:2)
(until google comes out and is free.)
Could file storage services use this?... (Score:4, Interesting)
This limits costs to storage actually used (at $0.15/GB which is a very fair rate), and bandwidth actually used. The cost that could add up is the bandwidth, where you'd obviously have to direct users to the amazon URL directly to avoid using bandwidth to get the file then to pass it on too.
Plus, at $0.20/GB of bandwidth, upload/download could get expensive still, with no cap on that cost. For example, 2,000 GB of bandwidth, which is bundled with most low-end dedicated servers nowadays (ie. even the sub-$99/mo. machines), this would cost you $400 from Amazon. That's pretty steep, and may be the limiting factor making it unfeasible for this idea. Interesting nonetheless.
Statistics can be your friend (Score:4, Interesting)
Amazon, or anybody who tries to build a large distributed storage service, can spread out the probability of disk errors over a larger set of users than you are able to do. The marginal cost to replace a disk that has failed, on a per user basis, is therefore lower for Amazon.
Moreover, the overhead to manage many disks does not increase linearly to the number of disks. Put another way, their per user cost to manage the disks is lower than you.
The cost equation is less about purchasing the storage than maintaining it through the inevitable failures over time. This makes the gigabyte-based usage cost very fair, since it is proportional to the rate of error. The access cost manages their bandwidth expense.
What I would like from a service like this is a pricing guarantee -- if they maintain the same pricing two years from now, it will be a ripoff given the diminishing cost of storage and bandwidth. It would be nice to have it pegged to some kind of disk/bandwidth industry index.
What's new about this idea? (Score:2, Insightful)
Not to mention the time it would take to upload a few gigabytes -- cable modems are good at downloading, but they are NOT good at uploading.
Re:What's new about this idea? (Score:2)
Plus $0.20 per Gb transferred!!! (Score:5, Informative)
From TFA: "Apart from the storage fee, you pay $0.20 per gigabyte transferred, but there are no minimum fees and no setup costs, so you pay as you go."
Still, not bad - but the economics for the home user are a little less ideal than first reported.
Re:Plus $0.20 per Gb transferred!!! (Score:2)
And you'd never have to download until you really need it, and at that time, cost is hardly ever a factor.
Re:Plus $0.20 per Gb transferred!!! (Score:5, Interesting)
As it is, online remote storage with ongoing upload, download, and storage fees hardly seems interesting.
Re:Plus $0.20 per Gb transferred!!! (Score:2)
I prefer that than some kind of restriction on the number of people I can share the data with. Anyway I agree on the importance of that figure.
Terms Of Service (Score:5, Insightful)
I am not sure I see the point of using a storage service that has the right to unilaterally terminate my agreement and thus, presumably, destroy anything I have stored.
]{
Re:Terms Of Service (Score:2, Interesting)
Can't wait to see... (Score:5, Funny)
Trust and the State (Score:5, Insightful)
Of course, my data is unimportant and the State has no interest in me; but *as such* it should be the case that my data isn't even *potentially* accessable to the State - and yet I rather suspect that it is.
As such, I am actually now being suppressed by the State; the State behaves in such a way that I, to preserve my privacy, have to protect myself.
The State is way, way too big for its own good; it's destroying now the freedoms it was created to protect.
Re: (Score:2)
Re:Trust and the State (Score:2)
This is a result of the current political climate.
File hosting services have been around forever, e.g. Yahoo Briefcase. It's when stories are leaked about domestic spying, governments pressing service providers for private information, law-abiding people getting bullied for being Muslim or critical of the current administration, that's when things you
Link to the actual site: (Score:5, Informative)
forget it (Score:5, Insightful)
Oh, and also it should come from a company that isn't running a vast data mining operation.
Re:forget it (Score:4, Informative)
I would like to see them implement rsync to get data to them, but as its primarily a data-storage service, and not a backup-service (ie its for your web app to hold and access data, not to dump nightly backups on), I doubt we'll see rsync ever, especially as rsync does require CPU time which I bet they have little of in comparison to the vast amounts of storage space.
Google are apparently working on a simialr storage system, so we'll have to see what they come up with. If you want backups.. bqinternet are very popular, and support rsync, and is roughly the same price as Amazon once you start storing a certain amount.
Re:forget it (Score:2)
rsync is a general protocol for transfering data between different machines; it's not just useful for backup.
Furthermore, many UNIX applications store data in directories and use file system primitives for moving them around; rsync is a good choice for that.
No thanks... (Score:2)
Re:No thanks... (Score:2)
Re: (Score:2)
Re: Keeping the gov't's nose out of your business (Score:2)
I think I have it! (Score:2, Funny)
2. Proceed to write internet worm that does a distributed upload of random garbage.
3. Amazon is unable to satisfy your requests since no one has ever produced an exabyte.
4. Demand Amazon pay you back damages with a multiple of your original investment.
5. Profit!
I rub my hands together in evil dictatorial glee. Mwahahahaha.
Backup Buddies? (Score:5, Interesting)
It seems like this new service would be best for offsite backup of prescious data.
However, it isn't all that cost-effective. A local disk is very cheap comparatively, but (as a friend of mine found out) if someone steals your computer, they steal your backup too.
Are there any services out there which connects people with reasonable connections over long distances to back-up eachothers data? I'd be willing to get a new 80GB drive and make it available via a private FTP server if someone else would do the same for me.
Or are there cheaper offsite solutions than Amazon's?
Re: (Score:3, Interesting)
DIBS: pretty slick. (Score:2)
It's a good concept -- everyone gives some space on their local drive, and in exchange gets to break up their files and store them across others' systems, in encrypted form.
It seems like another one of those projects where the most difficult part is going to be boot-strapping the userbase and community necessary to create the pool of always-available hardware resources so the thing will function. I haven't looked into it too much, maybe it's already there.
Re:DIBS: pretty slick. (Score:2)
Re:Backup Buddies? (Score:3, Insightful)
If you're silly enough to keep your solitary backup in your frickin' computer.
Go tell your friend that God has invented external drives. Then tell him that you'll keep his if he'll keep yours.
KFG
Allmydata (Score:3, Interesting)
I just got the most recent version (1.3) and haven't played with it much but the last one I had a lot of trouble with, sometime
Storage solution, pah! (Score:2, Insightful)
I think it was Linus that said... (Score:5, Funny)
Quickplacer, the fastest robot in the world [suvalleynews.com]
BitTorrent (Score:5, Interesting)
Amazon supports BitTorrent for the storage. Does that mean they run the tracker? Interesting way to save on transfer fees that
Very clever.... but? (Score:3, Insightful)
I can see why Amazon is providing this; to make money off their excess bandwidth and storage space.
I can't really see why a customer would want to use it though. Why not just use a real web host? Amazon S3 has is no minimum monthly fee, has redundancy built into it, guaranteed availability.
Compared with Dreamhost (say) which has a bundle for almost 10 USD/mo. That deal has 20GB + 1TB transfers. For the same amout on Amazon S3 you only 5 GB + 64GB transfers, and doesn't have FTP nor SSH access, nor your own domain, etc etc.
Maybe we should think of it as an inexpensive web cache, like akamai.
I suspect that even Amazon doesn't know what this will be useful for. The developed it for their own use, then polished up for resell. Now they wait for the applications to appear.
Re:Very clever.... but? (Score:2)
However, I have absolutely no idea what their fault tolerance story is there. They probably run them, but at best I'm sure its just RAID. I can't imagine they actually provide real backups for their hosted sites (although I may be wrong... I ran a hosting
I wonder how Google Drive will compare to this (Score:4, Interesting)
Giant Google vs Amazing Amazon (Score:2, Insightful)
Good for us though, Google and Amazon seem to take different approaches to most things, and ultimately that will provide variety, and good innovative competition (unlike MS).
As an aside, the fact that
This reminds me.. (Score:2, Funny)
What they don't say... (Score:2, Redundant)
Re:What they don't say... (Score:2)
I think this is a great idea, but I think I'll just wait for the Google offering before I start on writing my incremental encrypted backup Perl script.
X.
Backup (Score:2, Interesting)
Let's compare (Score:4, Interesting)
900GB at $0.20/gb = $180.00 transfer fee
Monthly at $0.15/gb = $135.00/month recurring charge
Weekly incremental of 30GB = $6.00/wk = $24/mo recurring charge
So $180.00 + $159/month = $2088 just for the first year, plus whatever you have to pay your ISP for abusive bandwidth charges.
Let's look at it from another perspective:
4 WD3200SD 320GB Raid-Edition SATA Drives: About $600
1 4-Port SATA Raid Card: About $250
Expected Lifetime: 5 years
So, buying a whole other raid-5 array to mirror your 900GB of stuff costs nearly $10K to store for 5 years on Amazon versus $850 to store locally. Hell, even if you were paranoid and replaced one of the hard disks every 3 months, you'd still be at less than half the cost.
I won't even get into which is more secure. If it's not on your site or some place you have physical control over, it is not secure.
Re:Let's compare (Score:5, Insightful)
5GB at $0.20gb = $1.00 transfer fee
Monthly at $0.15gb = $.75/month recurring charge
$0.75 for a year is $9.00 dollars.
Throw some pictures up there, taxes, and other essentials using a third party program that "helps" you gather what really should be stored in case of emergency (can you say this program might be a good idea for someone in the open-source community?)
Far better than what they have now and its safe from fire. Throw a little encryption through that 3rd party app accessing the Amazon storage and it would be secure.
The difference here is that I used numbers I expect of data that should be backed rather than just dumping stuff on the drives because its there. The amount of stuff people just dump on drives for backup is amazing and wasteful.
INCLUDING WITHOUT LIMITATION LOST DATA (Score:2)
who do you trust? yerself, or amazon?
fees and limits (Score:3, Interesting)
questions: how do i put a cap on my storage (and more importantly transfer) so a runaway service doesn't screw me?
Privacy (Score:2)
World's largest BitTorrent seed? (Score:2, Informative)
Consider.
NOT a backup solution (Score:3, Interesting)
2) You may make calls at any time that the Amazon Web Services are available, provided that you either: (i) do not exceed 1 call per second per IP address, or send files greater than 40K; or (ii) do not exceed the limits set forth in the Service Terms for a particular Service. If you build and release an Application, the stated limitations apply to each installed copy of the Application.
Re:NOT a backup solution (Score:4, Informative)
2) The limitation of 1 call/per second/per IP address set forth in Section 1.A.2 above is not applicable to your use of Amazon S3. You may not, however, store "objects" (as described in the user documentation) that contain more than 5 Gigabytes of data, or own more than 100 "buckets" (as described in the user documentation) at any one time.
Only a Developer-level API for now... (Score:2)
So for now, don't dump your
Undoubtedly we will see independent developers offering home and SOHO backup tools that use A3... and undoubtedly they'll mark up the price.
Why this is great for backups... (Score:2)
I definitely want an offsite backup for my family photos, this could be a good solution. Sure IN THEORY I could backup to USB Key, HDD, CD-R etc, and in fact I do have a one touch drive. But what happens if my house burns down (fairly unlikely), or worse get's broken into and my shiney looking HDD and PC gets stolen (possible)...bang photos gone.
So what about moving CD's or
Re:Google redux (Score:5, Insightful)
As far as the government wanting a pile of businesses data bases to search for porn or what not, that is simply illegal and would result in a prompt judicial smack down. If they want to know what is on someone's serve, they need to do it the old fashion way, with a search warrant... well, in theory at least. The executive branch these days seems to consider search warrants as being optional.
As tempting as it is to go off topic on the case of the executive branch ignoring warrant laws, I'll just drop a link and declare both parties worthless and pathetic.
http://www.washingtonpost.com/wp-dyn/content/arti
Re:Google redux (Score:3, Insightful)
Re:Google redux (Score:4, Insightful)
I don't understand why people don't automatically distrust the security of any data not under their direct control. Data held by anyone else could be (mis)used by someone you don't want using it - be it a government agency, an employee causing trouble, a naughty contractor, a script kiddie who got access to something he shouldn't have access to, or any one of a million other people or groups.
If you have sensitive data, you should be taking steps to ensure the protection and integrity of that data, no matter what you're doing with it. Encryption is the most obvious solution, although it's not the answer for every situation.
Re:Storage? Oh wow! (Score:2, Funny)
(Probably NSFW)