Ext4 Filesystem Enters Experimental Kernel Tree 237
An anonymous reader writes "Looks like the next version of the venerable Linux 'ext' filesystem is just around the corner. Andrew Morton has added an early version of ext4 to his 2.6.19-rc1-mm1 tree, enabling Linux to support storage volumes up to 1020 petabytes in size, and to write files in 'extents,' or contiguous, reserved areas. According to an article at Linux-Watch, ext4 will be ready for production use within six to nine months, if all goes well. On the downside, the new ext4 filesystem will offer only limited backward compatibility with ext3-aware Linux kernels."
Re:1020 Petabytes? (Score:5, Insightful)
By now you don't even now what to do with 1024PB. Just as we couldn't imagine filling a 250GB harddrive 15 years ago when 500MB were considered huge.
What will happen? We store our digital photos in raw format, not JPEG. We store our songs in raw format, not artificially crippled. We will store high-definition video, possibly even in raw format, not MPEG4 or the likes.
And, woosh, 1024PB will be nothing leaving us wondering how we could ever survive with a measly 250GB drive -- just as we ask ourselves today how life was with nothing but 170kB disk drives.
And how... (Score:5, Insightful)
I'm honestly more interested in someone coming up with cheap, long term archival storage. Hard disks have gone so far past our ability to archive information it's beyond comprehension.
Re:1020 Petabytes? (Score:5, Insightful)
The limits are set by our senses, more concretely, our ears and our eyes.
Our ears are only capable of hearing up to about 20Khz (less than that for most people) and 16-bit samplings are enough that most people cannot hear the difference with anything more. Thus CD-quality is, if not perfect, then good enough that further improvements are ignorable for most people. CD-quality losslessly-compressed music is around 300MB/hour.
In a year, there's 8760 hours, so you'd need on the order of 2.5 TB to store a year worth of around-the-clock never-repeateing losslessly-compressed music. If computers keep getting replaced at the current rate, this means you'll never need more than about 10TB to store sound. This assumes you don't store more than you listen to, if you choose to for example store all music ever produced for convenience, despite never listening to more than a tiny fraction of it, then this requirement goes up by a couple of orders of magnitude. Still, there's good reason to suppose that 10TB will suffice for most peoples sound-storage needs. (even if you wanted to store all the sound you've *ever* heard in your life, including traffic at nigth, that'd still only be 200TB or so)
The real killer is video. We can take in a *lot* more data with our eyes. 10GB/hour is in the ballpark of what you'd need for the sort of quality a modern cinema can deliver. (and there's no particular reason we couldn't go higher.) That works out to 100TB/year, more or less. A lifetime of high-quality video is thus on the order of 10PB.
In short, it is unlikely that an individual (or family) will be able to fill a 1000PB disc with sound and video-recordings. Infact it's unlikely they'll be able to fill it with anything, if that anything is to be consumed only trough their 2 eyes and 2 ears.
That doesn't mean it won't happen. Only that it'll be filled with something more. Once we fire up the holodecks all bets are off. I don't even want to try to estimate the bandwith needed for that kind of immersive experience.
Re:Advantages over XFS, for example. (Score:2, Insightful)
You mean, "exbibyte = 1024 pebibyte". An exabyte is exactly 1000 petabyte.
I used to think this was just fussy, but I am quite tired of guessing which system producers of hard disks/CDROMs/DVD+±×RWs use to figure out if that is enough for my needs.
Re:1020 Petabytes? (Score:3, Insightful)
On second thought, maybe it is Informative, since I was not previously aware you could cram that many puns into so few words.
Re:backups? (Score:2, Insightful)
Most of the databases I work on are 10-50Tb.
Initially we built 3Gb filesystems - we couldn't back then up, the sequential file pointer in HPUX can only address 2Tb, which meant we couldn't copy the whole filesystem to tape. I had to rebuild with max 2Tb filesystem.
Then, Veritas Netbackup can only parallelise backups in different directory trees so I was taking ages to perform a full back up - 18 - 19 hours (bit of a bugger in a 24hr backup window).
We don't do incremental backups because restoring takes an age with incremental. Even with fulls, if the backup takes 10 hours, the restore takes 15 - its to do with how different simultaneous backups are stiped across the tape. Therefore we always do fulls.
Eventually (and fortunately during test phase) we rebuild the filesystems with 500Gb limits.
Which makes growing the database really sticky because we have to allocate new filesystems, rather than grow the ones we've got.
There's a major move towards disk-based archival storage, but tape's reliable and extremely portable so its generally what we end up using.
Re:backups? (Score:3, Insightful)
First of all, disk drives are advancing faster than tapes.
But the problem is worse than that. Different aspects of disk drives are advancing at different rates. Capacity is increasing faster than interface speed is increasing faster than access speed is increasing faster than block reliability.
Consider an old 500MB drive from the mid-90s; it takes maybe a couple hours to read every block on the drive, and odds are that you won't have bad blocks before the disk dies entirely.
The new 1000GB disks we'll have soon, though... you're pretty much guarenteed some of the blocks will go bad before the disk fails; there are just so many of them. And if you can read at 10MB/second, sustained (pretty respectable, I think?), it literally takes 24 hours just to dump all the data from the drive, never mind back it up to something else.
The whole model of 'back up your fixed media to removable media' is not going to work anymore. It does not scale up to modern hardware. I think the answer will be some wacky combo of raid and file-system version control, but YMMV.
Re:Reiser4 (Score:5, Insightful)
As much as I agree with you questioning the many 'funny' ACs, I can't support this statement. One day you, yes, you, might also become a suspect. And then, suddenly, you would see the world with different eyes. Especially, when you're found innocent and you also find that your friends have deleted all memories of you since they don't want to be associated with you; as suspect.
I am sure many of us would feel similar once the investigation is over and Hans eventually found guilty. But as civilised people, we better wait. How old are you ? Have you never ever been suspected of something ?
Re:Reiser4 (Score:2, Insightful)
Re:1020 Petabytes? (Score:3, Insightful)