Archiving Digital Data an Unsolved Problem 405
mattnyc99 writes, "It's a huge challenge: how to store digital files so future generations can access them, from engineering plans to family photos. The documents of our time are being recorded as bits and bytes with no guarantee of readability down the line. And as technologies change, we may find our files frozen in forgotten formats. Popular Mechanics asks: Will an entire era of human history be lost?" From the article: "[US national archivist] Thibodeau hopes to develop a system that preserves any type of document — created on any application and any computing platform, and delivered on any digital media — for as long as the United States remains a republic. Complicating matters further, the archive needs to be searchable. When Thibodeau told the head of a government research lab about his mission, the man replied, 'Your problem is so big, it's probably stupid to try and solve it.'"
I've heard this problem over and over (Score:5, Interesting)
A huge problem, indeed! (Score:2, Interesting)
It's not easy, sure, but neither are many of the other tasks we take on as humans.
Re:Not too long... (Score:5, Interesting)
Making a format that will survive a thousand years so long as our advanced civilization is still around and still cares is pointless, because as long as there is a continuous line of people that care, they will be willing to transfer at least the more important stuff to new media. The trick is coming up with something that will still be readable when archaeologists dig it up 10, 50, or 100 thousand years from now.
The solution (Score:4, Interesting)
The real problem is 60 years of archives of antiquated, proprietary, task-spcific and mainframe computer data cards and tapes whose original programmers are halfway to cedar boxes; if the government can't get their support in time it may as well call all the early stuff a loss and hand it over to archaeologists.
Re:Open, well-used, file formats. (Score:3, Interesting)
I have simplified for the sake of being lazy but the essence of portable emulators + extensive software and data backup and storage is sound, you don't even have to concern yourself with speed if you are willing to accept that future hardware will be fast enough.
Re:How is this different (Score:4, Interesting)
Its different because of the sheer volume of information being created today. Ancient cultures were not creating millions of pages of information every day.
Your Rosetta Stone analogy is inappropriate. We have not discovered any sort of Rosetta Stone for the ancient Maya hieroglyphs but we have had success in deciphering them because we can apply linguistic analysis techniques to figure out what words correspond to what actions/things. Its a little more complicated for abstract concepts, but you can figure out a surprising amount from basic language knowledge.
Re:How is this different (Score:5, Interesting)
The era of restoration comes. However, when people blow the dust off those old DVDs and players, they discover that the DVDs have decayed to the point of unreadability. Massive quantities of archived data and knowledge are irretrievably lost.
The main problem in our age is thermodynamics -- information is stored so densely that it tends to decay naturally, on its own. By contrast, ancient stone carvings (as well as their keys, such as the Rosetta stone), are sufficiently durable to last (basically) for ever.
Re:Not too long... (Score:1, Interesting)
Granted it's not like most people care nowadays. Look at any slashdot discussion on education, rather sad how people complain about having to take history (heck or any subject they're not "interested in deeply") in school. People want to be ignorant sheep.
Hell look at Xena or the dozens of other "historical" tv shows out there, I shudder to think of how many people's knowledge of history is probably based on such crap alone. In 20 thousand years they'll have Princess Diana was running around with a lightsaber killing communists or something.
Stuff I can't read (Score:3, Interesting)
UK/BBC Domesday book (Score:3, Interesting)
Well, 15 years on, it was useless. The then-proprietary format was not readable on anything modern, and there was not much of the old hardware around either. You can google for it ("UK domesday bbc data" should do it), the first link I saw was on the Guardian Online [guardian.co.uk].
I've still got stuff on floppies, but no-one builds PCs with them anymore. I've got two old laptops with floppy drives, the other three computers have none. (OK, I also have two corpses with floppy drives, and the controllers on two of the new PCs will accept floppy drives, but, please take my point - they're going out of fashion.)
In 20 years time, there will probably be no CD/DVD drives, we'll all be using a new more portable, more backupable, lighter, faster, probably online-only storage medium. Kids won't recognize laserdisks, floppies, or USB ports. They might not recognise keyboards either - who knows?
Re:CDs (Score:2, Interesting)
The first burnable cds you could buy (in the 90ties) were of a decent quality, i still have some burned ones around, and they are still readable (older than 10yrs).
But some newer ones (cheaper, & mass-marketing 'mode') are of an awful quality: i have plenty that "died" when reading them: it begins with some bad CRCs, and then more & more & more, till nothing valuable can be read off it. This happened in LESS THAN 2 YEARS.
The problem with cds:
- They hate sunlight
- they hate being in a too hot, or too cold place
- they hate being in a place with too much/not enough humidity
- and the worst: they react with air (oxygen).
It's build with a 2mm plastic, the dye is on top, with some 'protective' layer over it. Some are better than others.
Now with DVDs, they seem to be from a much better quality already, the explanation is simple: the dye isn't on the surface anymore, but between 2 slides of plastic glued together. The reaction with air seems to be insignifiant. Atm, i have no single failing DVDR that i know.
But some brands are of better quality than others.
And btw:
"Real men don't use backups, they post their stuff on a public ftp server and let the rest of the world make copies." - Linus Torvalds
Re:CDs (Score:3, Interesting)
That said, there are some newer dyes that are claimed to be stable for a hundred years. I haven't ever seen these in stores, so they may be seriously expensive, or maybe I just don't know where to shop...
Re:How is this different (Score:5, Interesting)
Re:Not too long... (Score:4, Interesting)
Archaeology is the search for fact. Not truth. If it's truth you're interested in, Doctor Tyree's Philosophy class is right down the hall. So forget any ideas you've got about lost cities, exotic travel, and digging up the world. We do not follow maps to buried treasure, and 'X' never, ever marks the spot. Seventy percent of all archaeology is done in the library. Research. Reading.
-- Indiana Jones and the Last CrusadeRe:Not too long... (Score:4, Interesting)
History is interesting, school makes it suck: "In Year ABC, XYZ happened. Test next week - students who regurgitate well will get an 'A'."
People don't want to be sheep - totalitarian governments need populations to be docile. School is designed to suck the uniqueness out of children so, as adults, they'll take up a spot on a standardized assembly line.
Kinda cruel how the government has encouraged the shipping of assembly line jobs to China... Dumb down the population, then get rid of the reason for the dumbing-down.
See Gatto's Underground History [johntaylorgatto.com], for example.
Funding (Score:3, Interesting)
Relax... Google will take care of it... (Score:3, Interesting)
The Importance of Historical Records (Score:2, Interesting)
In the early 1960s a wise man spoke
/ quote
We choose to go to the moon. We choose to go to the moon in this decade and do the other things, not because they are easy, but because they are hard, because that goal will serve to organize and measure the best of our energies and skills, because that challenge is one that we are willing to accept, one we are unwilling to postpone, and one which we intend to win, and the others, too.
/ unquote
We Went to the Moon, and all the signals received including a high definition picture quality version (by the technology of the time) was recorded at Nasa (and also I believe at the receiving station of Parkes receiving station in Australia where the signals were received through their deep space network radio telescope), these most important "documents" of our time have been lost, lost and never able to be recovered leaving us purely with the broadcast version which was at a much lower quality standard (eg a poor quality photocopy).
Its important for the nature of our history and our essence of our technology and who we are as a people to preserve these important events for our future generations.
When you look at this Planet, we regularly goes on a rampage where the technology is lost and we are thrown back hundreds of years, Take Ancient Egypt, The Technology of the first milenium, The great library of Alexandria, (atlantis etc) so much of the past for which we have lost and are poorer for as a result.
Cant we get it right this time as we face our possible next destructive surge, whether it be by climate, economic, famine, nuclear war, microbiological warfare / disease (whether natural or manmade), chemical accident causing a chain reaction etc..., so many risks, lets do this before its too late, too late to be done and too late to be able to be done.
Darren Stephens
Adelaide, Australia
Re:How is this different (Score:4, Interesting)
Re:How is this different (Score:1, Interesting)
Re:Old stuff. (Score:2, Interesting)
As I perused the contents of said stack of discs, I found that almost 90% of them were redundant or out of date copies of files I had completely forgotten about.
Well then I have question that I would like to throw out to Slashdot readers. Like the person who wrote the parent, I have tons of old files on my hard drives. I always run at least two hard drives, using one for backups. Then when I upgrade computers, I bring over one of the old hard drives to the new computer, copy it to the new drive, then continue to use it to backup new material. By now I have files duplicated and triplicated all over the place. After almost a decade of this, I have many gbs of files which would probably condense down to a fraction if all the duplications were eliminated. What kind of software do I need that will analyze all my files and automatically find and remove duplicates? - or do I need to develop such software for myself? ...and if I do, then is there niche for commercialization of such software?
Re:How is this different (Score:4, Interesting)
They might not know that a byte is 8 bits, but with a little analysis, it shouldn't be hard to figure out. There are numerous statistical properties that can be exploited to figure this out relatively easily. For example, with most types of data, the higher-order bits (in any size byte) are more likely to be 0 than the lower-order bits are. Think about how booleans are stored in most systems. Think about the characters in this message: 100% of them have a zero high-order bit. To put it a little differently, there is more entropy in the lower-order bits.
So, to figure out how many bits there are in a byte, you take your data, and for all reasonable sizes of bytes (say, from 4 bit bytes up to 36 bit bytes), you compute the function that maps bit position (low- or high-order) to an entropy value for that bit. Then you can tell by the shape of that curve which guess about bits per byte was the right guess. Heck, it should be such a strong trend that you can probably automate it!
Remember that future civilizations will probably also use digital data as well, at least ones sophisticated enough to try to read the optical and magnetic media. They may not know the FAT32 filesystem, but they will have invented statistics and information theory, and they will be able to make some awfully good guesses at things. And yeah, it might take them 10 or 20 years to be able to read a FAT32 volume correctly if some poor college student of the distant future has to do it on a shoestring budget of grant money, but if they're reading 10,000 year old data, how much does that matter?
Re:Not too long... (Score:3, Interesting)
Solutions - EMC, par2, tape, optical media (Score:1, Interesting)
I still have some punch cards that I need to convert. It doesn't matter that IMSL solved all the same problems better.
I have some:
- 5.25" floppies
- RLL/MFM hard drives with data
- parallel port QIC80 tapes (250MB ea)
- 1/4" tapes (not so important)
All of these need to be converted to useful media. Or, I figure they are probably corrupt by now.
Hence, the addition of par2 http://www.par2.net/ [par2.net] which provides parity protection against partial media failures. Corruption can be handled by home users. For enterprise customers, EMC, Veritas and STK have handled this for years. For home users, the extra effort that par2 requires, http://en.wikipedia.org/wiki/QuickPar [wikipedia.org], may not be worth it. But if it is your wedding pictures, RAID on disk, off-site storage and optical media are **all** required to save your marriage. That 1GB of GMail space might be worth it?
If it is your corporate date - go ahead, take your chances. How much can it possibly be worth if it is missing? YOUR JOB perhaps?
The cost of time to find all the data and work interuption is nothing compared to gracefully handling a disk failure without a production impact. Where I work, we do backups **very** well and remote vaulting in alternate data centers.
Re:Not too long... (Score:2, Interesting)
The Waste Isolation Pilot Plant site marking (Score:4, Interesting)
While the WIPP site won't have the benefit of constant updating of the media (it's designed to be survive on its own for 10,000 years) it does address some of the same points; longevity of the media, a format that will be usable into the future, and ability of future civilizations to understand the message.
Off-topic perhaps but an interesting read.
Archival medium for more than 10K years (Score:2, Interesting)
Not every piece of digital info can be saved that way, or needs to be saved as others have pointed out. Current college textbooks, some history books, literature and music and an encyclopeadia will go a long way to create a useful memory of our times for the future.
Some years ago, in California, they opened up an 100 year time capsule. I do not remember the suff that was in it, but it was mostly useless junk by our standards today. If we could send an e-mail back in time, we would ask them to include totally different things. It is easy to make the same mistake now as to content.
Re:Not too long... (Score:3, Interesting)
There are two basic strategies to keep our digital files *currently* accessible:
1) emulation. Check out IBM's Universal Virtual Computer project.
2) migration. Not only migration of storage media, but migration to new and currently readable formats.
We will need to migrate all of our digital files every 5-10 years or so to keep them current. And yes, information will get lost along the way - everything decays eventually.