Follow Slashdot stories on Twitter

 


Forgot your password?
Close
typodupeerror
×
United States Education

Digital Future of the Library of Congress 141

Posted by Zonk from the yay-for-learning dept.
lesinator writes "On Monday the 28th the US Library of Congress is holding the eighth lecture in its series on Managing Knowledge and Creativity in a Digital Context. Previous speakers include David Weinberger on blogging, Brewster Kahle - founding member of archive.org and the wayback machine, and Lawrence Lessig on intellectual property and the creative commons. After the lecture questions will be taken from the audience and the internet. C-Span will be broadcasting the lecture live at 6:30 PM EST, and also has archives of previous lectures. Audio archives of previous lecture are available at Audible.com in the Selected Free Media section."
This discussion has been archived. No new comments can be posted.

Digital Future of the Library of Congress More

Digital Future of the Library of Congress

Comments Filter:

  • At last! (Score:3, Funny)

    by Shadow Wrought ( 586631 ) <shadow.wroughtNO@SPAMgmail.com> on Friday March 25, 2005 @12:18PM (#12046615) Homepage Journal
    We'll know just how much storage really is required to hold the Library of Congress.

    • Re:At last! (Score:5, Insightful)

      by cmburns69 ( 169686 ) on Friday March 25, 2005 @12:28PM (#12046708) Homepage Journal
      While it's an interesting question, it really depends on how you want to store the contents of each book.

      Would you store each page of each book as an image? As flat ASCII text (except of pictures and diagrams, of course!)? What kind of indexing would you do? Basic indexing of book names? Full-text indexing of the contents? All that storage adds up!

      In summary, the library of congress (depending on the method used) could probably fit into something ranging from a couple of gigabytes to a couple of petabytes.

      • Re:At last! (Score:5, Interesting)

        by Shadow Wrought ( 586631 ) <shadow.wroughtNO@SPAMgmail.com> on Friday March 25, 2005 @12:34PM (#12046764) Homepage Journal
        Well I owuld think that they would have to start with an image first. Once they OCR'd it and generated ascii text files, they could save a tremendous amoutn of space by simply deleting the images. However, after that much effort in imaging all those pages, I just can't see them doing that. The best bet is probably two databases, one of ascii text and one of images.

        They might even be able to generate revenue by having the ascii text freely available and searchable, while the images would cost money. That way folks just interested in the text can find it easily, while scholars and others who need to see the source material can have access at a moderate price.

        • Yes, and yet...no. (Score:3, Insightful)

          by oneiros27 ( 46144 )
          You're making a large number of assumptions in your first paragraph:
          1. The OCR is always correct.
          2. The documents could be represented in ASCII
          3. The text is the only part of the document with any value

          Of course, your second paragraph shows that clearly those assumptions can't be true -- why would someone pay more for something without an additional benefit?

          And you wouldn't maintain seperate databases -- pictures aren't searchable. You'd want to use any OCRd (preferably vetted afterwards) as the basis for inde

          • I did make quite a few assumptions, but it is, after all, a thought process. The actual image would have greater value for some people than for others. If you want to read Moby Dick, you can do that in just a text format, you don't have to view the actual images (which are significantly larger files). If, on the other had, you are doing your doctorate thesis on Moby Dick than you will likely want to view the actual images. It is also more likely that you, since you require more than just the text, would
        • Mmm. I've seen document archival formats available (patented, I think) taylored for printed documents -- using one of these, it should be possible to get your typical page well below 100K, and stay in that general range even with drawn illustrations (though anything w/ color or photo-style images is no longer suitable). DjVu is a prime example of these, though others exist.

          So keeping the scanned images shouldn't really require such a tremendous amount of space.
      • Uncompressed text, maybe markup, and you're looking at about 20 terabytes I believe. Adding in the works with either illustrations or photographs, in some decent but lossy compressed format, and you're easily quadrupling that (just a guess).

        Indexing, by what, subject, author, and title? 1% overhead at most. Fancier googlesque searching though, could be a big hit.

        And correct me if I'm wrong, but there are quite a few videos too.

        Not to mention some historical stuff that can't even be digitized all that wel

        • Re:At last! (Score:3, Interesting)

          by caseydk ( 203763 )

          I was working on this project just a few years back (2001-2002).

          Our estimates projected that by 2005, it would be take about 4 TB of digitization EACH day to keep pace.

          The first storage phase called for 180TB server.
      • Have you ever seen someone's hundred and fifty page thesis, diagrams and all, fit onto a 3.5" floppy? People who wrote their theses in TeX or LaTeX, with a few postscript diagrams. I was impressed by how tiny the code for a real, well-produced book could be.

        'Course, the problem is that these representations work if you're entering in the content with that method in the first place.

        --grendel drago
      • I can't stress this enough... whatever they do, for the love of God, DON'T USE .LIT!

      • Re:At last! (Score:2, Insightful)

        by aboyko ( 16319 )
        A couple of gigabytes?! Only if you burn it first. There's something like 10^8 books, nevermind the other stuff. How do you compress any given book into 100 bytes?

        The "20 TB" figure comes from the smallest possible measure, treating the flat books as ASCII text. Even just considering current digital content, it's also inaccurately small by >1 order of magnitude.

        It's a really really really big library.

    • Re:At last! (Score:3, Interesting)

      by WillAdams ( 45638 )
      There's a cue for a question I've been wondering about for a while.

      What was the first reference / usage of ``LoC'' as a unit of knowledge measurement?

      The first time I recall seeing it was in Michael Gear's novels, _The Artifact_ if memory serves, ~1976.

      Anyone have an earlier instance?

      William
    • I'm betting it takes about one Library of Congress to store the Library of Congress. Any takers?

  • Here's an idea related to audio archiving (Score:5, Insightful)

    by filmmaker ( 850359 ) * on Friday March 25, 2005 @12:19PM (#12046627) Homepage
    Maybe the fine folks at audio.com might consider making their audio clips available by means other than the Real or MS media players?

  • Dammit! (Score:2, Insightful)

    by dteichman2 ( 841599 )
    What are they thinking! Airing this at 6:30 PM EST! CSpan has just ensured that nobody on the west coast will see this. Or, is that what they are aiming for?

  • Nice, but how long? (Score:2, Funny)

    by Anonymous Coward
    How long is it going to take to digitize the entire library?

    Anyone have a good approximation? I'd like to know in Burning Libraries of Congress (BLC) please.

    I'm guessing somewhere around 10-200 BLC.

    • Re:Nice, but how long? (Score:4, Interesting)

      by yuriismaster ( 776296 ) <{tubaswimmer} {at} {gmail.com}> on Friday March 25, 2005 @12:37PM (#12046808) Homepage
      Well, I would imagine that unless they have a massive staff and many OCR scanners or automation with REALLY good OCR, this may take a LOONNNG time.

      I'm not quite sure about the length of a BLOC, but this is a job for not-quite-manual labor. Each book requires a simple task: Scan page 1, flip page, scan page 2, page 3, flip, ad infinitum.

      One way to save on time would be to contact the publshers of any book made after 1985-ish, where you can get electronic copies from the author. Some older books may have been already digitized, but it's still going to take more than 25 years unless there's a massive army working on this.

      • Re:Nice, but how long? (Score:4, Informative)

        by Blue-Footed Boobie ( 799209 ) on Friday March 25, 2005 @12:55PM (#12046973)
        Nonsense. I put together solutions with high-speed scanners all the time. Some of our highest-end average 118ipm (Duplex) and have 1000pg ADFs.

        Also, you would generally split the load between 4-6 of these scanners for a job this big. The software is automated, and will OCR/Convert/Archive the file is one step.

        As a general rule, you can fit 10,000 b/w text pages in 1GB of storage.

        • That's great for loose sheets but what about scanning bound books? Aren't you then back to scanning a page, flip a page, scan a page, etc.?
          • Nope, Canon (and others) make Book Scanners with actually flip and scan each page automatically. They can handle all sizes too.

            They are very expensive, but cool as hell.

          • I used to work in that industry too. Typically, bound material would be cut into loose sheets... you basically sacrifice a book to get the images electronically. Also, any decent high volume scanner can scan both sides of a sheet at once, so there's no flipping.

            As an unrelated asidem some even scan in color, but your storage requirements go way up if you do anything other than bitonal (even greyscale eats up the bytes pretty quick).
          • That's great for loose sheets but what about scanning bound books? Aren't you then back to scanning a page, flip a page, scan a page, etc.?

            Cut the binding off?
      • I'm not quite sure about the length of a BLOC, but this is a job for not-quite-manual labor. Each book requires a simple task: Scan page 1, flip page, scan page 2, page 3, flip, ad infinitum.

        Uhmm, no. You cut the binding off and run the pages through a document feeder, then rebind the book, using these things that some people refer to as "machines" ;-)

  • Some ideas (Score:5, Insightful)

    by gowen ( 141411 ) <gwowen@gmail.com> on Friday March 25, 2005 @12:24PM (#12046669) Homepage Journal
    Here an interesting talks they might give:

    i) What if the Apostles had had technological means to prevent the reproduction of the New Testament?

    ii) Would our culture be diminished if the people who rediscovered Beowulf had been unable to decrypt the manuscript?

    iii) Is the continual repitition and reworking of myth and fable through the Oral Tradition disrespectful of the content creators who first recorded these stories?

    • Re:Some ideas (Score:4, Insightful)

      by Scrameustache ( 459504 ) on Friday March 25, 2005 @12:29PM (#12046719) Homepage Journal
      i) What if the Apostles had had technological means to prevent the reproduction of the New Testament?

      Main Entry: apostle
      Pronunciation: &-'pä-s&l
      Function: noun
      Etymology: Middle English, from Old French & Old English; Old French apostle & Old English apostol, both from Late Latin apostolus, from Greek apostolos, from apostellein to send away, from apo- + stellein to send
      1 : one sent on a mission: as a : one of an authoritative New Testament group sent out to preach the gospel and made up especially of Christ's 12 original disciples and Paul b : the first prominent Christian missionary to a region or group

      They wouldn't have prevented the distribution of the story their mission it was to distribute, that's for sure.
      • Ooops my bad. What's the collective noun for the dudes that wrote the Gospels?

        • Re:Some ideas (Score:4, Interesting)

          by Anonymous Coward on Friday March 25, 2005 @01:25PM (#12047227)
          It's been continually re-written. For example, until 1954 Jesus never actually said "I am the Son of God"; when Pontius Pilate accused him of claiming to be the Jewish Messiah, he cryptically responded "It is you who said it." The fact Jesus didn't claim to be the Son of God but was surrounded by intense believers was one the essential "mysteries" of Christianity that you were supposed to accept as a Christian.

          In 1954, the American "New International" edition just editted the trial dialog and "re-interpreted" "it is you who said it" into "I am the Son of God." I don't think the European and Catholic churches have editted that part yet.
      • Authorship of the New Testament is not a simple question at all. First off, the Apostles didn't sit down and start collecting the New Testament. That was done hundreds of years later by some chaps in Rome or Turkey who also had political axes to grind. Every few decades or centuries, there's also Yet Another Translation, and in the forward they talk about the prayer, consideration, and attempts to divine the True Word of God that went into it. Common belief is that over the centuries there has been so much
    • You'd better read this [bbspot.com].
    • "iii) Is the continual repitition and reworking of myth and fable through the Oral Tradition disrespectful of the content creators who first recorded these stories?"

      iv) Why do people of oral traditions get no legal protections for their work? (From those outside their tradition who would fix it and lock them out from their own work?) Why must it be fixed?

      I know that is at least halfway to zany, but please try to give a halfway to reasonable answer.

      all the best,

      drew
    • This, folks, is why I read Slashdot. Despite all the dupes, trolls, groupthink and pseudoscience, occasionally I read a gem of a post. That is one of the most scathing, concise attacks on DRM and IP ridiculousness that I have ever read. Parent poster, I salute you!

  • Next series (Score:4, Funny)

    by E IS mC(Square) ( 721736 ) on Friday March 25, 2005 @12:34PM (#12046766) Journal
    "Managing Knowledge and Creativity with DRM"...

    Sponsored by Apple and Microsoft!
    • No, not Apple. Apple has gone out of their way to ensure that you have somewhat generous rights with the music you purchase from their store. This is keeping the RIAA people happy.

      More like: Sponsored by the RIAA and Microsoft!

      If you aren't happy with the DRM on the iTMS songs, I suggest the HYMN project [hymn-project.org].

  • Hello, Project Gutenberg?!? (Score:5, Interesting)

    by Infosquawk ( 131022 ) on Friday March 25, 2005 @12:40PM (#12046843)
    I can never understand why there isn't more acknowledgment of our debt to Project Gutenberg on these issues.

    Michael Hart was digitizing books before digitizing books was cool, as far back as 1971, and the Project's efforts have been hugely successful on very little money. Nevertheless, I rarely see any official or media acknowledgment of the Project's efforts. If anyone should be on that panel for their ability to give advice from practical experience and performance in this field, while on a shoestring budget, it would be Hart!
    • The fact that Project Gutenberg has not consumed huge amounts of money to produce a great amount of value is PRECISELY WHY it does not get more recognition.

      The business of charity does not want competition from groups that create better products for less money, as that would put pressure on them to create a reasonable amount of value themselves, without the benefits of cushy offices and hefty salaries.

      The business of education also does not want competition from organizations that produce greater value at
      • Man, you're appealing to malice a lot more than laziness and stupidity, when the latter is a much, much more likely culprit.

        --grendel drago
        • Let's just say that I have seen so many examples that I can only conclude that:

          (1) People in many "charitable" organizations and "educational" establishments are quite corrupt; or

          (2) People in many "charitable" organizations "educational" establishments are amazingly, astoundingly stupid.

          Neither bodes well, but only corruption seems to explain all the facts, especially in the case of the "education" establishment.

          Baldur of Asgard
          • Maybe:
            (3) People in many charitable organizations are out DOING charity, not talking about it. Kind of like Project Gutenberg.

            I suspect it's the (3)s that make charity work, and make people want to keep it alive, but it's the (1)s that make the most noise and draw the most money.

            IMHO there's an unfortunately large class of people who specialize in smelling the flow of money, and inserting themselves into that flow. The world would be for the most part better off without them.
    • It's just the way the media works. They are lazy, and point their sensor arrays at the noisiest targets. Look at the Terry Schaivo case. I've heard the televised opinion of eighty seven million doctors *EXCEPT* the ones that have actually examined her.

  • Outsource parts of LOC to Google or Amazon? (Score:5, Insightful)

    by G4from128k ( 686170 ) on Friday March 25, 2005 @12:44PM (#12046888)
    With the current wave of outsourcing, privatization, and government use of commercial contractors, I wonder if Amazon or Google don't have a major role to play in the process of cataloging/archiving/serving digital content in the future.

    Although LOC could never be replaced by a Google or Amazon, these private companies could provide services that augment or reduce the cost of LOC-like services. For example, if Amazon scans a book, why should LOC scan it too?

  • What about a backup copy? (Score:4, Interesting)

    by voss ( 52565 ) on Friday March 25, 2005 @01:10PM (#12047101)
    It would seem if the LOC is going to have X number of Petabytes on computers...why not have a second copy stored AWAY from DC. If something were to happen to DC at least we would have backup copies of everything...and we probably should have a separate backup location at a third site.
    • Say! that's brilliant! I'm going to go down the hall and mention that to the LC IT guys!
      .
      .
      .
      Oh. Apparently it had occurred to them. Well, thanks, just the same. You think of anything else, please, drop us a line!
    • Brewster Kahle said on a podcast (IT Conversations) that they are working on an agreement with the library at Alexandria Egypt to back up each other's archives. Sounds like a good deal, since Alexandria doesn't have most of the LOC content and the LOC has little of what Alexandria is archiving.

  • This just in... (Score:5, Funny)

    by SmokeHalo ( 783772 ) on Friday March 25, 2005 @01:54PM (#12047455)
    The LOC has announced that they are accepting volunteers to digitize texts. Their first volunteer is Earl the night janitor, who has been busily keying in the last 20 years of New York City phone books. He hopes to move on to Chicago soon.
  • I wonder how long before they merge with the CIA and become the Central Intelligence Corporation...

    (It's a joke.)

  • Are they requiring publishers to submit PDF files? (Score:5, Interesting)

    by melted ( 227442 ) on Friday March 25, 2005 @02:05PM (#12047537) Homepage
    Are they requiring publishers to submit PDF files for new entries yet? Or files in another open format? Man, I'd hate to see taxpayer's money wasted on doing work that they could avoid doing by simply mandating PDF submissions from publishers.

    I can see that some publishers may just say, "oh, my book isn't gonna be in libraries if I don't submit PDF, so much the better, I'll sell more copies". I hope these fellas realize how badly they're shooting themselves in the foot.
    • Since nearly all typesetting is done electronically these days, I wonder if they shouldn't just have publishers send them the raw typesetting documents in addition to a hardcopy. It wouldn't be much work for the LOC to write (or buy) software to convert all the common typesetting formats into whatever standard format(s) they would like to use internally, and for dispersion to the public.

      It would certainly be smarter than scanning them in themselves, or demanding extra work on the publishers part to to conv
  • Isn't the Library of Congress' digital collection, especially with respect to music, going to totally screw iTunes and any other online DRM stuff, in order to bring us our library materials?
  • ... that as a universal unit of measurement, it's gonna be around for a while.
  • sometime before the "finale" Enterprise gets destroyed and they rebuilt it as Enterprise B heh

Slashdot Top Deals

The one day you'd sell your soul for something, souls are a glut.

Close