Deja, Google, Open Source, Oh My

blkros writes: "Over on Wired there's an article about Deja News and the plans to try to get Google to open source the Usenet archives it got when it bought Deja News. Part of the plan is to have the Library of Congress oversee it and put it on university mainframes. Google has taken the archives off the web for now Aaagh!"

How much?

[BillGodfrey]

So how much would you pay for a years access to the archives?

I would happily hand over $10 a year. I really do miss dejanews in it's heyday.

Come to think of it, I would pay for access to Google web search as well, so long as the fluff is removed.

Bill, a liker of usenet.

Re:Where is the code? Who owns it?

[fiorelli]

I just found this article, which may or may not shed some more light http://www.localbusiness.com/Story/0,1118,AUS_6290 86,00.html

Re:USENET is a public forum already

[Eric Smith]

The library of congress contains just about every copywrited work ever written.

That is NOWHERE near true, alas.

Google makes REAL money from ads

[pq]

whether they make any money of [sic] google.com ... is a different story.

Actually, google.com makes real money off ads - its just that they're not obnoxious (and easily blocked) banners. Sometimes, when you do any of the somewhat generic searches, there is an URL returned at the top of the page, above the search results, which is 100% topical, but paid for. Advertising like that, I can appreciate.

Read more about it here: http://www.google.com/ads/index.html [google.com] - they boast a clickthrough rate 4-5x the industry average, and you bet they make you pay for it!

death of USENET predicted, gif at 11

[i0lanthe]

Based on the recent past, I expected this thread to contain a lot of whinging about how "Google took down Deja and my life lies in a shattered ruin about my feet because now I can't post anymore".

I can't tell you how pleased I am to instead see whinging about how "Google plans to let people post". I could understand this being a tragedy and commiserate if, say, the September of 1993 had actually ended.

I'm also tickled to see the phrase "quality web publishing" used with a serious face. ;-)

(Actually, it does make a nice change from "The People demand that Google open source [sic] the archives!" vs. "I'm going to sue for copyright violation!")

Re:USENET is a public forum already

[ktakki]

Historians love to read old snail-mail -- reading letters written by Victorians tells far more about their culture than any books written in the time. What should it be any different for USENET?

"MAKE.MONEY.FAST". Green Card Lottery. `P`H`E`R`O`M`O`N`E`S. Speed Seduction. C-A-B-L-E D-E-S-C-R-A-M-B-L-E-R-S. "Re: Songs about masturbation". Meow. Fuckhead cascades. "RE: re: Longest Thread Ever". Cocaine Pile #107.

Years from now, historians will be calling this the "SPAM Age", as in "Early SPAM Age Man lacked the augmented optics needed to filter advertisements from his field-of-vision".

k.
--
"In spite of everything, I still believe that people
are really good at heart." - Anne Frank

Re:Possible to get Usenet archives from US Gov?

[unitron]

They'll use the national security criteria to justify witholding the knowledge of whether or not they have any such archives.

Nobody else?

[fishbowl]

NOBODY else kept archives of Usenet? Not even the
core heirarchies like comp.* and soc.* ??
That's very surprising to me. It's not like
dejanews was ever that good, that *nobody* else
needed to keep a usenet archive.

Talk about your single point of failure...

Re:USENET is a public forum already

[Cloudmark]

I have to admit I'm of two minds on the matter of the maintenance of the USENET archives. While there is merit to the statement that the Library of Congress (or another national equivalent) has little or no need for the USENET archives, and certainly cannot be particularly interested in bearing the cost of their maintenance, they are a valuable resource. The archives can be seen as a sort of "racial memory" focussed on the period of transition to an information oriented culture. As another poster stated, we gain a great deal from the letters of the Victorians that they likely thought we would at the time of their archiving. However, the suggestion that a government library be responsible for them is not particularly different from leaving them under the control of Google (which paid for them, adding another significant factor to this equation). I think that, if Google is willing to release the content of the archives, it should be maintained by an international organization, in the same spirit as the W3C. Advances in distributed storage and processing make this easier to accomplish than it would have been even a few years ago. After reading the Wired article, there seems to be some indication that this might actually take place, and I must say I would applaud it. Something as valuable as the collected discussions of a developing community should no more be left to a single nation than it should be left to a single company. I would like to state, though, that I minored in History, and am biased towards the maintenance of records, as I encounter far too many situations where our understanding of the past could aid us in the present, but we lack any means of gaining that understanding. I also know the value of letters and seemingly pointless writings (at least those that I have had to make use of :-)

Re:They have to "Open Source" it.

[unitron]

What if, when they return it to you, the ones and zeros aren't in the original order?

Are they obligated not only to delete their copy of it but correct your oversight in not saving a copy for yourself?

Re:Google WAY better than Deja - even w/o archive!

[Col. Panic]

I think Deja dropped the ball when they "improved" the site to include all the ads and such.

One word: junkbuster.

Re:Calm down... Sorry, not just yet...

[fognugen]

"we'll be rolling out a variety of new features such as message posting and enhanced newsgroup browsing."

Reading this statement makes it sound like Google is preparing to transform a historically public and open forum into a proprietary gateway.

Praise Google all you'd like for saving the archive and adding "features", but I'd be weary of how they manage this system and it's content in the future. The acquisition will probably do just as much for USENET as Geocities has done for quality web publishing.

Re:Library of Congress?

[eudas]

if the library of congress had an archive of the alt.binaries.* hierarchy, including all the warez groups, that would be a pretty funny legal quandary. they'd be obligated to put it up (it's the public's archives!) and obligated to take it down (it's copyrighted material!) at the same time... hee hee.

eudas

Cut Google some slack -- they're the savior here,

[Scrymarch]

Exactly. And in the grandest western capitalist tradition, they're going to make some banner advertisement money out of it, too.

Piss-poor reporting on wIrED's part

[Anonymous Coward]

Pretty much everything this moronic reporter claims is no longer possible, is. The archives are a subset of what deja once had, but they are a superset of what existed for the last year.

Specifically, it is possible to:
search the alt. hierarchy
follow threads
search by keyword

It's baffling to think how this story could have been written with such pathetic fact-checking.
Have a look here:
http://groups.google.com/googlegroups/help.html

Maybe it's just the wired/hot-bot/lycos connection taking random pot-shots.

Re:Still should be archived elsewhere

[geomcbay]

Who is going to pay for the storage and the bandwidth?

Re:How much?

[Col. Panic]

You make a good point. A friend and I were recently discussing that very question. I would easily pay $50 per year to access the archives since 1995. I use them every day for troubleshooting or just to read and post in the comp.os.linux.* threads.

Re:What about copyright?

[arafel]

>Most were posted by the same crackpots who add X-No-Archive headers to their posts

Why does putting that on my posts make me a crackpot, exactly? :-)

Re:Google makes REAL money from ads

[MrBogus]

I'm aware of how google places ads. The question is if it's profitable.

Re:Clarification

[Eric Smith]

Even registered works don't all end up in the Library of Congress. They don't accept everything (due largely to space limitations). And for some classes of works, such as source code to computer programs, the registration isn't even required to contain the entire work.

IMNSHO, due to the way Congress has extended copyright duration to ridiculous terms, they should require full deposit of source code (in its entirety) for all software that is registered, make that source code available for inspection (but not copying), and guarantee that it is preserved for the copyright term so that the purpose of copyright law (making sure that works eventually enter the public domain) is not completely thwarted.

Re:Submitter was wrong.

[Denial of Service]

Google only has the archives from August of 2000 and after up on the Web at the moment.

Talk about eeeeeuseless. I'm willing to bet that there are $10 HAM radios with a better signal to noise ratio than Usenet in the last six months.

---

Re:What about copyright?

[Tim12s]

I am sure that the value of the content far surpasses the cost of the copy.

Value: analysis of problems, recommendations in areas, poetry, stories, jokes, designs, solutions, ideas.

Cost: man hours spent maintaining the database, economic cost of spending money archiving usenet, hardware, etc.

For this reason, that archive should never be deleted and preserved by the U.N.

You know... There should be a non-profit archival system accountable to the U.N. for the internet. Similar to project Gutenburg in its aims.

Many "services" are fundumental to the future of the human-knowledge-network that allows researchers to quickly access information empowering them to do what they want.

(researcher = anyone from academic to child)

Re:USENET is a public forum already

[NonSequor]

What? You mean what I was taught in school was wrong? You've just undermined my trust in everything I've ever known.

"Homo sum: humani nil a me alienum puto"
(I am a man: nothing human is alien to me)

Donate to Google?

[bob_jenkins]

I want to buy empty ads for Google and write it off as a tax donation. That way Google keeps running, they don't run ads, I've helped humanity help itself, and I get a tax writeoff.

Is that possible? Legal? It should be. It seems like more bang for the donated buck nowadays than donating to libraries.

Re:Barely got out the door with the data

[bughunter]

Yes, seconded. Especially in the comp.sys groups, there's still enough folks participating to make Usenet my first resort for tech support. There are a few exceptions, but generally Usenet is faster and exhibits more expertise than the typical vendor tech support - some schmoe with a headset telephone and a three-hole punched FAQ. But you do have to have enough skill yourself to sort out the bullshitters from the real gurus. And wow - you should have seen the discussion on rec.sports.auto.nascar this week!

Will I be able to get it all?

[Ayende Rahien]

Will I be able to download the entire deja archives as well as means to access it?
I wouldn't mind paying for a CD/DVD (how big is it, anyway?) of it.
One of the thing that really bugs me about search engines that they (undertandably) don't allow direct SQL queries against their data bases. Well, that and the ads.

To be able to refine my search as *I* wish it would be a great thing.

Anyone has any data about it?

Re:Does anybody read the linked articles?

[geomcbay]

If amazon fails, I'll be glad. Their patent abuses overwhelm their usefulness. And there's always other ways to obtain books.

eBay is currently in no danger of failing. They are one of the few dotcoms that consistently pull in a profit.

If Google fails -- that would be tougher. I love Google. But I used the net before it existed and I'll use the net after its gone, should it die.

Oh no.

[Seumas]

Just what we need. I'm sure if the Library of Congress takes control of it, we will NEVER have the ability to Nuke posts from the archive . . . Not to mention, they'll probalby ignore the X-NoArchive pragma altogether.
---
seumas.com

Re:What about copyright?

[raju1kabir]

USENET postings have expiration dates, which state to recipients how long the authors intends for them to be retained on public servers.

A tiny minority do. I just grepped through several thousand sitting in the spool here, and 47 articles had expiration dates. Most were posted by the same crackpots who add X-No-Archive headers to their posts. Expires: headers are basically irrelevant to the discussion.

That's nonsense. There is no "work" involved in archiving USENET postings. DejaNews didn't manually classify or edit articles. What you need for USENET archiving is storage and a simple script.

Storing a few weeks of Usenet isn't that complicated (but it's more than "a simple script"). Storing and being able to retrieve several years' worth is something else entirely. Come back to us once you've actually dealt with terabytes of data being randomly accessed by millions of people.

Clarification

[localroger]

The library of congress contains just about every copywrited work ever written.
That is NOWHERE near true, alas.

Only because the LOC can only contain works which have been registered. Copyright law currently recognizes your IP right (whether you like it or not) to anything you create at the moment you create it. Registration, which will get your work into the LOC under appropriate circumstances, is only a tool to strengthen your copyright which you have anyway if you are the creator of a new work. Of course, if you create it it's copyrighted by default but the LOC doesn't have a copy, which is true of many of the works ever created.

Library of Congress?

[Daemosthenes]

So does this mean that the Library of Congress will now be home to the greatest collection of ASCII porn ever assembled by man?


47.5% Slashdot Pure(52.5% Corrupt)

Re:Why they should open-source it.

[Sloppy]

I think that there's definitely a moral, if not legal cause to call for the open sourcing of the database. People who've posted technically own copyright on their postings. I think that it could be argued that most of us have expected that it was on the understanding that the postings be made freely available.

The postings were freely available. Deja/Google never did anything to restrict that freedom. Just because someone builds an archive, that doesn't mean they have to share it. If you clip an article from the newspaper and save it in a scrap book for a few years, do you then have the obligation to show your scrap book to anyone who asks to see it?

Dejanews was useful (and I suspect that by the time Google's people get done writing their software, Google's front end will be even better). But it really bothers me that so many people think that the services and labor that went into building the archive, should somehow be nationalized or forced into public domain, just because so many people want them. Deja never had a monopoly; anyone can archive Usenet (especially now that storage is so cheap).

I hope Google opens things up too (that would be really nifty), but they're under no moral obligation to.

---

Re:How much?

[John Miles]

In the early 1990s, I had a $2K/year subscription to ZD's Computer Select [uwaterloo.ca] , just for my personal use. Once I discovered Deja News, I was able to cancel that subscription.

So the answer would be: at least a couple $K a year.

Unfortunately, it never occurred to the rocket scientists at Deja that people might actually be willing to pay real money in exchange for real value. If they'd stuck with their core competency and implemented a subscription model, none of the bitching, moaning, and gnashing of teeth we're seeing now would be necessary.

They are not going to open source

[SquadBoy]

the archives the archives have no source they are not code. They are going to make them "free". I would guess under a license that does not make the public domain but it is not open source by definition open source is code. Sorry but this has been bugging me for a long time.

Re:Barely got out the door with the data

[crisco]

You're right.

While many web forums offer a search function, this is useable at the site only and not indexable by net-wide spiders (such as Google). While in some cases this is a feature, it locks up the content in a way that prevents it from being found, used and archived by net users in general.

I know a few subject matters very well and am happy to be helpful to pass on knowledge, answer questions and participate in dialogue. When this becomes lost I have to answer the same questions again and again, wasting my time. Furthermore, my answers that may be of help to others are lost, depriving them of knowledge that may have helped them.

I have surface knowledge of a great many more topics. I research these, I try to further my knowledge in some, I have to learn about others for work or for other reasons. Being able to easily find information is invaluable and my publicly archived questions may be useful to others.

I know little or nothing about an even greater range of knowledge. Being able to read what others have asked and answered is a wonderful way to start bridging those gaps.

Unarchiveable web forums, mailing lists that don't archive messages on the web and even IRC let this human knowledge slip away.

Not that there isn't a place for all of the above, but I wish more people would consider things beyond their immediate needs.

Articles on the Merger

[skatedork]

First, Is Google Up To the Task? [monstro.com], and secondly, What Harm Has Google Already Caused? [weblogs.com]".

Re:Sell the volume on dvd's? Possible...

[CaseyB]

If you get 10:1 (a stretch) compression that's still 20 DVD's.

When is the last time you looked at a typical Usenet thread?

Repetitive messages of 99% duplicate text with "Me too!" appended. Identical spam copied to thousands of groups. Heck, September ("newbie month!") from successive years probably has nearly identical content.

Usenet is a compression expert's wet dream. :)

Google WAY better than Deja - even w/o archive!

[ahigh]

Personally, I think Deja dropped the ball when they "improved" the site to include all the ads and such. I think the google interface, even though it's more limited, is light years ahead of where Deja was going in the last few years. Deja was soooo slow .. it went from like 2-3 seconds per page to like 20-25 seconds per page a few years back (when the ads came in with the new interface). Now with Google, it's lightning fast. Since google has taken over, I guess that I have benefitted from the improvement in speed that Google has to offer that in my mind makes the service much more valuable. I especially like the ability to view multiple responses simultaneously and the highlighting is much better. I can't say enough good stuff about the google interface, and as far as what's missing from Deja .. the _only_ thing I miss is posting .. Even the older articles I used less frequently and am willing to sacrifice in the name of the tremendous speed!

I use groups.google.com at least 10 times every day .. it's a major part of my very existence.

--
- Aaron Hightower - Lead Programmer - Rush2049 Coin-op

Re:Possible to get Usenet archives from US Gov?

[Denial of Service]

My FOIA inquiry got a PDQ PFO from the CIA. They can FOD, FWIW.

---

Re:What about copyright?

[raju1kabir]

>Most were posted by the same crackpots who add X-No-Archive headers to their posts
Why does putting that on my posts make me a crackpot, exactly? :-)

Because it's so much like pissing up a rope.

If your post was of any interest, it got quoted by all sorts of people, 80% of whom couldn't be bothered to add the header to their posts, and the other 20% of whom do want their posts archived and available.

Also, just because deja.com supported it doesn't mean that other archives did, so at best all it did was break threads when viewed through one particular site.

All of which is fine, except that so many people were so darn dogmatic about it, screaming at other folks who neglected to add the header to quoting posts... which is just plain silly. If you don't want your words saved, use the phone. If you don't want them attached to you, post anonymously. But don't yell at other people just because they're less weird than you are (not you personally).

Google will rebuild it - stronger, faster, better

[jaffray]

First Deja shifted from Usenet towards their lame shopping thing. Then they changed the Usenet searching interface to make it far less usable and more ad-heavy. Then they started inserting ad links into posts. Then they made all posts before ... June 2000, was it? ... "temporarily inaccessible" for many months, negating much of the value of the archive. And then they tanked; had nothing been done, they would have taken their entire archive with them.

In the middle of this, Google, the company with the absolute best search engine on the net and the most usable web site with the least ads and the best friendliness to geeks like us of any site like them, steps in and takes over the archive and promises development work to make ALL the data more available and more usefully searchable than ever.

This is GREAT! As a long-time Deja user, this makes me very happy and excited. Yes, it sucks that the archive is inaccessible for a few weeks, but that's not Google's fault. Deja shut down the servers, not Google - why on earth would Google ask them to? - and Google's providing some minimal functionality in the meantime. It beats the hell out of losing the information entirely.

Anyway, personally I'd choose some interruption of service with Google maintaining it afterwards, as compared to the status quo with Deja even if it were viable (which it wasn't). Having all the posts from 1995-1999 inaccessible was NOT an acceptable situation from my standpoint as a user.

I'm delighted that Google has done this. Mad props to the Googlistas responsible.

I don't like the Library of Congress idea..

[lpontiac]

.. because I'm not a citizen of the United States of America.

I don't pay taxes to the US government; they have no jurisdiction over me, and hence no obligations to me either on either moral or legal grounds. So why they might choose to make their resources (say, a Library of Congress USENET archive) available to me as a courtesy, such a 'right' to their newsgroup archives would be even more tenuous than the relationship between me and a company providing archive access to customers. (Be the customers paying fees, or viewing ads, or whatever).

So if the archive ever did go to the Library of Congress, I would encourage them to make the archive available for high-profile mirroring; if the National Library of Australia had a copy I'd feel a lot better.

Why I trust Google

[Lumpish Scholar]

I've been using Google since it's alpha test days. Back then, it was a very promising search engine with some serious limitations. (Example: you couldn't search for "AT&T" because all single letters were in the exclusion list of too-common words, so "AT&T" was treated as "AT".)

I've been pleasantly astonished at how Google has improved over the years. Even when they added advertisements, the ads didn't suck: they were on-topic, small, and loaded fast.

Yes, the current Google interface to the Deja archives isn't great. A lot of functionality is gone. Do I expect Google to make huge improvements in the next few months, even weeks? Based on their track record, yes, I do.

What I write on Usenet is *MY* copyright

[evilandi]

Sorry but I don't like the idea of some foriegn government (particularly the US) having control over my articles.

Given that what I write on Usenet is *MY* copyright (by default under the Berne convention- I don't have to put "Copyright 2001 Andrew Oakley" on my articles, just put my name), I presume I can deny the US government the right to use my works.

--

Still should be archived elsewhere

[Goonie]

You're right in that, from what I can see, Google deserves fulsome praise for attempting to save this important historical resource. However, the fact remains that because of its very importance, the preservation of the USENET archive shouldn't just be left to the fortunes of commerce. Librarians are the experts in the long-term storage and archiving of information, and an instrument like the Library of Congress is an appropriate body to do so - even so, *several* bodies in several nations should be archiving this stuff, not one.

Re:Google WAY better than Deja - even w/o archive!

[MrBogus]

Problem was that the number of WWW users was growing at 10x times a year for quite a few years. Which left deja in the position of having a supposed massmarket service when there were 100 users who had no clue what USENET was for every 1 user that did. Not to mention that archiving USENET never made them a dime in profit.

So they had to try something to save the business while they still had some capital left. What they tried was stupid, sure, they couldn't just sit there and go out of business.

Google has a much better strategy. Google's real product is sold to Yahoo and others. Google.com, on the other hand, is specifically target- marketed towards computer nerds, which explains the excellent indexing of Linux and Unix related issues. This allows them to sell ads for top dollar, but whether they make any money of google.com and usenet archives is a different story.

hey- it's all coming together...

[fatmantis]

all you 1337 war3z k1dzi3z and mp3 pirates listen up- here's the plan...

Deja never bothered to archive binaries. and google won't either... but I think I've figured out a way to post binaries directly to google's caching system- break the file up into 98k pages, hosted on a free system like geocities. imbed a title, description and a link to an index of all the other segments of the file at the begining. Voila`! instant free binaries archive.

thank you.

Why should Google have to do this?

[Meg Thornton]

The tone of the article was such that it implied that Google should be providing this information to the public at large, simply *because* they bought it from what was Deja.com. There's the accusation that Google are doing something morally wrong by taking the archive offline - meanwhile ignoring the fact that Deja.com had already taken a large portion of the archive offline with little or no warning. Google, apparently, are online villains of the deepest dye for wanting to get some form of commercial return for the money that they paid to acquire the archive in the first place.

So let's start from first principles here: the fact that Deja had such a comprehensive archive is not remarkable. The remarkable bit is is that *nobody else has done anything similar*. Deja's value as a resource, both in the commercial sense, as well as in the historical sense, is in its rarity. Goggle, in acquiring the deja.com archives, *prevented* this resource from being lost forever. Yet they're apparently villains for not immediately doing whatever the Open-Source community wants them to. Talk about bloody-minded ingratitude.

There's an argument being made that this information is ours already, although from what I understand, this is legally problematic. However, if you don't agree with Google being able to commercially exploit *your* precious Usenet postings, the answer is straightforward: start posting with "X-No-Archive: Yes" in your headers, and write a *polite* email to Google asking them to remove all your posts from their archive.

For myself, I'm quite glad to see that Google have obtained the archive, and if they do as good a job of running it for easy access as they have with their search engine database, I'll be extremely pleased.

Meg Thornton.

Re:USENET is a public forum already

[Jonathan]

Historians love to read old snail-mail -- reading letters written by Victorians tells far more about their culture than any books written in the time. What should it be any different for USENET?

Why they should open-source it.

[Stephen Samuel]

I think that there's definitely a moral, if not legal cause to call for the open sourcing of the database. People who've posted technically own copyright on their postings. I think that it could be argued that most of us have expected that it was on the understanding that the postings be made freely available.

Although that should probably include enough information to access the postings as separate items, there's a little bit less excuse to ask for the searching code (though there's no reason to not at least ask).

I remember that DEC had an archive of older postings in the late '80s/early 90's. I think that it was Gene Spafford that put it together. Does anybody know what happened to the older archive?

(as an aside, I remember a reference to the value of being able to access DEC drives at internal cost price)
--

Re:What about copyright?

[q000921]

The DejaNews service and its scalability aren't at issue.

What is at issue is the archiving of USENET messages, and archiving them isn't difficult. If you can't figure it out, you can even buy commercial off-the-self solutions to deal with it.

Re:Does anybody read the linked articles?

[Scarblac]

What will happen to the 'Net when the next big dotcomm to fall is eBay or Amazon, or Google? Especially since Google's USENET archive and WWW cache have become invaluable to a number of people.

Oh really? I bet a large number of the people to whom the Usenet archive is 'invaluable' would be absolutely outraged if they had to pay some small amount per month or per search. At some point, it's going to be charge or die for most of these companies (well, Amazon is a shop, that's different).

Re:What about copyright?

[q000921]

Traditionally, postings were assumed to expire within a standard period (a few weeks). You could override that with headers, and people used to do that pretty frequently. DejaNews ignored those headers.

Nowadays, I think most peopled don't bother setting headers. But USENET is still a discussion medium. Just because a few companies decided at some point to archive the stuff doesn't mean that the user's presumption should change.

Re:What about copyright?

[q000921]

Actually, I have run USENET servers, and, yes, it can be a pain. But that's not relevant.

For archiving, all you need is a leaf node connection with no users. You don't even ever have to store incoming messages in a news hierarchy, you just send them off to your archival storage system as they come in.

Another choice might be to run a traditional news server, turn off article expiration, and keep the news hierarchy on a file system with a hierarchical storage manager (too heavy-handed for me).

Google doesn't own the content.

[q000921]

News articles are implicitly owned by the author, and they expire. DejaNews already has violated copyright by keeping the content longer than its expiration date. Redistributing the archives as a whole ist just not acceptable as far as I'm concerned. USENET is a discussion forum, not a long-term archive.

Google should become more compliant with copyright law, not less. But, then, the company blatantly copies and retains other content as well ("cached pages").

Caught my eye

[timmyd]

Does slashdot have to use this buzzword everyday? "open source" newsgroup archives? since when were newsgroup archives closed source---or any source.

Some random comments

[LS]

First, as others have said, the archive is still there, at the same address (groups.google.com).

Also, the Wired article mentions that a single person called for the release of the archive, but no mention is made of a response from anyone at Google or the Library of Congress. So what? I call for Microsoft to release the windows source code and have the DOJ supervise it. Where is my Wired and Slashdot article?

There are multiple archives in existance, and Deja owns one of them. They collected it and maintained it privately, and they nor google owe you alimony because of the lifestyle you are used to.

Are there no other public or private archives out there? It seems strange that only there is only one service available. I recall a public web and usenet archive out there that stored a limited chronological range, but I don't remember the address...

I don't doubt that google will eventually do a good job with it, even though it does suck in it's current form.

LS

Why would they open source it?

[alteridem]

Google has taken the archives off the web for now Aaagh!

Google has taken the archive down only until they can integrate it with their own archive. Once this is done, it sounds like we will once again have a reliable source of old newsgroup postings.

I highly doubt that they will ever open source the information though. The terabytes of data that they purchased as a part of deja.com is probably the most valuable part of the deal. Why would they then want to turn it over to the government? What financial incentive is there for them? The only way they are going to recover their investment is to create a service like Deja's, only better and integrated with their own.

The following is from Google's press release on their aquiring the data;

Available now at http://groups.google.com, this powerful new Usenet search feature enables Google users to access the wealth of information contained in more than six months of Usenet newsgroup postings and message threads. Once the full Deja Usenet archive is added, users will be able to search and browse more than 500 million archived messages with the speed and efficiency of a Google search. In addition to expanding the amount of searchable data, Google will soon provide improved browsing capabilities and newsgroup posting.

Barely got out the door with the data

[swb]

A friend who works at Google said that they got the archives "barely" -- they were apparently copying data as technicians were tearing apart what was left of Deja's systems and hauling the equipment away.

I got the impression that there was a lot of work to be done to fix the data so it was in a coherent form, much less fit into Google's existing storage and databasing environment.

As long as they're still collecting news, plan on improving the existing search engine (my source says yes to this one thing) and it remains free-as-in-beer I'll be satisified.

What kills me overall is the decline in the overall quality of USENET. Too much good content has gone to crap, non-archived, non-searchable web forums (ahem) and what's left on USENET outside of a few newsgroups is spam, porn and isn't worth the time to search.

I got quite mad

[*xpenguin*]

When they took down the old deja, I was quite mad. Google's way of viewing the messages sucked. I can't post there anymore either. So I found news.interbulletin.com [interbulletin.com]. It is a usenet service that allows you to view about 30,000 newsgroups and lets you post under any name. It also uses frames which I like becuase the whole page doesn't have to be reloaded. Their server is very slow right now becuase lots of deja users switched to their service. Its usually around 200/bytes per second on my 56k modem. Anyone who was mad too you can use news.interbulletin.com for a while and hope the old deja.com will sometime come back alive (probably not)

--

Re:Sell the volume on dvd's? Possible...

[the eric conspiracy]

Well, they are talking about 1 TB for the 5 year archive. At 5 GB per DVD uncompressed that would be 200 DVD's. If you get 10:1 (a streach) compression that's still 20 DVD's.


MOVE 'ZIG'.

Re:What about copyright?

[AaronStJ]

Hm. Does Google own the database of Usenet postings?

Google does indeed own a copy of the database of usenet postings. More on this later

You see, since every person ever write to the Usenet still retains copyright to their postings, isn't it in the slightest bit illegal to actually *sell* the database? Or at least immoral?

This is a funny bit of Usenet culture/law. While it is generally accepted that usenet users are giving others permission to copy there works they *do* retain copyright. So why can deja go around selling this work? IANAL, but here is how I see it, I think I'm (mostly) right.

1) When you post to usenet, you're sending your work to whatever every archives are in place, and you know it. By posting, you are giving any other user permission to view and archive the material. In fact, you yourself are commanding that the message be forwarded to all other connected computers, and therein lies the implied permission.

2) This strikes me as an important point. What deja.com is selling is not the rights to the posts, or the posts themselves, but the work that they put into archiving the posts, which is considerable. It is the same way that free software sell CDs with open source programs on them. They are selling the data itself and the work that went into collecting the data, not the rights to the data. So, while you may have put a lot of effort into writing that post for alt.silly.rantings, deja.com didn't sell that work, deja.com merely sold the work that went into collecting your work.

Do you see what I'm saying? Or am I just rambling?

Question?

[mindstrm]

How do you 'open-source' something that isn't source code? Usenet archives are not 'source'.. they are documents.

Re:What about copyright?

[q000921]

When you post to usenet, you're sending your work to whatever every archives are in place, and you know it.

USENET postings have expiration dates, which state to recipients how long the authors intends for them to be retained on public servers. Keeping postings any longer than that looks like a pretty clear violation of copyright. The argument that "you know when you post that..." is bogus. Music publishers also "know when they publish CDs" that their music will get copied, but that doesn't invalidate their copyright.

The fact is that DejaNews got away with this because they were big, and, hey, what could a random USENET poster do?

What deja.com is selling is not the rights to the posts, or the posts themselves, but the work that they put into archiving the posts, which is considerable.

That's nonsense. There is no "work" involved in archiving USENET postings. DejaNews didn't manually classify or edit articles. What you need for USENET archiving is storage and a simple script. The whole USENET system is designed for easy, reliable replication and archiving. Yes, the storage costs money and it is "work" to buy new CDs and disk drives. But a "software pirate" doesn't acquire a copyright to the stuff he is copying just because it is takes time for him to copy the stuff.

What's the BFD?

[MrTilney]

Ok, I believe pretty strongly in the open source movement and all that goes with it, but some people just love to complain.

Ok, here goes. Usenet posts are public, so anyone can copy them (although they do all exist on private servers-except universities etc, may want to talk to a copyright lawyer). So, Google bought a copy of these posts from Deja. If others wanted a copy of these, they could have made one.

So, now, let me get the straight, someone had the foresight to make an archive of these posts. Then they had the gall to give access for free. Then, when they couldn't make any money (running a server that big costs a fortune) they had to sell it. Now, people who have been benifitting from the debt of others are mad because the owners of the archive (which anyone with a lot of tape could have made) want the government to run it? I repeat, THE GOVERNMENT!!! Are you kidding. Face it, doing something that anyone could or should have done does not indebt you to society because you decide not to do it anymore.

And that whole bit about the underground Dela guy sounds like some guy trying to make himself famous. If he was so great maybe he would give his actual name. Famous open source people actually do something. Not try and get someone to give you all of their work (a search tool for Tux's sake) so you can do nothing.

Let's stick to stopping Microsoft, RIAA, the MPAA, and the government from taking all of our rights away. Google is one of the best net companies right now. They make a useful product and don't bury us under banner ads. Plus they have a kick ass search engine. Give 'em a break before you try to treat them like Bill Gates.

Data

[harmonica]

I don't really understand what's so interesting about Deja's code. Should be no major problem to create a search engine / interface with all the code that is out there for indexing etc. and all the capable people willing to write / enhance free software.

The archived postings are the interesting part. At groups.google.com it says that there is a terabyte of data. Maybe it could be made available for download per FTP, one tar.bz2 file per month per newsgroup. Different projects could then try to use the data... Tools like MG [freshmeat.net] (Managing Gigabytes) can create an inverted index that reduces textual data to about 40 percent and is searchable. Well, that's still 400 GB, but HDDs are getting cheaper all the time ;-)

Re:Today is a sad, sad day!

[scarhill]

Results pages are also strangely similar to Yahoo's results for the same search.

Um, that's probably because Yahoo is now using Google as its search engine. See this press release. [google.com]

Does anybody read the linked articles?

[Carnage4Life]

As I am posting this there are 5 posts moderated to +3 or higher and it is quite obvious that none of these posters actually read the article. Anyway here's the capsule summary for all the moderators and posters who refused to read the story.

Google does

not plan to "open source" or give away the Dej&agrave archives. Instead a lot of former Dej&agrave users, specifically a Frank Davies who is a student and research assistant at Columbia University, would prefer if the archives did not belong to a commercial entity but instead to the Library of Congress or to an Open Source non-profit.

The freaking article is entitle Deja 'Revolt' Against Google, how anyone could have completely misread it and gave the horrible write up we just got is quite amazing.

This leads me to the main question: Major sites such as Google [google.com], eBay [ebay.com] and Amazon [amazon.com], have become a valuable part of the 'Net and have become an intrinsic part of the World Wide Web experience for many people. Yet, these companies are yet to prove their viability and could collapse at any time if their investors grow tired of shouldering their debts and underperformance. What will happen to the 'Net when the next big dotcomm to fall is eBay or Amazon, or Google? Especially since Google's USENET archive and WWW cache have become invaluable to a number of people.

Does this justify asking the government to step in and take over these resources so they are preserved for posterity as Frank Davies and many others have suggested or is would this be undue interference by the government?


Finagle's First Law

Where is the code? Who owns it?

[Wesley Felter]

I read that Google only bought the archives from Deja, but not their code or hardware. So where is that code now?

Re:What about copyright?

[the eric conspiracy]

Does Google own the database of Usenet postings?

isn't it in the slightest bit illegal to
actually *sell* the database?

I would have to think that by the act of posting a message on the a newsgroup you have given permission for it to be distributed and copied via NNTP to the various and sundry news-servers on the net.

Very few of these servers are available on an open basis. ISPs almost always require some sort of compensation for access.

Whether the Deja archive is a news-server or something more woud be a point for lawyers to argue.

I would say that it is quite clear that the transfer of a news-server and it's contents from one commercial entity to another is a common occurance - any time an ISP is bought out this will obviously occur. So the idea of your posts getting bought and sold - get over it, it's already happened, and will continue to happen.

For my own case, I feel that the usefulness of the Deja archive as a source of knowledge far outweighs the loss of whatever small value my postings may have, and as such I happily provide such under the BSD license.

I hope that other Open Source users will take the same view.


MOVE 'ZIG'.

Re:Deja is the only archive?

[simong]

I don't think there are, TBH. There are individual archives of various groups at universities in the US (and presumably in Europe) but even now, when many people claim that Usenet is dead, the daily throughput on a fully loaded news server is huge (Colt's peering news server [colt.net] suggests 23Gb since 1am today which translates to between 3 and 7Gb to its downstream feeds in the last 11 hours). Dejanews made the commitment not only to keep a running archive of Usenet but also to collate those archives stored by other agencies. I think the only other near competitor was Remarq and they fell over last year. Deja's store, as Google said the other day, runs to many terabytes of data and must have a pretty solid dedicated feed (the last time I was involved with news server management was about five years ago and the company I was working for recieved its primary news feed from UUNet's satellite delivery service, a dedicated 47Mb downlink), so it was a major investment to keep it going, hence the attempts to leverage the data by turning it into discussion channels for shopiing and whatever, something that was doomed to failure as, as useful as Usenet is, the signal to noise ratio in the most dedicated of groups is sufficient to make searching a frustating process if only for the number of 'me too' noise that most discussions generate. It's good that Google have managed to secure the archive in that respect, but maintaining it and making it usable isn't going to be easy.

Submitter was wrong.

[jidar]

They most certainly have not taken the archives down. They can be accessed Here [google.com].

What part of Open Source don't they understand?

[boinger]

Oh, wait. Obviously it's the Source part.

Today is a sad, sad day!

[atrowe]

I remember as recently as last year when I could find literally *anything* relating to Linux on either Google or Deja. I cannot begin to relate how many hours I've saved by using these two intuitive bases of knowledge. In fact, I can't remember ever having a Linux related question or problem that I couldn't solve with a quick search of either of those sites.

Sadly, those days of readily available, easily navigable information are long since gone. With Google's recent sell-out to Yahoo, I've been getting results that seem to be less focused on accuracy, and more focused on whoever is paying Google the most for the top ranking. Results pages are also strangely similar to Yahoo's results for the same search. I'm not saying this is necessarily a bad thing, but it's surely a shame to see one of the major backers of open-source information sell out to corporate politics without so much as a fight.

Deja is sadly, heading in the same direction. Back in it's hayday, I could always count on Deja to find those obscure little tidbits of Linux information when I needed it, but now, it seems to be nothing more than a shell of it's former self, providing a haven for spammers, crapflooders, and again, the corporate machine.

It's a shame that what were possibly the last two refuges that us Linux user's have from the corporate machine that is MS, Yahoo, and AOL are apparently being stripped for parts and left to rot at the hands of the almighty dollar.

Archive the Backups!!!

[Baldrson]

At the annual Singularity Gathering last week one of the Udanax guys responded to my concerns about Deja News by telling me backup policy at Berkeley University is to keep the tapes around for 10 years. If true, that means Berkeley University alone has the history of Usenet content going back to 1991!

This means it should be feasible, not only to get at least a decade of Usenet archives, but, if other universities have similar backup policies, to get multiple redundant, geographically distributed Usenet archives that can be cross-checked agaist each other for historical accuracy.

This may seem like a minor point, but I do know of at least one instance where Deja's archives appear to have been tampered with -- and it would be unrealistically naive to expect that there is no incentive for certain parties rewrite history.

Finding the spots where people may have tampered with history could turn out to be as interesting as anything that comes out of the Usenet archives.

USENET is a public forum already

[pgpckt]

I have to admit, the immediate goals of such a project illude me. USENET postings are already public. These are open forums, and the groups can be read from most libraries or other public sources.

I don't see the value in the long term achival of USENET posting. The library of congress contains just about every copywrited work ever written. This serves not only as a national archive of our author's produced works, but gives our legislature access to the documentation and research they need to do their job. Would the archiving of USENET posting serve the long term mission of the nation's library?

It also bothers me slightly to think that people's comments and flame wars will langish forever in the federal library. I don't think access to USENET postings is something the nation craves or needs. What the nation needs is access to works that have been researched and published, works from professionals. The library of congress is a lbrary of professional works, not the "my 2 cents" postings that tend to dominate USENET frequently.
----------------------
Kurt A. Mueller
[e-mail]
PGP key id:0x4FB5FB1D

What about copyright?

[Ecyrd]

Hm. Does Google own the database of Usenet postings? You see, since every person ever write to the Usenet still retains copyright to their postings, isn't it in the slightest bit illegal to actually *sell* the database? Or at least immoral?

At least I am giving no permission whatsoever for someone to sell my posts...

Someone could argue, though, that by posting to the Usenet you have implicitly maken your work public domain, but I doubt that you can get rid of your copyright that easily. Books still have copyright, and you even paid money for them, so shouldn't you be getting more? :-)

Re:USENET is a public forum already

[AaronStJ]

What? You mean what I was taught in school was wrong?

Yes. We all thought you'd figured it out already. We didn't mean to break it to you like that.

You've just undermined my trust in everything I've ever known.

Someone had to do it.

Re:What about copyright?

[ZanshinWedge]

In my opinion usenet postings are not public domain, but as long as they are kept unmodified (essentially) I don't see any reason why they can't be archived for ever. It seems that people implicitly allow this usage of their (copyrighted) posts when they post on usenet. i.e. they agree that their post can be stored and distributed in normal usenet fashion. Plus, there is a way to prevent your post from being archived (the X-noarchive: header) so I don't see any problems with this long term archival of usenet posts.

Now, if they modify the posts or do something wacky with them (like embed advertisements in them as deja.com was doing for a while) that's a whole different kettle of copyright law.

Re:USENET is a public forum already

[CaseyB]

It also bothers me slightly to think that people's comments and flame wars will langish forever in the federal library. I don't think access to USENET postings is something the nation craves or needs.

"We've all heard that a million monkeys banging on a million typewriters will eventually reproduce the entire works of Shakespeare. Now, thanks to the Internet, we know this is not true."

- Robert Wilensky

Re:Submitter was wrong.

[pclinger]

The submitter was correct. Google only has the archives from August of 2000 and after up on the Web at the moment. Currently the archives going all the way back to 1995 are offline.

Re:They are not going to open source

[Tiroth]

Well, the /. comments are poorly written. If you read the actual article the "open source" bent is more towards allowing open source access to both the archive and the old Deja code that mined it.

That is, open source the database access so others can write front ends, and give out the old code to be publicly worked on.

2 options

[Pfhreakaz0id]

We're going to have to accept the realization that for having public archives of Internet material, there are really only two options: 1: Collectively do it ourselves through taxes and the government(s) (what I'm in favor of) or 2: Actually have a pay service that you pay a monthly fee for. It simply requires to much manpower and storage otherwise. I mean, can you imagine a library operating through selling advertising? Aside from the practical business realities, would you want to live in a world so corpratized that you have to endure ads in the library? Or strategic "upsells" and "click throughs" that mean when you look in a how-to book a speaker comes to life and says you can buy that saw for $49.95 at Home Depot? (this is pretty similar to what Deja was doing in its late incarnation).
---

Google announces....

[GoRK]

Google today announced plans for an open-source public celebration of the event. Attendence will be open-source as guests will not have to pay to attend and there will be open-source food for all. Though bands have yet to be announced, there will be an open-source concert to entertain partygoers at this historic event.

Public domain is not the same thing as open-source you fucking morons at /., Wired, et. al.

~GoRK

Oh, yeah, and since this comment is open-source for you to read, i'm selecting to license it under the LGPL so Jon Katz can go publish it in his damn book if he wants and I dont have to get my colon in a knot over it!!!

Slashcode NNTP?

[Pseudonymus Bosch]

this is useable at the site only and not indexable by net-wide spiders (such as Google). While in some cases this is a feature, it locks up the content in a way that prevents it from being found, used and archived by net users in general.

Actually many static pages of Slashdot are included in Google.

I remember that before the "real" publishing of Slashcode (1.0?), lots of people were saying that they were waiting the source to implement NNTP output from Slash. Did any of them succeed?

I suppose that anyway Slashdot wouldn't enable it as it goes against its business model.
__

not fair

[stigmatic]

Well if Google plays into the Al Gore tune, and doesn't concede within an ample amount of time here's a nice list for someone to wget/snarf up old Usenet archives. [google.com]

I wonder what if any, are the underlying factors for this move.

Chicks worthy of Usenix pornspam status [antioffline.com]

The Internet Archive

[Pseudonymus Bosch]

What will happen to the 'Net when the next big dotcomm to fall is eBay or Amazon, or Google? Especially since Google's USENET archive and WWW cache have become invaluable to a number of people.

Does this justify asking the government to step in and take over these resources so they are preserved for posterity as Frank Davies and many others have suggested or is would this be undue interference by the government?

I think that Google is the main contributor to the Internet Archive [archive.org], a non-profit that holds teras and teras of Geocities pages.
__

Re:Question?

[PigleT]

I think you know the answer to that. The way Deja treated the articles was as source to stick hilights to products in. "Documents", don't kid yourself.

Why doesn't someone with more Gbs than common sense just mirror the blighter when it comes back anyway? Could presumably be possible to write a suitable bot - heck, we could even have a distributed project to do it for us!
A use for Freenet, anyone?
~Tim
--
.|` Clouds cross the black moonlight,

Re:What about copyright?

[Eric Smith]

There are a lot of legal theories about this, but no case law. On theory is that since the messages are by their very nature expected to be copied to an arbitrarily large number of computers, they are public domain. If this is the case, people who put explicit copyright notices on their posts may be accomplishing nothing. Another theory is that the act of posting grants an implied license to reproduce the post. The scope of such an implied license is unclear. It is not clear that selling a collection of Usenet postings infringes any rights, since ownership of the postings is uncertain.

Re:What about copyright?

[mr3038]

...the work that they put into archiving the posts... It is the same way that free software sell CDs with open source programs on them. ...

So now that Google has a copy of its own, would it be legal for Google to sell another full copy of the database to someone else? In my opinion, answer should be yes. Exactly because the points you've written. Another question is whether or not Google is allowed to do this by law. Lawyers can make miracles, unfortunately.
_________________________

More on The Internet Archive

[gbnewby]

I sent the below as feedback to wired.com. The Internet Archive doesn't have a nearly complete Usenet archive, but they've got some and could accept more from other sources. They're NOT the place to run a search engine, but ARE the place to hold the content that could go into a search engine.
--------

Nice article, but you need to use your own resources a little better. Wired has had several articles about Brewster Kahle and The Internet Archive (archive.org).

TIA is *already* archiving Usenet, although its coverage has been less thorough than Deja. They're a .org with the clear mission and mandate to archive Usenet, and make it available to those who want it. They're not in the search-engine business, but would supply the content to those who want to build one.

Hopefully this info will be good for a follow-up article.

X-no-archives, nuking, IP, public forum: Re:Oh no.

[mami]

Just the contrary, that's exactly what I hoped would be accomplished, if LOC would get involved.

Everybody says usenet is a public forum, but the medium on which the forum were uphold and their public speech recorded, was never public. Not all the newsservers and connections were government owned and paid by people's tax dollars.

The usenet archives collected by deja news, the data, are owned by a commercial company. They can decide whatever they want to do with them, the IP value of the posts is neglible, but the monetary value of a complete set of data of all archived posts is high. The fact that deja-nes.com originally treated the data from usenet groups the way oldtime usenet users were expecting them to be handled, doesn't mean that they couldn't have done differently.

I used to use deja-news.com since early 1997. At that time deja-news.com was still ok, but usenet itself was a pretty mess. Many technical worthy groups were complaining about serious flamewars and thousands of useless posts. Old usenet users were whining and crying out for the good ol' times, when men were real hackers and did their business "among themselves". Deja-news.com was respecting poster's right for nuking their posts out of the archives, something which I consider a right, which needs to be protected, as well as posters x-no-archive requests.

Later, when deja-news.com turned into deja.com, the whole thing became more commercial. Now, as they can't make a profit, it's sold. The new owner can do whatever they want with the archives. Who guarantees you that they don't go down the drain in a year from now and sell the archives to the ... chinese government:-). To me it would seem to be the best thing which could happen, if LOC could be asked to become owner of those archives. This is in order to protect those archives and the posting rights of the posters from any biased group. This is also to make them available at no cost to the public, and protect them from being distroyed or misused.

If usenet is supposedly to be a public forum, then the public should own the archives of this forum and LOC should get involved and protect those archives for the public. At least then for the first time the archives of a "quasi public forum" would also be owned by the public and the regulations about how to handle the archives is under democratic rules determined by the tax payer and voters.

Why you conclude that a commercial company is more prone to respect those rights to nuke, x-no-archive etc than LOC, who is funded by tax payer's money, I don't understand. The public is posting, so the public can demand that LOC respects those rights.

Contrary, any commercial company would want to archive EVERYTHING against the will of the posters to collect larger archives.

LOC's involvement could be very well the best which could happen, because they would be the most unbiased and most professional archivers.

I would understand if you would say LOC might be the toughest cataloger, because contrary to any open/free source zealots, they are possibly courageous enough to classify and discriminate content while cataloging. That's what librarians usually use their brain for, but of course, I hear already thousands of kids screaming "no, I want my 0.02 cents worth of flamewar/porn archived til the next millenium". If you could fear something from the LOC than it might be their professional scrutiny towards what they consider content worth archiving. I doubt that they would trash anything which is worth to keep, certainly not all the historically important archives of the beginning eighties and the like.

They have to "Open Source" it.

[Auckerman]

Since that archive is nothing more the collection of the copy righted writings of 100s of thousands if not millions of people, they have to "Open Source" it. If they don't I will write a cease and desist letter and send it to Google that they return my copy righted work to me and then delete them from the archive.

Why is everyone making Google the bad guy?

[eschatfische]

It seems like the vast majority of posters are saying that Google has bungled the transition between the old Deja service and their new USENET service. It's fairly clear that this is not the case -- by all indications, Google obtained limited resources from a cash-strapped, poorly-managed company being run by a skeleton crew. As anyone knows who has been through a last-minute purchase of such a company, the purchasing party obtains limited resources and little support from the purchased party, resulting in a period of disarray.

People are misreading Google's press release, which states: "This acquisition provides Google with Deja's entire Usenet archive (dating back to 1995), software, domain names including deja.com and dejanews.com, company trademarks, and other intellectual property." Folks, this does not mean that Google had the ability to just keep the status quo with Deja's existing server farm. Note that the press release says nothing about hardware, staff, or the full intellectual property behind Deja's operation -- it's obvious that Google didn't get a whole lot from Deja other than the archives and the domains. The fact that they're in the process of transitioning the service to their own server farm and search interface hardly means that they made a bad business decision -- it was likely the only business decision.

That's right. Otherwise, Deja would have just disappeared, like so many other .com tragedies have lately. Someone else has already posted on the fact that Deja had techs in the server farm ripping out hardware as soon as the sale happened.

So cut Google some slack -- they're the savior here, not the destructor. Think proverbial silk purse. Nobody likes not having access to the full archives, but I feel confident that Google wants to get them up and running, and they're almost invariably the best people to get the job done.

Just get the date sort working, Google guys, OK? ;)

Eschatfische.

Re:USENET is a public forum already

[CaseyB]

Good lord. At around 100 Gigabytes a day, who does?

They should have waited

[twivel]

I tell you what, I like google quite a bit. I think they provide an awesome search engine and I especially enjoy their platform-specific search options.

Even though I really believe they should have waited until their new solution was complete before pulling the deja site down to avoid interruption of service, I'm going to give them the benefit of the doubt in believing they are going to do the "right thing".

--
Twivel

Re:Archive the Backups!!!

[rhaig]

• ...backup policy at Berkeley University is to keep the tapes around for 10 years. If true, that means Berkeley University alone has the history of Usenet content going back to 1991!

Yeah, but what sort of silly admin backs up usenet spool for transient articles?!?!? It's just usenet! Unless they were decidedly in the business of archiving usenet, backing up news spools is just silly.

Re:Library of Congress?

[StandardDeviant]

Good, it ought to be able to keep Congress from doing anything for at least another few years... ;-)

And if they have archives of the alt.binaries.* or alt.bainaris.* heiarchy, that should keep the Congresscritters occupied for a good loooooong time...

--
"Overrated" is "overfuckingused".

Possible to get Usenet archives from US Gov?

[Eric Smith]

Has anyone tried submitting FOIA requests to the CIA, FBI, NSA, NRO, etc, to try to get copies of any Usenet archives they may have? If they have such archives, it is unlikely that they will meet any of the criteria that would allow them to deny a FOIA request, e.g., privacy, national security, etc.

Calm down...

[the real jeezus]

Okay, abruptly pulling the plug on the Deja archives last week was definitely tacky. However, the inconvenience won't last for much longer, according to Google [google.com]:

"Our goal is to offer the ability to search the entire 5-year archive by the end of next month," Krane said. "And over the course of the next two to three months, we'll be rolling out a variety of new features such as message posting and enhanced newsgroup browsing."

This is really good news, considering the push for open-sourcing the project and the potential cooperation from the Library of Congress. The LoC website [loc.gov] rocks!

If you love God, burn a church!