Google De-indexes Talk.Origins, Won't Say Why UPDATED

J. J. Ramsey writes "Talk.Origins is an archive with thousands of pages exposing creationist pseudoscience. Rather mysteriously, Google pulled the plug on its search engine, giving only the vague reason: 'No pages from your site are currently included in Google's index due to violations of the webmaster guidelines.' This was apparently triggered by a recent cracking of the site that added 'hidden links to non-topical sites,' but Google won't say just what the violations were. Talk.Origins webmaster Wesley R. Elsberry believes that this Google policy harms honest webmasters." From the article: "My mission, whether I liked it or not, was to find and fix whatever problem the [Talk.Origins Archive] might have, with no guidance as to what the problem was and nothing at all about where to start looking... I was extremely lucky. The damage to my site was limited and in the first place that I happened to look. Other honest webmasters might not be so lucky. They may have to undertake an arduous process of vetting pages, essentially having to second-guess the mind of the cracker in trying to locate a problem that Google knows the exact location of." Thanks to an alert reader who sent in Matt's blog posting about how Google handles hacked sites.

huh?

What's this? [google.com]

Re:huh?

That's the Google Groups archive of the talk.origins newsgroup, which is a different animal (an ancestral form, one might say) from the Talk.Origins Archive web site. It was the site that was delisted. [talkorigins.org]

And indeed, as of right now (10:35 PM CST) a Google search for "talk.origins" doesn't show any links at all to the Talk.Origins Archive. In fact, the first link that comes up is to a young-Earth creationist site which claims to offer "intellectually honest responses to the claims of evolutionism's proponents, including--but not limited to--the 'Talk.Origins' newsgroup and the 'Talk.Origins Archive' website."

Conclusions about species competing in crowded niches are left as an exercise to the reader.

Re:huh?

evolutionism, homosexualism - they're all new, and evil ! Better stick to the good old isms such as religionism, wifebeatism and alcoholism.

Hmm

While, I have some sympathy for the guy, just because you think your an honest webmaster does not mean that Google should have to vet you and your content. They have a business to run too. At some point a webmaster has to put themselves in a position to recognize and address these sorts of problems BEFORE Google gets involved.

Google censoring Usenet? Not!

The writeup sucks. It implies that Google is censoring Usenet.

Backups?

You'd think they'd keep regular "Last Known Good" backups and just be able to do a simple diff between the current page & their backup.

Or even just MD5 sums of all their pages, once a day, with known updates marked as such.

There should be no reason anyone has to even contemplate manually digging through thousands of pages if they've prepared sufficiently beforehand.

Maybe they'll take some very simple & no-cost precautions now that they've been burned.

Re:Backups?

That presupposes the site was intelligently designed. Starting with that kind of assumptions is completely unscientific.

Whine, Whine, Whine

So many people refer to Google as if it were a human looking at web sites and giving it the big thumbs up or down. As part of the indexing if the spider finds "violations" such as presenting a different page to spiders than to humans, it risks being dropped from the index. To expect a human response to why each site triggered the de-indexing is not reasonable.

In the webmaster's whining about Google, he complains about the request to be re-indexed containing:

                    *I believe this site has violated Googles quality guidelines in the past.

                    * This site no longer violates Googles quality guidelines.

He thinks these are "an admission of guilt", but they dont' say "I violated" they say "the site violated". So, if the site were hacked and did violate their indexing policy, fix it, say you've fixed it and move on. How many hits has he had over the years that came directly from Google? And did they come from Google due to all those people choosing Google to search for his site or it's topics? But now he whines about being delisted for the time it takes him to fix a site he should have kept unhacked in the first place.

Re:Whine, Whine, Whine

Heresy! Google sees all, Google knows all! Google is a man, a spider, and the holy page in one!

Brin 3:14 "And Google so loved the internet, that he sent his only-born son Larry Page to it so that any who believe in him shall not perish but have ever-lasting life in the Googleplex."

So you see, there *is* a person, Larry Page, who is also the spider that indexes everything and is also the page that serves up results. Only through this holy trinity could results as good as Google's result, thus proving Google's divinity. If the almighty Google has delisted this sinner's page, then we should not be looking at it in the first place, yes? To go against the wishes of Google brings hellfire!

Same thing happened to the first wiki

The first wiki, c2.com, also has a similar problem. Google stopped indexing it (or at least listing it), and nobody is sure why. It may be a side-effect of anti-spam features that c2 added, but this is just speculation. Site custodians debated removing the anti-spam features because of this, but it has yet to be settled.

Synopsis

"Talk.Origins is an archive with thousands of pages exposing creationist pseudoscience"

This article is a submission containing a biased summary which has little to do with the actual topic, which is the enigmatic status of Google's search algorithms.

Re:Synopsis

I think that the person you are replying to is referring to the big bang. If you look far back enough into the history of the universe, you get to a point where everything began to exist. At the singularity of the big bang, we find that both time and space began. There is no "before" the big bang, as time did not exist. This is a central part of the cosmological argument for God's existence:
1. Everything that begins to exist has a cause
2. The universe began to exist
3. Therefore, the universe has a cause
What came before the big bang? That question is meaningless, as time did not exist. So you have a few options, only one of them feasible. The first is that the universe is infinitely old and had no beginning. Once a view of atheists, this is no longer scientifically plausible. The second answer is that the universe came into existence from nothing - absolutely nothing. The third, and most reasonable, is that something else caused the universe to be created. This cause must itself be timeless, and spaceless, as time and space began to exist with the big bang.

So the atheist must either claim the absurdity that the universe came from nothing, or he(/she) must acknowledge that there was something that created it. And that *something* is inaccessible from scientific analysis. It is not, however, too far from the reach of philosophy and logic. We can draw reasonable conclusions about this entity.

Re:Synopsis

What came before the big bang? That question is meaningless, as time did not exist. So you have a few options, only one of them feasible. The first is that the universe is infinitely old and had no beginning. Once a view of atheists, this is no longer scientifically plausible.

Sorry, you are incorrect. There wasn't any time before the big bang, thus nothing could have come before it as you yourself stated. Not even time, as the big bang was the beginning of time. However one alternative view is that the universe oscillates between a big bang and a big crunch, thus from a timeless point-of-view the universe would be infinite.

What I really find worrying (hello, 1500's are calling) is the method of reasoning by creationists, like yourself.
A: There was a big bang.
B: We currently don't know what was the cause of this.
C: There must be 'some higher being' that created the universe.

Now A and B do not lead to C, no matter how you reason. If you want to have a drop of credibility, you'll have to support your claims. However, you can not, thus your logic is flawed. What created the 'entity' you speak of? What came before it? Why did it create the universe? If you want to play the science game, you should be answering those questions. Science allows questions to be left open, but tries to answer as many as possible by using facts. Creationism is not, and is unlikely to ever be, scientific or logical. You are allowed to believe in the toothfairy for all I care, but unless you have evidence that a mystical entity is willing to pay for your teeth: keep your belief to yourself.

Re:Synopsis

To paraphrase your post: "science is a religion". I disagree, please correct me if I misunderstood.

Science is based on a single "article of faith", ie: I belive the real world exists as I and others who inhabit said "real world" collectively percieve it. I can only prove with certainty that I exist and furthermore can only prove it to myself. If I do not have faith in the real world then "others" must be a figment of my imagination, a troublesome state of affairs since the imaginary/real others will declare me a psychopath [google.com] and lock me up in a real/imaginary padded room.

Since I and "others" can observe and agree on things in and about the real world we can create testable theories that can be refined to better fit our observations and accurately predict outcomes. ie: We can practice the scientific method and refine our theories until we reach a (possibly non-existant) point where the only "assumption" is that the real world exists, or as I like to put it the Universe "just is".

So regarding a belief in evolution - The only faith required is the faith that the real world exists.

As for religion, it is based on blind faith, blind since I and "others" cannot percieve the same observations, these observations and associated theories fail the "real world" test because they cannot be demonstrated to "others" using their own perceptions. This does not mean religion is pointless or even psuedoscience, it simply means religion is not comprable to science (apples vs oranges). In my mind making such comparisons entirely misses the point of both endevours.

Psuedoscience, litteraly "fake science" is blind faith dressed in a lab coat. Sure creationism is a theory but it's NOT a scientific one, claiming otherwise is by definition, psudoscience.

Finally the lack of a strong scientific theory for the origin of life does not validate creationisim, nor does it invalidate the theory of evolution.

Bias: I suppose you could argue on some deep philosophical level that faith in the "real world" makes me biased toward...um...the real world, I can only wonder if that automatically means psychopath are unbiased? What does "science is a religion" prove? - I'm biased because science has a demonstratably superior track record of explaining and predicting the real world's behaviour whereas blind faith performs no better than random chance. Is that the kind of "bias" we are talking about here? - Because if it is, I am wondering how a non-phycopath can go to bed confident they will awake on the same planet the next morning?

Short cut to scientific enlightenment: Carl Sagan's book "A demon haunted world". It's also serves as an outstanding example of what a skeptic should be.

Understandably confused that some is not all

They may have to undertake an arduous process of vetting pages, essentially having to second-guess the mind of the cracker in trying to locate a problem that Google knows the exact location of.

Bzzt. The website admin needs to locate one or more problems (== however many the cracker planted), and Google knows the exact location of at least one. "one or more" >= "at least one". If google tells people where their problems are, google will be playing whack a mole for eternity. There are contractors/services that should be able to help them/anyone, google is not one of them.

Caped Hacker

This was quite obviously the work of the Flying Spaghetti Monster.

probably just bad algorithms

While it's natural to sympathise with the victimized website, it doesn't follow that Google is doing something Evil(TM) in this instance, rather it's most likely that their current algorithms are badly tuned.

With the index sizes that are being collected by search engines these days (on the order of 10 billion entries), it's completely naive to think that some humans are sitting at a terminal choosing to delist websites for some policy reason or other. It's also completely naive to think that a human email monkey can do any sort of digging to find out the exact reason that Google's automated algorithm has censored this particular site.

Instead, Google's engineers have automated algorithms which do all the censorship, and the policy is just there as a thin cover for whatever the algorithm happens to be doing today. It's worse of course, because 1) algorithms change every few months and 2) there's simply no comprehensive way to test the quality of the implementation.

Anyone who's programmed a nontrivial algorithm knows that obscure edge cases are a bitch, and with 10 billion websites, any algorithm will have plenty of obscure edge cases which nobody has ever tested, nor ever will. The most likely explanation is that the website in TFA is a false positive of some subsystem, but fixing it will require changes to the algorithms, and Google don't want to risk that, would you? The problem will probably go away in a few months when the algorithms are scheduled to be updated.

Re:probably just bad algorithms

While it's natural to sympathise with the victimized website, it doesn't follow that Google is doing something Evil(TM) in this instance

They really trademarked that? Shit, they don't really leave people much choice than being good these days.

Tell whom precisely?

Look for the webmaster's email address on the pages, assume it's [e-mail] or try [e-mail]? All these could bounce or simply disappear. Should Google put the effort into finding out who the webmaster really is?

The Truth

Let's face it. It's all a vast right wing conspiracy by fundamentalist Christians to remove any website that counters their beliefs... either that or it's simple Google policy and posting this story was a waste of everyone's time and only served to try and stir up debate.

Why should google have to tell him anything?

It's not like google is running his board or that he is paying google to debug his site.

I felt like making some silly comment about how lost the site is with out creationism and science.
Without either the site couldn't exist, or at least there would be nothing to "talk" about.

Some paradoxs:

We have science and use it to do things, to create things. Someday we may have the know how to create a galaxy and care for it lie a farmer does his crop. To inject life and help mold it to a conscious life from. But what would hat life form believe? Creationism, that they can never be such creator we have become? Maybe they would rather develope science and learn how to.

One cannot exist without the other.

Right to life vs. freedome of choice is another paradox exposed by a starving child who doesn't seem to have the freedome of choice to eat some food and there for will lose his right to life. Imagin all teh children that could have been feed, clothed, taught, given shelter and medicine with all the resources that have been wasted in the arguement....

People who something for themselves can sometime take advantage of something symbotic by falsely splitting it to create a public arguement that sells their book or presence, wins them an election, etc..
.

 

Google is acting like a woman

Or at least like a particular ex of mine... (I know -- I shouldn't generalize)

"If you don't know what you did wrong, I'm not going to tell you!"

cue jokes about /.ers and girlfriends...

Google Webmaster Tools

Google have a set of http://www.google.com/webmasters/tools/ [google.com] tools for webmasters. essencially it give out every diagnostic needed to fix your site for Google. Additionaly you have statistics for searches and how GoogleBot see your site. So, you shouldn't blame until you googled for the answer! Searching for "Google index tool" shows up "Google Webmaster Central"...

Re:Google Webmaster Tools

Google certainly has some useful tools, but when they don't work you are screwed. I have a site which I won't name which is not indexed by google and I have absolutely no idea why. I've submitted the url, built a sitemap using their own tools, validated it and even submitted the site for relisting. It still isn't there. What have I done wrong? The tools say everything is fine except it isn't. I could go to the web forum but other postings suggest the employees will likely just tell me wait for indexing. Except its not indexing me.

The sick thing is that I have Google Adwords on that site so each day that Google don't list me, THEY are losing money. I estimate I get 10x the click through business from MSN search than I do from Google. I'd probably make 3x the profit (as would Google) if they'd index.

Easy to check.

I recently had several sites that I host find their way into Google hell. I wrote a small 15 line Perl script (using File::Find) to hunt thru thousands of pages and return all url patterns.

Filtered out local domains and found easily found the problem with a run away forum that spammer zeroed in. Looks like they were also using a Perl script :{

They had uploaded over 3000 links - the bastards.

Ironical

A site gets delisted by google.
Then scores a slashdot homepage link.

People complain about this?

Really. Google has no obligation to anyone but its shareholders. You can start calling google evil when it pulls a microsoft and says "Redhat, install us as your default search engine and delete yahoo from your computers or we'll de-list your site".

Where to start looking

with no guidance as to what the problem was and nothing at all about where to start looking...
So you're a web master and you don't know how to check if the content you're mastering is OK. We clearly have to redefine the word master .

Talkorigins hacked by porn spammers

The site www.talkorigin.org is not the only site to have been de-indexed by Google.

This is a google cache of talkorgins.org [72.14.203.104] showing the porn spam links.

However, I checked on deepx.com [deepx.com] and it is *not* a porn site.

From DeepX.com's about page:

XML provides an open and flexible language for the creation, management and exchange of electronic content. Founded in 2000, deepX has an experienced team of consultants and developers, who specialise in the design and development of solutions using XML and the emerging technologies related to XML.

Also, another link shows www.theoi.com [theoi.com] and it is *not* a porn site, either:

Here's how THEOI used to look via the Wayback machine. [archive.org]

Theoi.com has been banned by Google (no reason given) and forced to close down as a result. There are no plans to re-establish this site in the future.

wu.edu.gh is Valley View University is a Seventh Day Adventist college in Ghana.

Both deepx.com and wu.edu.gh redirect to porn sites.

Unsurprisingly, wu.edu.gh, theoi.com and deepx.com have been de-indexed by google.

I speculate that all these sites that have been de-indexed were tagged by automated processes.

No Free Consulting

Basically, this "so called" webmaster wanted free consulting from Google. I don't think so. My personal response would have been, "I'll be happy to supply you with the information you request. It will, however, cost you my standard consulting rate of $xx/hour, two hour minimum."

Only friends and family get free computer help from me, but I'm rethinking that policy since I spent half a day cleaning the malware off my brother's computer during the last family holiday. He probably won't ask me to do it again, though. When he asked how his system got so infected, I answered (in front of the entire family), "You got infected from all those lesbian porn sites you've been visiting."

happens all the time

This is how google does business

I used to work on a site that had this happen. It ends up that past practices from the company led to the penalty and delisting. Unfortunately, google will not tell you exactly what you are doing wrong.

It pretty much led to the demise of the company. Sales plummeted so far that the investor pulled the plug. We did actually end up fixing the issue and relisted but the damage was done. (amongst other problems the company had...wasn't only google that did them in)

There really should be a tool provided by google that tests your site and tells you if and what it finds wrong. You would think this would be easy considering the code already exists.

Perhaps it could even just be a tool provided only to advertisers.

Welcome to the Real World

Have a peek over at the forums at WebmasterWorld, DigitalPoint, SearchEngineWatch, or any number of other webmaster related sites. This happens all the time. It is an issue that webmasters have had to deal with for some time now. Google at least provides some input for you if you can be bothered to register a sitemap with them.

Google has several billion pages in their index, and a significant portion of them are spam. Their business model relies on them having internal methods of dealing with web spam and it is not feasible or desirable for them to produce a list of violations to each and every person who runs afoul of their algorithms.

This is far from the most popular or important site this has happened to. Wordpress was delisted, as was BMW, Syndic8, and many others. This guy is using the controversial nature of his subject matter in an attempt to draw more attention. Get in line buddy, there is a long list of people whining all over the web about the same thing. Are you more important because the word Christianity is loosely affiliated with your site? Nope.

Do a little googling yourself and you can pretty easily figure out how to resolve the problem. It takes some time, and there are ways to accelerate the process. If you are that reliant on Google, it is time to start participating in some webmaster communities and figure out how to play ball with the Search Engines. Just like everybody else.

Google emailed this site

If you dig deeper, it turns out that Google emailed talkorigins.org to alert the site that it had been hacked and was stuffed with rape and animal porn spam. Google's head of webspam has posted a full write-up [mattcutts.com].

The only reason

The only reason this article appeared on the /. is because crusaders against creationism (rightfully) suffered (for the reasons unrelated to the content), and the viciously atheistic vigilante crowd of alarmists started to moan and moarn. The /. editors smelled the opportunity to add some hits (why? /. is very popular without this sensationalism).

evolution

So, if evolutionary theory is correct, it seems to have favoured a line of cry babies. There's evidence against, if ever there was any.

I suppose he could be a mutant....and his predecessors are all non-cry babies.

They Saw Some Spotted Somes Signs of Design...

in their website. But if you wait a million years, Google's algorithm could randomly change.

Insert creepy music here.

After seeing the headlines, I popped "talk.origins" into my Google search bar and got "trueorigin.org." Which claims to "Expose the myth of Evolution."

WHAT THE FUCK.

Who are Google's customers?

This whole article assumes that Google exists for the convenience of websites -- it doesn't. Google exists for the convenience of USERS who, as a group, view advertisers' paid links. As a user, I don't give a shit if Google notifies website owners about their reasons for delisting.

It isn't googles fault, false assumptions

The assumption that google "knows" where the problems are is based on a probably false assumption. Many times these algorithms use merely ranking and automated inspection.

Google has billions of pages indexed, there isn't enough time or manpower to have humans inspect pages. Even if they could elevate to a human, google could not possibly inspect for every domain.

The truth is that the webmaster let his site get hacked, Google delisted the site to protect the integrity of its product. It is the responsibility of the WEBMASTER to protect the integrity of their site. He may complain that he has to assume "guilt" (He doen't really, he merely has to affirm he has corrected the problems and believes that there are no more.) The problem is he IS guilty of being a bad admin that allowed someone to hack his site, and he wants to blaim google and not himself.

Is this suprising? News?

If Google revealed the core techniques used by its index to filter out hacked pages, then it would be effectively making those techniques useless. Its the same reason Microsoft and Firefox have not told the world how its anti-phishing features work, even if you have a domain that appears erroneously as phished. Its also the same reason that only dozens Google employees REALLY know how AdSense works. We know it depends on the context of the page, but clearly there is more to it (this confirmed by an Google employee friend of mine who is under NDA; could noy say more).

Its security through obscurity, sure... A valid form of security when the goal is to prolong discovery of methods until new ones are in place.

Google Secretly Creationist?

I hope this gets fixed soon. Because now when you search for talk.origins the first thing that pops up is true.origins. www.trueorigin.com which I won't link to. They are a bunch of willfully ignorant creationists. Sorry that was redundant.

Anyway, here's the proper link http://www.talkorigins.org/ [talkorigins.org] if you can't find it with google. I occassionally use other search engines like AllTheWeb [alltheweb.com] if I can't find what I'm looking for with google.

And there's nothing on the Talk.origins news page [talkorigins.org] about the delisting.

Nazis on Usenet

Has anyone called anyone a nazi yet?

http://en.wikipedia.org/wiki/Godwin's_law [wikipedia.org]

Re:Words are Meaningless

Nobody was evil here. The guy's site got hacked and spam links added, Google rightfully de-listed him, and then the webmaster found the problem, fixed it, and asked Google to re-list. Am I missing something?