Fake Scientific Paper Detector

moon_monkey writes "Ever wondered whether a scientific paper was actually written by a robot? A new program developed by researchers at Indiana University promises to tell you one way or the other. It was actually developed in response to a prank by MIT researchers who generated a paper from random bits of text and got it accepted for a conference."

Yes!

I am always wondering what those damn robots are up to!

That's good and all

but I wonder if it can tell if a paper was written by a million monkeys pounding on typewriters?

Re:That's good and all

Obligatory Simpson's Quote

Monkey's typing on a typewriter as Mr. Burn's is working on the next great american novel:

Burns: This is a thousand monkeys working at a thousand typewriters. Soon they'll have written the greatest novel known to man.
(monkey smoking cigar typing on a typewriter)
Burns: Lets see. It was the best of times, it was the BLURST of times! You stupid monkey! (Smacks monkey upside his head)

Re:That's good and all

Oh, I'm sure the work of monkeys is quite easily identifiable [vivaria.net].

Re:That's EASY!

I, for one, peruse the blogosphere. On my Powerbook, wearing a black turtle neck and beret. Stroking my goatee thoughtfully. Sipping a latté in a café

If I could just find a way to recharge my PowerBook from your hatred, I could stop carrying this ugly power adaptor.

Turing test?

so can a robot write a paper and then decide whether the paper was written by a robot (itself)?

Self defeating?

It seems like it wouldn't be too difficult to modify the MIT program to use this new anti-robot robot to write papers that this anti-robot robot would not be able to detect. Ideally, this would be done with a learning algorithm (so that it could easily be extended to other anti-robot robot programs), but reverse-engineering the anti-robot robot (by humans) should also provide a solution.

Now that Indiana U has thrown down the gauntlet, I wouldn't be surprised if MIT responds. Hopefully it will result in an even better paper-writing robot. Ideally, it will lead to dissertation-writing robots. :)

Re:Self defeating?

I recently had to check out an essay-grading robot for my Introduction to Natural Language Processing class.

I'd fed it the introduction of a randomly generated essay. It got a 4/5 on all counts.

I figure, if teachers are going to use robots to grade essays, we should use robots to create them in the first place.

Re:Self defeating?

Eventually my students won't have to write papers and I won't have to grade them! Think of the potential application of this technology towards education!

Discrimination

I hope the ACLU will ensure that discrimination against metal people will not be allowed to continue.

An interesting experiment

Has anybody fed Dvorak's latest column [slashdot.org] to this program? I've often wondered if he actually writes his columns, or just generate verbiage at random.

Re:An interesting experiment

"This text had been classified as
INAUTHENTIC
with a 24.9% chance of being authentic text"

No kidding.

Re:Typos

E-mail spambots have been making typos for years.

Sadly, It appears that I am a robot.

I've taken a long posting that I wrote on my blog and dropped it into the site. And I am Inauthentic. Now I understand the "Bladerunner Moment" comment in the article. I shall begin to surround myself with oddly colored polaroids and snapshots of theoretically implanted ancestors.

The nice thing is that we've finally settled the argument if machines can be made to drink beer and like it !

See what it says about slashdot

According the the program, the comments to this article are rated as follows:

This text had been classified as INAUTHENTIC with a 32.2% chance of being authentic text

Bearing in mind that text over 50% chance will be classified as authentic, this add credence to the theory that slashdot comments are generated by monkeys randomly typing on keyboards.

The program is a failure.

Apperantly I'm on average 49% artificial, based on school papers I wrote. I dub thee program: a failure.

Only works for scientific papers

If you try to use it on any human written NON scientific paper, such as Lincoln's gettyburg address, it almost always considers it false.

I suspect that it is looking for the conventional thinking with conventional word structure. As such, it is NOT a good idea i

Re:Only works for scientific papers

No, it doesn't even seem to work on scientific papers. I submitted four papers from the latest issue of Inorganic Chemistry and it thought 2 out of 4 were false:

Inauthentic: Assembly of a Heterobinuclear 2-D Network: A Rare Example of Endo- and Exocyclic Coordination of PdII/AgI in a Single Macrocycle.

Inauthentic: Pyrazolate-Bridging Dinucleating Ligands Containing Hydrogen-Bond Donors: Synthesis and Structure of Their Cobalt Analogues

Authentic: Manganese Complexes of 1,3,5-Triaza-7-phosphaadamantane (PTA): The First Nitrogen-Bound Transition-Metal Complex of PTA

Authentic: Structure, Luminescence, and Adsorption Properties of Two Chiral Microporous Metal-Organic Frameworks

Based on this (small) sampling, the program doesn't appear to do any better than if it were to guess randomly. I wonder if this thing is even supposed to work, or if it just returns a random result based on a hash of the paper or something?

I am in awe

So I go there, and I start shoving it text from my hard drive. I try:

A) Text of an article (Philosophy) I (native English speaker) wrote in Italian: 98.5 Authentic.
B) Text of an article I wrote in English (History): 87.8
C) Text of an article (History) written in French by a native French speaker and translated into English: 93.2
D) Critical edition of a 14th-century Latin text (Theology): 97.7 Authentic.
E) Documentation to a Field Artillery Simulation: 95.3
F) A completely bogus narrative for a monastic order that doesn't exist, written in a style that mimics A)-C): 16.8% Inauthentic

So in this case, we have a human written document that has superficial meaning, but is written as a "fake scientific paper", and registering as such.

And yes, I did read the "purpose" of the page; I know it's not supposed to detect it.


And yet it does, decisively.

False positives

Hmmm, it's an interesting idea, but it seems to give a lot of false positives. (So naturally, it will detect fake papers, if it thinks every paper is fake.)

First thing I tried was some pages on computational oncology website [uci.edu], in particular, my cancer primer [uci.edu], which I wrote in not a short time. Everything I fed was determined to be inauthentic. Perhaps I just write like a robot. :-) I figured that perhaps the detector was more primed for real papers, so I figured it wasn't too big of a deal.

So, next I tried my most recent research paper [sciencedirect.com], and it, too, was determined to be inauthentic, and in fact with less authenticity than my website. So much for the theory of being primed for scientific papers only. This thing is starting to look pretty bogus to me ... but an interesting idea, nonetheless. -- Paul

Trying Wikipedia articles

I've been trying my own papers and articles from Wikipedia. My own papers all score around 90%. Wikipedia articles that I consider good ones seem to score in the 80% range. Badly written fancruft scores very low.

Some variant on this thing might be useful as a new article filter in Wikipedia. We need more automation over there to stem the flow of incoming dreck.