Computer Program Makes Essay Grading Easier 666
phresno writes "c|Net is running a short article on Prof. Bent at the Columbia, Mo., University. The Prof. has developed a computer program which he now uses to grade his sociology students' essays. He claims the program can discern content, and argument flow within sentence and paragraph structure, and has saved him over two hundred hours of reading per semester. How long before he's replaced entirely by his own program to cut down on staff costs?"
Okay, so... (Score:1, Interesting)
WTF?! (Score:0, Interesting)
Neither than new, nor a silly idea (Score:3, Interesting)
It makes more sense than you'd think: it turns out that knowledgable essays in a particular domain cluster statistically in useful ways. Yes, it does mean that something like Molly Bloom's Soliloquy [answers.com] wouldn't necessarily score very well, but then if you didn't know it wsa a Nobel Prize winning classic, would you think it was well written?
Qualrus wasn't initially intended for essays (Score:2, Interesting)
Re:Not the world's best plan (Score:3, Interesting)
I fear that my experience was not unique. I wonder how well I would do if I turned in a chapter of Moby Dick or Les Miserables.... Or maybe a section of Physics and Phylosophy by Heisenberg.
Devil's Advocate (Score:1, Interesting)
Suppose you have a stack of papers to grade. Even a simple tool (not really a big step beyond spelling and grammar checkers) could help to triage the papers.
Then you can skim the bad papers looking for redeeming ideas, read the good papers in detail looking for reasons not to give top marks, and read the middle papers with an eye to both.
Of course once you've done this once for a given class, you can probably pre-bin them by who wrote them...
Re:Wouldn't want to be in his class. (Score:5, Interesting)
I go to the University of Missouri. I wouldn't want to be in his class either. Hmm, I don't have enough of the keywords in my paper? How about you actually read the paper you made me write?
a culty/Edward_Brent.html [missouri.edu]
http://sociology.missouri.edu/Faculty_and_Staff/F
Buffer overflow (Score:2, Interesting)
This reminds me... (Score:2, Interesting)
I later found out why from someone who had been taught by him before. He would take 100% - the percentage of passive sentences found by the Word program. So I intstantly started handing in garbage essays with 0% passive sentences.
If a professor can do that... (Score:2, Interesting)
A While (Score:4, Interesting)
I would say a long time. A program that tries to understand natural language requires some sort of "intelligence," a quality that humans definetly possess and computers, up to now, definetly do not.
AI still mostly consists of certain hacks to trick other people into thinking the programs are intelligent - basically attempting to fool the Turing Test. This can often produce great results and can be very useful, but almost never replaces a human in complex tasks (such as natural language processing).
The difficulty arises because humans cannot easily (or perhaps possibly) comprehend their own intelligence. It seems so natural to read a sentence and make sense of it, but when it comes time to program a computer to do it, most people try to emulate the behavior of their own comprehension. This may trick some people, but the simple nature of the programs cannot possibly be as powerful as an actual human.
The best solution, in my opinion, is a closer study of neuroscience and how it can be applied to silicon (or how new technologies need to arise to emulate the complex neural structure of the brain).
I know that people are starting to use computers to grade standardized essays, but there (currently) must always be a human checking the results because of the small number of unforseen cases that the hacked algorithms cannot do a good job. After all, the programs do not "understand" anything that is written. That is why I postulate it will be a long, long time before computers can truly emulate humans.
Fire the professor... (Score:4, Interesting)
If a professor does not care enough to read my papers, then to hell with him. There is more that a professor does than just check grammer, or look for passages that deals with the question and used terms from the book. The best professors I had were the ones who wrote all over the margins, sharing their thoughts about my ideas. Those are the ones who I would meet in their office to chat with. They are the ones who I went to for advice.
I had one teacher in english who graded the first paper, reading them all. She then never read another paper, only skimmed them. She pretty much gave out the same grade on all your papers you got on your first paper. I got an "A" on my paper, and another student got a "D". So I was working with the "D" student, and no matter what was done, the "D" grades went up to "C-" but stuck. So for the last paper, we switched our papers. Guess what? My paper was still an "A" even though it belonged to the other student, and the other paper was a "C". We went to the teacher to explain what we did, and rather than the professor owning up to what was done, we the students got blamed.
This really pisses me off. Professors get paid over $70,000 a year, some over $100,000 a year, they work 20 hours a week, and they have job security and a union. Then they want to slack off. Fucking asshats. Something like this makes me want to vote to remove public funding from schools, to always vote no whenever there is a refferendum to increase property tax. With those kinds of professors, people might as well get their education at the public library.
Shotgun... (Score:3, Interesting)
Re:Fire the professor... (Score:3, Interesting)
I noticed something like this, too, and what we did was myself and another student submitted the SAME paper. Not only did the proferssor not notice, he also gave me a C and the other guy an A. Obviously, we complained, but nothing ever happened.
Better teaching tool than grading tool (Score:4, Interesting)
You remember peer editing in 4th grade? Did that have any value? Not really - but if you got instant feedback on papers, that makes it easier to just write better in the first place.
Especially if this technology is combined with this technology [slashdot.org].
Re:Cheating (Score:2, Interesting)
And I'd be quite upset if I was paying $35k/yr for an education where the professors couldn't be bothered to grade my work themselves. Just because I need your accredited institution on my resume bad enough to pay the college tuition doesn't mean you have me by the shorthairs and shouldn't provide me with your full skill, experience, knowledge and attention.
Have they replaced the Creative Writing courses with a copy of the early 1990s Word Perfect grammar checker, too?
Re:Absolutely nothing (Score:1, Interesting)
As for the field of Sociology, it really says nothing of importance, but perhaps it says something about writing an argumentative text?
That's exactly what I was thinking (Score:3, Interesting)
Sure, the program can analyze that the sentence flow and structure looks like it's analyzing/arguing/explaining/whatever a point. But is it even arguing the right point? Does that paper even _have_ a point at all, or is it just a babbledygook of random nouns/verbs/adjectives/etc that fits a structure?
I'm not even sure it has to end up "beautifully expressed babble", it can just be any collection of random words that fits the structure the program is looking for. I.e., I'm sure it can be _awful_ babble and still pass.
And indeed, a student then doesn't even have to understand how the program works or anything. A script will do just fine. Just download a paper that got good grades (hence, fits the idiot's program) and run it through a script that replaces each word with a random word that's in the same category. (Transient/intransient verb, noun, etc.)
E.g., take the following two sentences:
"The Electronic Frontier Foundation announced its prestigious Pioneer Awards today, and one of the three lucky winners for 2005 is Mitch Kapor."
"A Pink Smell Stew impaled its dormant Turkey Shaddow tomorrow, or five of the two spotted continuums towards 2005 equals Mitch Kapor."
It's just the random word substition I was talking about. It isn't even beautifully phrased babble, it's just awful even by dadaist criteria. Yet it has the exact same sentence flow and structure. Can the program even tell that the second is random blabber?
Re:term papers... (Score:5, Interesting)
I really, really, really wish someone told me this. I lost a letter grade in a class because I had a differing point of view from the professor. I wrote a kick ass term paper, I spent countless hours in the library doing research, I had other people proof read my paper. It was one of the best papers I wrote. But it was the exact opposite of what the professor believed.
We expect our teachers will grade us on our work. But every now and then we get a professor who probably spends too much time writing letters to the editorial section of the new york times.
Re:Cheating (Score:3, Interesting)
Ah, and while we're at it, use if for the moderation system as well. After all, its objective is to give grades, and moderations are a sort of grades.
Of course, those Insightful moderations might get a bit unreliable
SCNR
Re:Calculus (Score:3, Interesting)
Hmmm.
1.0 falls in the range [0.95,1.04].
1.00 falls in the range [0.995,1.004].
Worst cases:
0.95 + 0.995 = 1.945, rounded off to 1 decimal makes 1.9.
1.04 + 1.004 = 2.044, rounded off to 1 decimal makes 2.0.
Isn't the number of significant digits after the decimal point indeed zero?
Re:Fire the professor... (Score:2, Interesting)
I don't know of that professor and grad student, but Alan Sokal [nyu.edu] did something like this a few years ago, and published a paper on the influence of quantum mechanics on something related to sociology in one of the most respected sociology journals (of course the article was 100% pure bullshit, just using the right terms). Turns out they didn't much appreciate it.
Coming from a more hard-science background, I tend to dismiss social sciences as not being real science, but mostly a stupid rewriting of simple facts. While I know my view is quite extreme, this guy just confirms it: (from TFA) "In sociology, we want them to learn the terms,"
If they grade their students based on their ability to spurt some inintelligible mumbo-jumbo, they're a bounch of clowns. And if his software does really go beyond that, maybe he chose the wrong career option and he should be doing some CS work on Natural Language...
Venting frustration. (Score:2, Interesting)
I have a teacher this semester who doesn't teach sigfigs but expects you to get it right. Then he applies rules like "but with this info, you always show 3 sigfigs, with this info, 2 unless blah blah blah blah blah"... and he says this after marking everyone down in the test, mind you. Good thing I had a High School teacher who taught me this stuff.
His excuse? "You should have asked more questions in class". I swear, he must have expected us to actually ask "how do sigfigs work?" without knowing it would be an issue. Doesn't help that he makes you feel stupid when you ask a question in class.
He's not all bad, but still pretty bad. For more information: click [ratemyprofessors.com].
Re:term papers... (Score:2, Interesting)
Re:term papers... (Score:4, Interesting)
You CAN disagree with the prof, but you'd better be prepared to go into bat.
L
Re:Cheating (Score:3, Interesting)
That sounds like a good course for anyone to take. How to write a paper effectively. There's definitely a science to creating great works. Not necessarily the creative content but there are archetypes for a reason.
This will be a great language study to see if culture has a deciding factor in natural responses to how information is presented. For example I developed a great fascination for space and physics reading wikipedia articles where the way the content is presented and the information delivered has made it easier to understand and build upon. I was not surprised to find that most of the articles I read were created and maintained by just a few individuals.
Automated Learning Concept (Score:1, Interesting)
This could simulate the feeling good writers have when they know the passage they are writing will need to be rewritten, give them a nice feeling about a well-written sentence etc.
Soon they'll try to avoid the bad feelings and try to reflect this in their writing. The same patterns the professor's program uses will show up in the brain of the student.
I think I should finish this comment up with a paranoid pop-culture reference, but I can't think of one that fits well enough (just imagine something).
Re:Hmmm (Score:2, Interesting)
So then, you are given the option to go back and redo the questions. This is necessary because you have to enter answers such as (cos(x^2))^(1/3), hence most people who know the right answer don't enter it properly on the first try; some even go nuts trying to figure out how to properly enter an answer they know is right. But bottom line is, if you have the time to spare, you can get perfect on every homework - just try again if you're wrong!
Re:Fire the professor... (Score:3, Interesting)
A Physicist Experiments With Cultural Studies [nyu.edu]
Physics hoaxers discover Quantum Bogosity [theregister.co.uk]
It's about following instructions. (Score:5, Interesting)
When I was in college in the late '80's, the "trick" to getting good grades really was to understand what the professors were looking for, and give it to them.
For example, I had a professor who distributed pre-printed pieces of paper that had a line drawn around it indicating the margin (something like two inches at the top, a half inch on the right and bottom, and three inches on the left. The large margins on the left and top were for the prof's notes and comments to us.) We had to type (with a typewriter) our papers to fit within the bounds of the margins, and spelling and grammar counted (and this was a Psychology class, not an English class.) He would not accept more than the one page. Another professor required papers be written in a specific topical order. If you deviated from those models, you got marked down.
The point was not that the profs were trying to be devious, but to make us take into account the instructions they gave us. If we followed the instructions, we got better grades. If we didn't, we would get marked down. And yes, content did count too.
Yes, it was a hassle, but the result is that now, when my bossed give me instructions, I follow them. The times when I deviate are the times when I really hear about it. Lesson learned!
Re:Cheating (Score:2, Interesting)
I doubt you're allowed to keep submitting the same essay until you're happy with your mark, so no, it is not effectively a filter for better papers unless, as the GP said, you get a copy of the program unconnected to a school system where the mark you get is applied to your academic record.
Re:Cheating (Score:5, Interesting)
If done successfully, you might get a good grade in sociology for something that would deserve a good grade in computer science
Re:Better teaching tool than grading tool (Score:5, Interesting)
the red squigle under the mispelled word
Back when I was grading papers, I used to recommend the exact opposite to the students -- turn off the "instant" spell-checker, then run the "full" spell-checker and re-read the paper. I found that, in many cases, students would correct anything that had a red squiggle underneath it, but would get a false sense of security that all of the errors had been detected by the word processor.
Oddly enough, when they had a squiggle-free page before them, they were much more attentive to detail and caught the "spelled-correctly-but-used-inappropriately" words.
well, actually... (Score:2, Interesting)
Re:Fire the professor... (Score:3, Interesting)
With all due respect, I disagree. While in theory there are a multitude of choices, in reality it is quite different. There are scheduling considerations: i.e. if I'm trying to keep one day free of class for work purposes, or am only taking night classes due to work, or if at a school with a large campus and there is not sufficient time to transport between class sites, or desired classes/sections are offered at the same time, or only during one semester of the year, or the course is a pre-req for several other courses and I'd like to graduate 'on time', etc.
When you get into the higher level courses, from my experience at a large state school, you just don't have a choice. Class xxx is offered in the Spring only, at such and such time, taught by so and so, and that's it. You either take it then, or take it at another school.