Computer Program Makes Essay Grading Easier 666
phresno writes "c|Net is running a short article on Prof. Bent at the Columbia, Mo., University. The Prof. has developed a computer program which he now uses to grade his sociology students' essays. He claims the program can discern content, and argument flow within sentence and paragraph structure, and has saved him over two hundred hours of reading per semester. How long before he's replaced entirely by his own program to cut down on staff costs?"
Cheating (Score:4, Insightful)
Hmmm (Score:5, Insightful)
Angst
Structure (Score:5, Insightful)
So it measures structure and argument.
How's it going to measure creativity of thought? Are we going to just pump out logic machines from colleges?
Not the world's best plan (Score:5, Insightful)
Intresting (Score:5, Insightful)
Undergrad is usually a waste of teaching resources (Score:2, Insightful)
So more power to him. He is unlikely to be getting anything better or more insightful than a parroting of what he has already delivered in his monologues to his class. Same papers, year in and year out. No big deal to grade these kids with an automated program.
Wouldn't want to be in his class. (Score:2, Insightful)
This is rubbish (Score:1, Insightful)
If a program is really good enough to mark an essay, then reversing the function should allow it to create an essay that is perceived by human assessors to be of good quality. And I suspect we are a long was away from that.
Has this guy even assessed the correlation between the marks the program gives and the marks he would give?
Re:Hmmm (Score:2, Insightful)
And talent may remain unfound (Score:5, Insightful)
But how would hidden talent and creativity be found? How will the teacher know if his students are actually trying hard to write their papers when all he does is check the thing with a computer program?
It's a really terrible idea and I think it's really cheezy. Ohh, he saved some time. So does that mean he now gets paid less? Does this automation get the students a discount? Yea, right.
If I'm going to put a lot of work into writing an interesting paper about something, I want someone to read it.
Re:Hmmm (Score:5, Insightful)
Re:Cheating (Score:5, Insightful)
Re:Cheating (Score:5, Insightful)
Re:Intresting (Score:5, Insightful)
More importantly... (Score:5, Insightful)
"the Columbia, Mo., University" ? (Score:3, Insightful)
Gee - you'd think the submitter could RTFA...
Developed with NSF money... (Score:1, Insightful)
Re:Cheating (Score:1, Insightful)
/ can't remember the name.
Cheating? Teaching! (Score:5, Insightful)
Would be great for high-school students. Have students write an essay or paper and analyze it right in front of them. Then the program highlights their errors (or what the program perceives as an error). Even better, complaining students would help fix bugs in the software because they know their intent - they could send off a highlighted error-ridden version to the developers with an explanation of why they think they are right.
Better yet, give it to everyone! It's not like you can cheat, you still have to rewrite and resubmit your papers. Shit, I say build it into text boxes on slashdot and wikipedia to start!
please do not hold this post to the standard of the Qualrus [qualrus.com] (real page of the software)
Regurgitation (Score:3, Insightful)
Which is to say, this prof is asking students to regurgitate data. Given, a certain level of base knowledge is necessary in any class and topic, and regurgitation (aka parroting) is an easy way to check that base knowledge. If a paper is assigned on a particular topic that they've been studying, then this sort of program can easily check for base level ability to spit back key words and phrases.
But, I seriously doubt that the class is ONLY about that base knowledge -- or that the program can reasonably check for anything more. I've had classes where the prof or graders did basically the same thing that this program does (i.e. check only for key words, phrases, and patterns they want to see), and I have little respect for those profs.
If you don't want to put even a basic amount of effort into checking a paper, don't assign it -- find some better way to check students' progress.
Just as bad as plagarism (Score:4, Insightful)
This one is just nuts. Why on earth am I writing essays which are going to be marked automatically by a machine? It's bad enough that scantron cards have found their way into subjects where they're totally irrelevant (a multiple-choice test for a university level Shakespeare course?), this is just another reason why post-secondary education has become increasingly less complete.
If he's allowed to use a machine to save him the effort of reading an essay, I should be able to use a machine so I don't have to go through the effort of writing one. Trust me, as arduous as it is to read a 20 page essay on the relative merits of liquid rubber concrete compound fasteners, writing it takes a lot more effort, a lot more time, and it damn well deserves to be read by the professor who assigned it.
An easier solution (Score:0, Insightful)
Re:Intresting (Score:2, Insightful)
Still, it seems like something that could be at least partially reverse-engineered.
The one concern I'd have about the program in general is that it would teach people to write in a very particular way to make the program happy. When I used to run all my papers through MS-Word's grammar check it really hammered certain rules into my head. If the program continually makes the same kinds of mistakes it could hammer mistakes in there (In MS Word 2000's spellchecker, "ridiculous" was misspelled "rediculous". That made me look dumb quite often).
As an aside, there was a story recently that where there was some discussion about the difficulty of automatic grammar checking (I think the story was just some guy bitching about the current state of grammar checking programs). It would seem that this would be just as complicated if not more so (analyzing flow), and also solving a very similar problem. It would be interesting to see what some people that really wanted to attack grammar checking could do with the source code to this program (even if it could not be open-sourced because of cheating paranoia, there would probably be some kind of option to license the technology, at the right price)
Re:Not the world's best plan (Score:4, Insightful)
It's possible to train computer programs to translate text between languages by feeding examples of good and bad translations to pattern-recognition algorithms, which start with simple rules. Most of these models are similar to neural-net machines, which is in turn based on the fundamental theory of how animal brains (including human brains) operate. You don't design and code an algorithm, you train the machine by example, with some human-assisted trail-and-error.
This often works because that's how human judgement works: we learn just about everything by example and trial-and-error, and we're VERY good at it (look at what millions of years of evolution can accomplish!). This isn't to say that a trained neural net machine is "intelligent" or "conscious", just that solves problems by the same mechanism that a human brain does, albeit in a much more limited fashion.
Of course, the effectiveness of a trained machine is limited by how big a computer you have, and how well you train it. Re-creating the complexity of the human brain in software with present-day techniques and equipment would be impossible (neural net software is VERY memory intensive when it gets complex). This may change in the future, but that's another debate that I won't get into.
I'm not saying that this professor's software actually works or not--he could easily be full of shit. I'm also not saying that you can't game one of these machines the same way spammers game Bayesian anti-spam filters: use trial-and-error to figure out how to trick the machine consistently.
In fact, I'm assuming that a canny student could steal the software and do exactly that. After all, the human brain is a much more powerful learning machine than the program, and could probably outsmart it in the same way that people can outsmart rats.
But then again, this is a socialogy course, so his students probably won't think of it on their own.
Re:Structure (Score:5, Insightful)
Perversely, the worse a paper is, the more time it receives; it's more important, and more difficult, to motivate a failing grade than a good one. Also, to some extent all good papers (or assignments) are alike and can be spotted fairly easily; it's the bad ones that are (regrettably) unique and need individual attention.
Re:Cheating (Score:3, Insightful)
From snopes.com [snopes.com]
Re:Hmmm (Score:5, Insightful)
What's the point of the class? In Calculus 1, the point is to learn concepts and methods that allow you to perform basic operations, as proven by your ability to work out problems on homework and tests. You're not asked to be creative or anything--that comes later, in 300 or 400 level classes or graduate work. First, you have to learn the basics.
I imagine sociology isn't that much different--at least, it wasn't in Poli Sci when I was in college. First, you have to learn a bunch of basic facts and rules and concepts, and demonstrate that you have a know them. You should be able to talk about them, define them, and answer questions about them. Anybody who's being creative in a freshman sociology class is ahead of the game.
And don't give me no shit about "I spent hours making it, you should spend hours reading it". That's like the
I mean, shit--it took me DAYS to write my first couple of C programs in CS 101. Does that mean that the professor is shorting my education if he takes 10 seconds to grade it?
Re:And talent may remain unfound (Score:3, Insightful)
Re:Wouldn't want to be in his class. (Score:2, Insightful)
The program saves him over 200 hours of DOING HIS FUCKING JOB. If he didnt want to read papers -- hes in the wrong line of work. End of story, finish him.
The other half of this story is -- universities make students write TONS of worthless papers. Students leave a program having written scores of papers -- and they never learn HOW to write papers. I am amazed at how many college graduates I meet who do not understand formal writing.
Re:Cheating (Score:5, Insightful)
The real question in this scenario is whether or not they will learn enough by cheating to have gained something valuable.
If they have to write a program to beat the teacher's program, are the students not learning something very valuable (at least in the marketable business skills department)?
Also, in order to write a program that creates essays that conform to the teachers program, will it not also be necessary to learn the grammar and logic rules the teacher considers to be important and even ponder those rules for extended periods of time?
It seems to me that the cheater (or at least the first cheater) will do more work than the professor did and thereby become quite familiar with English grammar, organization of arguments, and computer programming. All of these are useful skills.
how would he like a taste of his own medicine? (Score:2, Insightful)
Re:Not the world's best plan (Score:3, Insightful)
Why not? What universal principal or physical law states this?
I probably wouldn't disagree with you if you said that the current state of technology can't emulate humans here ... But to say it will never happen, no matter what?
I think John von Neumann once said --- "If you can tell me exactly what it is that a machine cannot do, then I will build a machine to do exactly that!".
Re:More importantly... (Score:2, Insightful)
A physics professor goes to the provost of the university and says that he needs 2 million dollars to build a particle accelerator. The provost shakes his head and says "You physics profs and your damn requests for money. Why can't you be more like the math department, they only need pencils, paper and trashcans. Or better yet, why not like the sociologists, all they need is paper and pencil."
From what I notice, most of the time a great vocabulary is an adequate substitution for original thought. I have bullshitted through my share of courses with interesting turns of phrases but little to no creativity. Works with most of the liberal arts degree programs. So insofar as that generalization is true, the grading is nearly automatic anyway.
--Joey
Fire this professor... (Score:3, Insightful)
This professor should be fired.
Re:Structure (Score:5, Insightful)
Re:Intresting (Score:3, Insightful)
Re:Structure (Score:4, Insightful)
He's a sociology professor, and he's managed to write a natural language parser that can actually decipher meaning, and mark the relevance of the content of the essay against the question.
Re:Hmmm (Score:1, Insightful)
I'm all for computers being used to catch plagiarism, but if you aren't reading the actual content turned in, you aren't doing your job.
Re:Cheating (Score:4, Insightful)
If they have to write a program to beat the teacher's program, are the students not learning something very valuable (at least in the marketable business skills department)?
No. Only the person who writes the cheating program does.
See, that's the beauty of the internet, my friend. I don't even have to know how to program to beat CSS encryption on DVDs. I merely have to download said program from someone else (maybe even the only person in the entire world) who does know how.
Also, in order to write a program that creates essays that conform to the teachers program, will it not also be necessary to learn the grammar and logic rules the teacher considers to be important and even ponder those rules for extended periods of time?
My previous point aside, why do you assume that the class is on English? Why should a history teacher give a damn about your understanding of English? Even IF it was an English class... all of my college-level English classes haven't even touched on grammar or syntax. Those things are assumed.
Moreover, beyond looking for keywords, how does this program actually prove that the student knew what he or she was talking about? I think we have all come across beautifully expressed babble. What prevents a student (or a script?) from doing the same? Lastly, how can this guy claim victory while at the same time admitting that he never read the papers? How has he proven the program was functioning as intended?
For all of those seriously interested in this program: I've got a anti-baboon charm here to sell to you. Does it work, you ask? Well you don't see any baboons do you!?!
-Grym
So what? (Score:5, Insightful)
If I can write a program to automate a menial task so I don't have to do it, then by all means, I should do it. If grading undergrad papers is a menial task that can be automated, then it should be automated.
I mean, just because a freshman writes a bad paper doesn't mean a professor has to actually read it.
term papers... (Score:4, Insightful)
For a term paper, you additionally have to use correct grammer and spelling. Also, do not try to argue something stupid. Don't take a contrary opinion to the professor or to popular opinion on the college campus. You won't be able to convince the grader, and they'll think that if your argument isn't convincing, then it must be flawed and you deserve a bad or mediocre grade.
These are things that I wish someone had told me when I was an undergrad.
Yeah, I'm outraged, but here's a solution: (Score:3, Insightful)
Wait - what the fuck am I saying? ALL college students should be graded by the non-computer criteria I just listed, and those who can't do (or at least attempt) that kind of work shouldn't be in college.
Re:Fire the professor... (Score:5, Insightful)
I used to give short answer/essay questions to my astronomy students the first couple of semesters I taught the big non-major course. It took a tremendous amount of time to grade which was one reason I stopped, but not the primary reason. I'm a novelist, and I know how to write, and there was a consistently high fraction of exams written so badly it was very painful to read. Perhaps I should have kept at it, with the idea that it's good for the students. But a few essays in a science class won't dent the problem that starts in k-12 education.
"written using a word processor worth 50 points"!? (Score:1, Insightful)
"There will be one paper based on a combination of Internet and library resources that must be written using a word processor worth 50 points."
If this is any indicator of his understanding of grammatical rules, I pity his students.
Question: How many points is your word processor worth?
Re:Fire the professor... (Score:3, Insightful)
I mean what's the point? if the paper doesn't really help to demonstrate your mastery of the subject, and it's not going to be marked properly anyway, why waste everyone's time.
Why not get the students to mark each others papers, for the papers that don't count anyway. And only mark a small sample, and then mark the final paper properly.
Re:Fire the professor... (Score:3, Insightful)
That is an interesting idea. I had one unorthodox professor who did something like that. We had a term paper, but we had to work in groups of 4, and the group submitted the paper and everyone got the same grade. It cut down the 30 papers he would have had to under 8. And it forces the students to talk about what they wanted to include in the paper, how valid points of view were. In essance, we were teaching each other.
But I have never been one who liked the idea of having my grade tied into the work of other people. I asked the teacher about that, saying "it is unfair for you to give me a grade for what other people do, I want to be judged based on my work". His responce was "in the real world, the sucess of your business depends on how well your group works as a team, so consider this a heads up".
In _sociology_ class (Score:3, Insightful)
Having sociology grades that reflect purely English grammar skills, is as sick a joke as grading someone's data structures course based purely on indentation. It misses the whole point and makes a mockery of the whole teaching process.
Re:term papers... (Score:3, Insightful)
Re:Intresting (Score:3, Insightful)
Re:Calculus (Score:3, Insightful)
These days, there's no reason not to just type up your math if you can't write legibly or draw a decent curve on a graph. In some cases, it was actually fun asking a student "what is that letter"?
The perfect evidence that education... (Score:3, Insightful)
I'd be interested to put the Gettysburg address, MLKs "I Have A Dream" speech, major works of literature, etc through Mr. Brents meat grinder and see what grade they get. The whole thing reminds me of that scene in Dead Poets Society where they try to measure the "greatness" of a poem using trumped up terms like "importance" and "perfection". Sprinkle in a little computer wizardry, and suddenly you've got a mysterious, unbending, rule machine.
Frankly this kind of thing just disgusts me. I'm no romantic, but you can't analyse how good a paper is based on some algorithm. It's like the idiots who try to analyse a songs potential through computer analysis.
Re:Fire the professor... (Score:5, Insightful)
One of my teachers at school tried that. Unfortunately what happened is that in each group the rest of the members (the groups were arranged so each group was mixed ability) ganged up on the 'smart but weak/shy' one to do all the work then goofed off.
Actually, now I come to think about it, that's exactly what happens in the workplace.
Stephen
For cargo-cult pseudo-sciences only. (Score:4, Insightful)
Yes, it might well be a better learning technique from a usability standpoint
Cargo cults a la Feynman are all about form, and this tool can indeed detect the presence of form and even distinguish form that is considered "good" by some metric from form that is considered "bad".
But unless it actually understands what is being written through deep semantic analysis performed against a thorough database of relation-interlinked concepts, then there is no way the tool can detect hard scientific content (to the small extent that it occurs in sociology) from gibberish that just obeys the right forms.
As Brent himself says, "In sociology, we want them to learn the terms." And that pretty much sums it up.
To Teach (Score:0, Insightful)
Re:term papers... (Score:5, Insightful)
So RTFA already? (Score:3, Insightful)
But if you actually RTFA you'll see that we're talking about a _sociology_ professor and _sociology_ papers.
"If it's an English class, that's the whole point."
Well, precisely. That's the whole point: it's not an English class.
He's grading a _science_ class based on form instead of content. _That's_ the problem.
Unlike the English class in your example (which you are right about) his job _is_ to read and judge the content.
Re:Fire the professor... (Score:4, Insightful)
With regards to students' written work: My field is meteorology. I too used to give students in my survey-level meteorology class opportunities to "express themselves" via short answers (a paragraph or two) on exams. I stopped because it was so hard to grade many of them because they were written so poorly. In addition to that, it is very difficult to grade short answers in a consistent way. For many of the short-answer questions I would usually end up just writing a number down ("Hmm.. this feels like a 3-points-out-of-5 answer") which real doesn't feel right... but what do you do when the concepts are confused, spelling and grammar are terrible but they have expressed some knowledge of the material?
I have talked with professors who have been doing this stuff for a much longer time than I (some of whom are into the latest trends in teaching etc.) and many of them are gravitating towards all objective tests (multiple choice and true false) for their survey level classes (and some upper level). A well-written objective test should adequately test a student's knowledge of the material in a fair way, especially in the sciences where there truly are right and wrong answers. Still, I don't like giving these kinds of tests - it just doesn't feel right - but like grading the others even less.
In my upper level classes all of my testing is subjective, and I do assign papers such as case studies where a storm system is described and analyzed. Some of my seniors can write well, most of them are so-so and a few are truly terrible. I tell them up front that spelling, grammar, style etc. counts on these assignments, and I find that if you tell students that these things are part of their grade they will put in an effort to write well.
I suppose I could just "blame the high schools" but I think the problem is deeper than that. In the US grade inflation is a huge problem in many universities and at the college level, student evaluations of faculty are often very highly regarded (and if you are evaluated poorly it can keep you from getting tenured or promoted). So a logical response is for faculty to go easy on students, rightly assuming that this will return higher evaluations. I don't know if that is a part of the writing problem, but I know an A today isn't an A 20 years ago at many universities.
Re:Cheating? Teaching! (Score:4, Insightful)
That would be counterproductive. If the program actually works with even 70% reliability, I'll eat my hat. In other words, I guarantee it's worse than the average student. Natural language processing is AI-complete. Every six months somebody claims to have solved the problem, and it always turns out to be another Eliza ("Did you come to me because the fact that question that concerns you is the real reason?") or babelfish ("To celebrate the score and seven years, our suffered ancestors brought ahead on this continent a new nation, taken in freedom and devoted to the proposal which all gecreeerde people are equal") or, frequently, even worse.
"I have a computer program that understands English sentences" is roughly the same as "I have some really great real estate a quarter-mile north of downtown Chicago that will fetch a fortune on the market, but because I'm in a hurry I'll let you have it for half price."
Re:Cheating (Score:4, Insightful)
Is it cheating to run your HTML/XML through a validator?
Is it cheating to test-compile your scripts with "perl -c"?
Is it cheating to run a hardware diagnostic to check for faults?
Humans are tool-users, technologists doubly so. This is how civilization advances: by developing processes to eliminate typical sources of error, allowing man to apply his thinking mind to higher-level problems.
Re:term papers... (Score:2, Insightful)
For the price of classes? Definately. You're not paying to change the mind of your instructor or impress anyone. This sadly, is what it is reduced to.
Re:term papers... (Score:2, Insightful)
BTW, is there any good reason you chose to turn the discussion into a US vs. Country B affair?
Comment removed (Score:3, Insightful)
Crappy profs? (Score:4, Insightful)
I had a professor that said he would give his opinion but he would grade based on how well the argument was crafted and backed up by facts/references.
ANother professor complained that too many papers where just repeating what he had said in class and he had marked those down for not having enough original thought.
Grammar and spellling are alway important..
Re:Cheating (Score:2, Insightful)
When I was a student, I believe the goal was learning. The end product was the education, not the grade. While the grade is intended as a measure of how well that goal was accomplished it is not, in itself, the goal of the exercise.
So, yes, using this product repeatedly to improve a single score is cheating by effectively allowing a student to "redo" the assignment multiple times. It provides an unfair advantage. Although I could see that there is potential value in having special assignments now and again where students are allowed to do just that simply for the benefit of learning how to improve their own papers.
Re:term papers... (Score:3, Insightful)
Apparently, they don't teach respect in your non-American "world-class university". The United States have (yes, I still hold on to the older plural use of United States) literally *thousands* of colleges and universities. And, like everywhere else in the world, they range from world-class to the terrible. I've personally attended schools at several different levels: community college, state university and private university. They were all within the same state and varied greatly.
They also must not teach drawing clear analogies either as your parallel needs work. If you had said "I was at a world-class university *in Europe*", your pointing to "America" as your contrast might have made college-educated sense. However, you didn't. Rather, you revealed your open hostility and likely ignorance by saying that your one, anecdotal instance of a non-American university supplies enough evidence to damn the whole American post-secondary education system.
Perhaps your world-class university wasn't.