Computer Program Makes Essay Grading Easier

phresno writes "c|Net is running a short article on Prof. Bent at the Columbia, Mo., University. The Prof. has developed a computer program which he now uses to grade his sociology students' essays. He claims the program can discern content, and argument flow within sentence and paragraph structure, and has saved him over two hundred hours of reading per semester. How long before he's replaced entirely by his own program to cut down on staff costs?"

Re:Cheating

[subrosas]

Qualrus doesn't operate using a set grading criteria, but trains based on the users' grading markups. Therefore, you'd need the teacher's copy (complete with its "learned" patterns) to fool the system. Actually Ed Brent encourages the students to use Qualrus to write rough drafts, as it gives instant feedback - arguably a better learning technique from a usability standpoint (faster feedback == more retention).

i don't see why his studenrs would be doing better

[wintermute1000]

I was required to take a lot of writing classes for my college (and still haven't finished them all) and I've observed the quality of my writing go up appreciably since I began school. However, the reason I've become a better writer is because my essay graders write copious comments about where I'm going wrong in my papers and what I should do to improve - and they read the next paper I write for the class with those things in mind, and tell me whether I've improved sicne the last one.

The article didn't say anything about what kind of feedback the program provides, but I can't imagine it's anywhere near as helpful as the paragraph-long evaluations of my logic, style, and structure, which I got back with every paper I ever turned in, and I'd be impressed but surprised if his program took each student's previous weaknesses into account in the course of the evaluation. In writing, practicing can only do so much - the real help is in constructive feedback, and I just can't imagine where these students are getting it if not from the human graders of their papers.

GMAT exam has been doing this for awhile now...

[Golgafrinchan]

The company who administers the GMAT (Graduate Management Admission Test) has been using a computer grader for the analytical writing portion of the exam for several years now. They call it the e-rater. Both a human and the e-rater grade every essay.

According to ETS [ets.org], the e-rater agrees with the human grader 98% of the time.

The program DOES NOT grade the essays!

[rainwalker]

From TFA, which apparently no one has read yet:
"The final papers, which he does read, are usually much better as a result of Qualrus, too."

There you go! For the reading and comprehension impaired, here's a summary of what's actually happening, which even the reporter didn't get:
1. Students write a draft of their essay, which they then upload via a Web form to this program
2. The program gives them a score on various parts of their essay, giving them valuable feedback on what needs to be improved.
3. Students improve the pieces of their essay that the program suggests.
4. Students submit the final draft to the professor, who reads and grades each one by hand. Due to steps 1-3, the quality of the final draft is much higher.

This sounds like a great thing to me. Wish I had something similar for my students. I don't have the time to read through dozens of drafts for every student. Too bad I'm not in sociology.

More Feedback

[subrosas]

Actually Ed Brent encourages his students to use Qualrus on earlier drafts of the papers. This provides immediate, extensive feedback. And by "extensive" I mean more detailed and descriptive comments than those that a single teacher/TA could supply for each and every paper in a large lecture. The immediacy of this feeback is what is really important - immediacy is KEY to learning.

Essay Generator

[daveb]

Actually that's already been done [elsewhere.org]. Quite effectivly too.

Re:The program DOES NOT grade the essays!

[VeryProfessional]

From TFA:

The computer-generated scores count for about a third to a quarter of students' final grade for Brent's class.

There you go! Make sure you RTFA very carefully before accusing others of being reading and comprehension impaired.

Re:Cheating

[Anonymous Coward]

This is similar to what happened in the essays in the Civil Service examinations in Ancient China. All essays needed to follow an exacting formulaic structure and as a result the system, though complex, was easy to manipulate if one knew the proper structure. This can be a problem since unique thought and creativity are not given a great value.

Re:Taxpayer ripoff?

[mbrother]

Hmm...the NSF grant provisions I recall seeing the last time I submitted one discouraged, or even ruled out, supporting commerical activities. This does sound a bit funny to me, too.

Re:term papers...

[Anonymous Coward]

I wrote a midterm and a final paper for an anthropology class in college. The Prof talked about how he wanted us to do in-depth analysis and blah blah blah. For the midterm, I focused on a a few details and examined them closely, but did not cover all of the facts that were relevent, and got a C. For the final, I spent an approximately equal amount of time and basically rephrased the first sentance out of every paragraph in the relevent book chapters and got a near perfect grade.

An important lesson to learn freshman year: the TA grades the papers, not the prof. (in this rare case, the prof might have actually graded differently -- he was pretty good).

Re:Fire the professor...

[paragon_au]

My father is a professor at the Australian National University (Physics Dept).
He gets paid US$60k a year, works 8 hours a day at work.
Then comes home and spends his evenings on his laptop working for another 1 to 3 hours. And then on weekends spends another 3+ hours a day working.
None of which he gets paid extra for, as he is on a fixed salary.

Don't taint all professors with one what professor did.

Re:Fire the professor...

[call -151]

Professors get paid over $70,000 a year, some over $100,000 a year, they work 20 hours a week, and they have job security and a union.

Well, there are some professors that meet that description, but at a reasonable university, those tend to be in the minority. At a reasonable university, most faculty work more like 60-80 hours a week, particularly if they are active in research. I certainly have pulled many more all-nighters as a professor than I did as a student and I pulled a lot of them as a student. A few things that students tend to overlook:

• Usually, students have a choice about professors and courses and in my experience, don't sufficiently take advantage of that choice. A reasonable strategy is to "shop around" and visit multiple sections of a course, and choose a professor who seems engaging and valuable. If there is no such professor, it may make more sense to concentrate on something else and to wait for that course in another term. It may make the first two weeks of the term very busy, going to lots of extra classes, but it can be an excellent investment.
  
• Short term concerns about which professor is the easiest are often overvalued compared to which professors do a better job getting their students to understand. Don't complain about how lousy your professors are if you are always taking the easiest route. Consider the source when taking recommendations about which professor is "good"- if it's from a student who doesn't wan't classwork to make a dent in social activities, keep that in mind.
  
• You may have to strategize to get the really good professors. Research superstars do not always make great teachers, but often they can do a great job conveying the important notions, and research superstars tend to have reduced teaching loads. If you are choosing instructors who are teaching four or five classes a term, they are more likely to be less engaged in research and perhaps more likely to be overwhelmed by or disengaged from their increased teaching obligation as well.
  
• Pick your university wisely, if you have a choice. One of the key variables that many people underestimate is how important strong classmates are. You can have the best professor in the world, with one strong student and nineteen weaker students (poorly prepared, missing prerequisites, distracted by other attractions, unwilling to work hard...) and the class may end up being not so useful for the strong student, simply because the choice is to have 19 people are lost and one person understanding, or 19 people kind of understanding and one person who is bored. At universities where teaching evaluations matter (most places they matter at least somewhat), the choice for the professor in that case is usually to reduce expectations and try and make the class valuable for most people, even if the class will end up being not so useful for the strong student.
  

There are terrible professors and great professors at every university- the fractions may change from place to place, but with some seeking out and strategy, usually it's possible to do well.

Re:Hmmm

[gryphon_church]

I taught first year eng lit to a class of 30. I read each paper 3 times. 1. To order the papers from bad to very bad. 2. Read, marked and made comments. 3. Read again to make sure the marking spectrum was fair (worse papers got worse marks ect.)

Re:Hmmm

[Anonymous Coward]

I'm an English teaching assistant, and I can tell you that every TA and professor I know reads each paper carefully. One of the main purposes of handing back students' essays, especially in a first-year English course, is to give them personalized tips on how to improve their writing (argumentation, flow, grammar, etc.). It takes care to give each student useful comments.

If we end up sorely overworked it can be hard to maintain that standard of marking, but even then we're certainly not just skimming for keywords.