Follow Slashdot stories on Twitter

 



Forgot your password?
typodupeerror
Education Software

Computer Program Makes Essay Grading Easier 666

Posted by CowboyNeal
from the scantron-grows-stronger dept.
phresno writes "c|Net is running a short article on Prof. Bent at the Columbia, Mo., University. The Prof. has developed a computer program which he now uses to grade his sociology students' essays. He claims the program can discern content, and argument flow within sentence and paragraph structure, and has saved him over two hundred hours of reading per semester. How long before he's replaced entirely by his own program to cut down on staff costs?"
This discussion has been archived. No new comments can be posted.

Computer Program Makes Essay Grading Easier

Comments Filter:
  • Cheating (Score:4, Insightful)

    by fembots (753724) on Thursday April 07, 2005 @11:36PM (#12172989) Homepage
    How long until some students get hold of this program and tweak their essay until it's puifect? It's similar to spammers using spam filters to test their emails first.
    • Re:Cheating (Score:5, Insightful)

      by Anonymous Coward on Thursday April 07, 2005 @11:45PM (#12173051)
      If you'd read the fucking article, you'd have known that the students get instact feedback on their scores, so it effectively is a filter for better papers.
      • Re:Cheating (Score:4, Insightful)

        by mzieg (317686) <mark@zieg.com> on Friday April 08, 2005 @07:34AM (#12174843) Homepage
        It's no more cheating than spell-checkers and calculators.

        Is it cheating to run your HTML/XML through a validator?
        Is it cheating to test-compile your scripts with "perl -c"?
        Is it cheating to run a hardware diagnostic to check for faults?

        Humans are tool-users, technologists doubly so. This is how civilization advances: by developing processes to eliminate typical sources of error, allowing man to apply his thinking mind to higher-level problems.

    • Re:Cheating (Score:5, Insightful)

      by RazorX90 (700941) on Thursday April 07, 2005 @11:46PM (#12173058)
      God, imagine what test prep courses like Princeton Review are going to do. Essay writing for the SAT will turn into a new branch of science. They'll teach you exactly what your ratio of compound sentences, to complex sentences, to clauses, to action verbs, etc. should be.
      • Shotgun... (Score:3, Interesting)

        by John Seminal (698722)
        That reminds me of a startegy some people had when I was an undergrad. Since they knew teachers only skimmed the writings, they used a shotgun approach. They threw everything in the essay including the kitchen sink. They figure somewhere in there, the terms the professor was skimming for would be included. Sadly, these were the "A" students. The "B" students, who tried to write a real paper and make a point, did not get any worthwhile feedback.
        • term papers... (Score:4, Insightful)

          by Cryptnotic (154382) * on Friday April 08, 2005 @02:03AM (#12173690) Homepage
          Unless it's an English class, they're just looking to see if you've learned the material for that course. Especially for the essay questions in the midterm and final, they will always just skim the papers for key words, phrases, and dates. They underline them and add up the points. That's just the way it works. The "B" student, who writes something interesting, but doesn't cover all of the relevant material, loses points.

          For a term paper, you additionally have to use correct grammer and spelling. Also, do not try to argue something stupid. Don't take a contrary opinion to the professor or to popular opinion on the college campus. You won't be able to convince the grader, and they'll think that if your argument isn't convincing, then it must be flawed and you deserve a bad or mediocre grade.

          These are things that I wish someone had told me when I was an undergrad.
          • Re:term papers... (Score:5, Interesting)

            by John Seminal (698722) on Friday April 08, 2005 @02:36AM (#12173792) Journal
            Also, do not try to argue something stupid. Don't take a contrary opinion to the professor or to popular opinion on the college campus

            I really, really, really wish someone told me this. I lost a letter grade in a class because I had a differing point of view from the professor. I wrote a kick ass term paper, I spent countless hours in the library doing research, I had other people proof read my paper. It was one of the best papers I wrote. But it was the exact opposite of what the professor believed.

            We expect our teachers will grade us on our work. But every now and then we get a professor who probably spends too much time writing letters to the editorial section of the new york times.

            • Re:term papers... (Score:3, Insightful)

              by orangesquid (79734)
              My mom was an A student. One professor refused to agree with my mom and gave her an A- even though my mother did well in that class, too. Differing opinions out to be REWARDED--they show students who have formed their own opinions (generally).
            • Re:term papers... (Score:4, Interesting)

              by Louis Guerin (728805) <guerin AT gmx DOT net> on Friday April 08, 2005 @04:28AM (#12174169)
              My wife got a grad-level essay back with B on it, knowing that she'd gone against the opinion of the HOD and course convener, who'd marked it. No surprise. She submitted it for remarking (where it gets sent to another faculty member at a sister university) and it just came back with an A on it - overall gain about 15 percentage points.

              You CAN disagree with the prof, but you'd better be prepared to go into bat.

              L
            • Re:term papers... (Score:3, Insightful)

              by hahiss (696716)
              I'm sorry to hear about your experience, and it is a shame.

              However, an alternate explanation is that the professor, who is, after all, the expert in your classroom, just thought the paper wasn't as ``kick ass" as you, a student with 15 weeks worth of experience in the field, did.

              It is interesting that students believe their opinions on the subject matter of a course to be as insightful, important, and correct as someone who has been working on the subject for many years. It sounds to me like the professo
          • by jbarr (2233) on Friday April 08, 2005 @07:31AM (#12174826) Homepage
            For a term paper, you additionally have to use correct grammer and spelling. Also, do not try to argue something stupid. Don't take a contrary opinion to the professor or to popular opinion on the college campus. You won't be able to convince the grader, and they'll think that if your argument isn't convincing, then it must be flawed and you deserve a bad or mediocre grade.
            You make some interesting points.

            When I was in college in the late '80's, the "trick" to getting good grades really was to understand what the professors were looking for, and give it to them.

            For example, I had a professor who distributed pre-printed pieces of paper that had a line drawn around it indicating the margin (something like two inches at the top, a half inch on the right and bottom, and three inches on the left. The large margins on the left and top were for the prof's notes and comments to us.) We had to type (with a typewriter) our papers to fit within the bounds of the margins, and spelling and grammar counted (and this was a Psychology class, not an English class.) He would not accept more than the one page. Another professor required papers be written in a specific topical order. If you deviated from those models, you got marked down.

            The point was not that the profs were trying to be devious, but to make us take into account the instructions they gave us. If we followed the instructions, we got better grades. If we didn't, we would get marked down. And yes, content did count too.

            Yes, it was a hassle, but the result is that now, when my bossed give me instructions, I follow them. The times when I deviate are the times when I really hear about it. Lesson learned!
          • Crappy profs? (Score:4, Insightful)

            by acomj (20611) on Friday April 08, 2005 @08:00AM (#12174990) Homepage
            If the profs at your school mark grades down because they have a different opinion you should change schools. No offense.

            I had a professor that said he would give his opinion but he would grade based on how well the argument was crafted and backed up by facts/references.

            ANother professor complained that too many papers where just repeating what he had said in class and he had marked those down for not having enough original thought.

            Grammar and spellling are alway important..
        • by Headcase88 (828620) on Friday April 08, 2005 @03:36AM (#12174003) Journal
          Some profs at least claim to be a little more discerning when a page is longer than recommended. Everyone probably has at least one prof with the story (whether fact or fiction) of a student that handed in 3 pages of worthwhile material with multiple page data from a semi-related source sandwiched in, and how perople like that get a lower mark. Maybe I should make a long post and see what happens [homestarrunner.com].

          First, moving around quickly, and with purpose, is a true sign of character. Secondarily, bustle(e.g. hustle) yields more product for the working types. "Hustle and bustle are like my right and left arms," said Li'l Spicy in his famous "Hustle and Bustle Are Like My Right and Left Arms" speech. Webster's defines bustle as "excited and often noisy activity; a stir." A stir, indeed. Finally, sometimes gross stuff can be funny.

          Here are some links:
          It is now my intention to play video games for several hours.

          Sources:
          The Brothers Chaps (2004).Homestar Runner. Retrieved April 8, 2005 from www.homestarrunner.com

          Random Source (2005). that you won't read because you were too lazy. Retrieved April 8, 2005 from www.toreadthisfar.com

          (I have four words for this post: "Too much half-asleep effort")
      • Re:Cheating (Score:3, Interesting)

        by Queer Boy (451309) *

        They'll teach you exactly what your ratio of compound sentences, to complex sentences, to clauses, to action verbs, etc. should be.

        That sounds like a good course for anyone to take. How to write a paper effectively. There's definitely a science to creating great works. Not necessarily the creative content but there are archetypes for a reason.

        This will be a great language study to see if culture has a deciding factor in natural responses to how information is presented. For example I developed a gre

    • Re:Cheating (Score:5, Informative)

      by subrosas (752277) on Thursday April 07, 2005 @11:47PM (#12173063)
      Qualrus doesn't operate using a set grading criteria, but trains based on the users' grading markups. Therefore, you'd need the teacher's copy (complete with its "learned" patterns) to fool the system. Actually Ed Brent encourages the students to use Qualrus to write rough drafts, as it gives instant feedback - arguably a better learning technique from a usability standpoint (faster feedback == more retention).
      • by raehl (609729) <raehl311@yaho[ ]om ['o.c' in gap]> on Friday April 08, 2005 @01:48AM (#12173634) Homepage
        My spelling has improved massively with the advent of the red squigle under the mispelled word - and not just because I fix the error, but I now just don't make the errors in the first place. (The green squiggle is not so useful - sometimes it's just quite wrong.)

        You remember peer editing in 4th grade? Did that have any value? Not really - but if you got instant feedback on papers, that makes it easier to just write better in the first place.

        Especially if this technology is combined with this technology [slashdot.org].

        • by mopslik (688435) on Friday April 08, 2005 @09:14AM (#12175642)

          the red squigle under the mispelled word

          Back when I was grading papers, I used to recommend the exact opposite to the students -- turn off the "instant" spell-checker, then run the "full" spell-checker and re-read the paper. I found that, in many cases, students would correct anything that had a red squiggle underneath it, but would get a false sense of security that all of the errors had been detected by the word processor.

          Oddly enough, when they had a squiggle-free page before them, they were much more attentive to detail and caught the "spelled-correctly-but-used-inappropriately" words.

      • Re:Cheating (Score:3, Interesting)

        by maxwell demon (590494)
        What about using it for slashdot? Train it on slashdot stories and replace the editors. Done right, this should not only reduce spelling errors, but also misleading front page content. Also, if the idea is that the teacher doesn't have to read the essay himself, it must have built-in dupe detection (because that's the most conventional form of cheating, after all). Therefore this way there wouldn't even be dupes on slashdot!

        Ah, and while we're at it, use if for the moderation system as well. After all, it
      • by Morgaine (4316) on Friday April 08, 2005 @05:40AM (#12174404)
        arguably a better learning technique from a usability standpoint

        Yes, it might well be a better learning technique from a usability standpoint ... but only for a content-free "discipline" like sociology or other pseudo-sciences.

        Cargo cults a la Feynman are all about form, and this tool can indeed detect the presence of form and even distinguish form that is considered "good" by some metric from form that is considered "bad".

        But unless it actually understands what is being written through deep semantic analysis performed against a thorough database of relation-interlinked concepts, then there is no way the tool can detect hard scientific content (to the small extent that it occurs in sociology) from gibberish that just obeys the right forms.

        As Brent himself says, "In sociology, we want them to learn the terms." And that pretty much sums it up.
    • by ImaLamer (260199) <john@lamar.gmail@com> on Thursday April 07, 2005 @11:49PM (#12173085) Homepage Journal
      Why not give the program to the English department and use it for teaching?

      Would be great for high-school students. Have students write an essay or paper and analyze it right in front of them. Then the program highlights their errors (or what the program perceives as an error). Even better, complaining students would help fix bugs in the software because they know their intent - they could send off a highlighted error-ridden version to the developers with an explanation of why they think they are right.

      Better yet, give it to everyone! It's not like you can cheat, you still have to rewrite and resubmit your papers. Shit, I say build it into text boxes on slashdot and wikipedia to start!

      please do not hold this post to the standard of the Qualrus [qualrus.com] (real page of the software)
      • by jonadab (583620) on Friday April 08, 2005 @07:25AM (#12174801) Homepage Journal
        > Why not give the program to the English department and use it for teaching?

        That would be counterproductive. If the program actually works with even 70% reliability, I'll eat my hat. In other words, I guarantee it's worse than the average student. Natural language processing is AI-complete. Every six months somebody claims to have solved the problem, and it always turns out to be another Eliza ("Did you come to me because the fact that question that concerns you is the real reason?") or babelfish ("To celebrate the score and seven years, our suffered ancestors brought ahead on this continent a new nation, taken in freedom and devoted to the proposal which all gecreeerde people are equal") or, frequently, even worse.

        "I have a computer program that understands English sentences" is roughly the same as "I have some really great real estate a quarter-mile north of downtown Chicago that will fetch a fortune on the market, but because I'm in a hurry I'll let you have it for half price."
    • Re:Cheating (Score:5, Insightful)

      by LuYu (519260) on Friday April 08, 2005 @12:16AM (#12173271) Homepage Journal

      The real question in this scenario is whether or not they will learn enough by cheating to have gained something valuable.

      If they have to write a program to beat the teacher's program, are the students not learning something very valuable (at least in the marketable business skills department)?

      Also, in order to write a program that creates essays that conform to the teachers program, will it not also be necessary to learn the grammar and logic rules the teacher considers to be important and even ponder those rules for extended periods of time?

      It seems to me that the cheater (or at least the first cheater) will do more work than the professor did and thereby become quite familiar with English grammar, organization of arguments, and computer programming. All of these are useful skills.

      • Re:Cheating (Score:4, Insightful)

        by Grym (725290) on Friday April 08, 2005 @01:56AM (#12173661)

        If they have to write a program to beat the teacher's program, are the students not learning something very valuable (at least in the marketable business skills department)?

        No. Only the person who writes the cheating program does.

        See, that's the beauty of the internet, my friend. I don't even have to know how to program to beat CSS encryption on DVDs. I merely have to download said program from someone else (maybe even the only person in the entire world) who does know how.

        Also, in order to write a program that creates essays that conform to the teachers program, will it not also be necessary to learn the grammar and logic rules the teacher considers to be important and even ponder those rules for extended periods of time?

        My previous point aside, why do you assume that the class is on English? Why should a history teacher give a damn about your understanding of English? Even IF it was an English class... all of my college-level English classes haven't even touched on grammar or syntax. Those things are assumed.

        Moreover, beyond looking for keywords, how does this program actually prove that the student knew what he or she was talking about? I think we have all come across beautifully expressed babble. What prevents a student (or a script?) from doing the same? Lastly, how can this guy claim victory while at the same time admitting that he never read the papers? How has he proven the program was functioning as intended?

        For all of those seriously interested in this program: I've got a anti-baboon charm here to sell to you. Does it work, you ask? Well you don't see any baboons do you!?!

        -Grym

        • That's the first thing that came to mind as I was reading the summary: well, gee, so he can now grade purely on form, rather than content.

          Sure, the program can analyze that the sentence flow and structure looks like it's analyzing/arguing/explaining/whatever a point. But is it even arguing the right point? Does that paper even _have_ a point at all, or is it just a babbledygook of random nouns/verbs/adjectives/etc that fits a structure?

          I'm not even sure it has to end up "beautifully expressed babble", it
      • by Moraelin (679338)
        Dunno about you, but I'd expect someone's sociology grades to actually reflect some understanding of sociology. Same as I'd expect that their math grades reflect having learned some maths, or that the grades they get in their Java course reflect _some_ knowledge of Java.

        Having sociology grades that reflect purely English grammar skills, is as sick a joke as grading someone's data structures course based purely on indentation. It misses the whole point and makes a mockery of the whole teaching process.
    • by John Seminal (698722) on Friday April 08, 2005 @01:12AM (#12173499) Journal
      This reminds me of something that was in the papers a few weeks ago. A professor and graduate students wanted to show that most journals will publish anything if it sounds "academic" enough. So they wrote a paper that was hog-wash, made no point, was just a bunch of academic sounding prose. And guess what? They got published.

      If a professor does not care enough to read my papers, then to hell with him. There is more that a professor does than just check grammer, or look for passages that deals with the question and used terms from the book. The best professors I had were the ones who wrote all over the margins, sharing their thoughts about my ideas. Those are the ones who I would meet in their office to chat with. They are the ones who I went to for advice.

      I had one teacher in english who graded the first paper, reading them all. She then never read another paper, only skimmed them. She pretty much gave out the same grade on all your papers you got on your first paper. I got an "A" on my paper, and another student got a "D". So I was working with the "D" student, and no matter what was done, the "D" grades went up to "C-" but stuck. So for the last paper, we switched our papers. Guess what? My paper was still an "A" even though it belonged to the other student, and the other paper was a "C". We went to the teacher to explain what we did, and rather than the professor owning up to what was done, we the students got blamed.

      This really pisses me off. Professors get paid over $70,000 a year, some over $100,000 a year, they work 20 hours a week, and they have job security and a union. Then they want to slack off. Fucking asshats. Something like this makes me want to vote to remove public funding from schools, to always vote no whenever there is a refferendum to increase property tax. With those kinds of professors, people might as well get their education at the public library.

      • She pretty much gave out the same grade on all your papers you got on your first paper.

        I noticed something like this, too, and what we did was myself and another student submitted the SAME paper. Not only did the proferssor not notice, he also gave me a C and the other guy an A. Obviously, we complained, but nothing ever happened.

      • by mbrother (739193) * <mbrother.uwyo@edu> on Friday April 08, 2005 @02:14AM (#12173729) Homepage
        I'd be pissed off, too, but don't let one bad professor sour you on all of us. My base pay is less than $60k a year, but with summer salary from grants I'm in your range. And I work way more than 20 hours a week. Way, way more. But what is this "union" you speak of? I know of no such union (although tenure can provide significant job security).

        I used to give short answer/essay questions to my astronomy students the first couple of semesters I taught the big non-major course. It took a tremendous amount of time to grade which was one reason I stopped, but not the primary reason. I'm a novelist, and I know how to write, and there was a consistently high fraction of exams written so badly it was very painful to read. Perhaps I should have kept at it, with the idea that it's good for the students. But a few essays in a science class won't dent the problem that starts in k-12 education.
        • by Orp (6583) on Friday April 08, 2005 @07:17AM (#12174769) Homepage
          Same here. Base pay here is not much more than 50k but I'm not complaining. I work 40+ hours a week too. Most nontenured faculty do, as do most tenured who still want to get promoted or who actually want to stay active in the field. I know of a few faculty who just show up and teach but they are a year or two from retirement.

          With regards to students' written work: My field is meteorology. I too used to give students in my survey-level meteorology class opportunities to "express themselves" via short answers (a paragraph or two) on exams. I stopped because it was so hard to grade many of them because they were written so poorly. In addition to that, it is very difficult to grade short answers in a consistent way. For many of the short-answer questions I would usually end up just writing a number down ("Hmm.. this feels like a 3-points-out-of-5 answer") which real doesn't feel right... but what do you do when the concepts are confused, spelling and grammar are terrible but they have expressed some knowledge of the material?

          I have talked with professors who have been doing this stuff for a much longer time than I (some of whom are into the latest trends in teaching etc.) and many of them are gravitating towards all objective tests (multiple choice and true false) for their survey level classes (and some upper level). A well-written objective test should adequately test a student's knowledge of the material in a fair way, especially in the sciences where there truly are right and wrong answers. Still, I don't like giving these kinds of tests - it just doesn't feel right - but like grading the others even less.

          In my upper level classes all of my testing is subjective, and I do assign papers such as case studies where a storm system is described and analyzed. Some of my seniors can write well, most of them are so-so and a few are truly terrible. I tell them up front that spelling, grammar, style etc. counts on these assignments, and I find that if you tell students that these things are part of their grade they will put in an effort to write well.

          I suppose I could just "blame the high schools" but I think the problem is deeper than that. In the US grade inflation is a huge problem in many universities and at the college level, student evaluations of faculty are often very highly regarded (and if you are evaluated poorly it can keep you from getting tenured or promoted). So a logical response is for faculty to go easy on students, rightly assuming that this will return higher evaluations. I don't know if that is a part of the writing problem, but I know an A today isn't an A 20 years ago at many universities.
      • If you're not going to mark the paper. DONT SET THE ASSIGNMENT...
        I mean what's the point? if the paper doesn't really help to demonstrate your mastery of the subject, and it's not going to be marked properly anyway, why waste everyone's time.
        Why not get the students to mark each others papers, for the papers that don't count anyway. And only mark a small sample, and then mark the final paper properly.
        • you're not going to mark the paper. DONT SET THE ASSIGNMENT... I mean what's the point? if the paper doesn't really help to demonstrate your mastery of the subject, and it's not going to be marked properly anyway, why waste everyone's time. Why not get the students to mark each others papers, for the papers that don't count anyway. And only mark a small sample, and then mark the final paper properly.

          That is an interesting idea. I had one unorthodox professor who did something like that. We had a term pap

      • My father is a professor at the Australian National University (Physics Dept).
        He gets paid US$60k a year, works 8 hours a day at work.
        Then comes home and spends his evenings on his laptop working for another 1 to 3 hours. And then on weekends spends another 3+ hours a day working.
        None of which he gets paid extra for, as he is on a fixed salary.

        Don't taint all professors with one what professor did.
      • by call -151 (230520) on Friday April 08, 2005 @05:23AM (#12174356) Homepage
        Professors get paid over $70,000 a year, some over $100,000 a year, they work 20 hours a week, and they have job security and a union.

        Well, there are some professors that meet that description, but at a reasonable university, those tend to be in the minority. At a reasonable university, most faculty work more like 60-80 hours a week, particularly if they are active in research. I certainly have pulled many more all-nighters as a professor than I did as a student and I pulled a lot of them as a student. A few things that students tend to overlook:
        • Usually, students have a choice about professors and courses and in my experience, don't sufficiently take advantage of that choice. A reasonable strategy is to "shop around" and visit multiple sections of a course, and choose a professor who seems engaging and valuable. If there is no such professor, it may make more sense to concentrate on something else and to wait for that course in another term. It may make the first two weeks of the term very busy, going to lots of extra classes, but it can be an excellent investment.
        • Short term concerns about which professor is the easiest are often overvalued compared to which professors do a better job getting their students to understand. Don't complain about how lousy your professors are if you are always taking the easiest route. Consider the source when taking recommendations about which professor is "good"- if it's from a student who doesn't wan't classwork to make a dent in social activities, keep that in mind.
        • You may have to strategize to get the really good professors. Research superstars do not always make great teachers, but often they can do a great job conveying the important notions, and research superstars tend to have reduced teaching loads. If you are choosing instructors who are teaching four or five classes a term, they are more likely to be less engaged in research and perhaps more likely to be overwhelmed by or disengaged from their increased teaching obligation as well.
        • Pick your university wisely, if you have a choice. One of the key variables that many people underestimate is how important strong classmates are. You can have the best professor in the world, with one strong student and nineteen weaker students (poorly prepared, missing prerequisites, distracted by other attractions, unwilling to work hard...) and the class may end up being not so useful for the strong student, simply because the choice is to have 19 people are lost and one person understanding, or 19 people kind of understanding and one person who is bored. At universities where teaching evaluations matter (most places they matter at least somewhat), the choice for the professor in that case is usually to reduce expectations and try and make the class valuable for most people, even if the class will end up being not so useful for the strong student.

        There are terrible professors and great professors at every university- the fractions may change from place to place, but with some seeking out and strategy, usually it's possible to do well.
        • Usually, students have a choice about professors and courses and in my experience, don't sufficiently take advantage of that choice.

          With all due respect, I disagree. While in theory there are a multitude of choices, in reality it is quite different. There are scheduling considerations: i.e. if I'm trying to keep one day free of class for work purposes, or am only taking night classes due to work, or if at a school with a large campus and there is not sufficient time to transport between class sites, or de
      • by mikael (484)
        There have been several experiments with fake papers:

        A Physicist Experiments With Cultural Studies [nyu.edu]

        Physics hoaxers discover Quantum Bogosity [theregister.co.uk]
    • Re:Cheating (Score:4, Funny)

      by Cryptnotic (154382) * on Friday April 08, 2005 @02:14AM (#12173727) Homepage
      79% gave up on 1st day in iCLOD city. Can you survive there? [iclod.com]

      I spent 3 minutes reading about it and I gave up.
  • Hmmm (Score:5, Insightful)

    by Anonymous Coward on Thursday April 07, 2005 @11:37PM (#12172995)
    I'm not so sure I like the idea of a computer grading my work.. I spent hours making it, but the guy doesn't even give it the time of day....
    Angst
    • Re:Hmmm (Score:2, Insightful)

      by Embedded2004 (789698)
      Indeed, when I pay good money for an education I expect I will at least be taught, not given an automated spit out mark. I find the majority of learning done by doing reports is the comments you get back from the prof.
    • Re:Hmmm (Score:5, Insightful)

      by CSMastermind (847625) <freight_train10@hotmail.com> on Thursday April 07, 2005 @11:44PM (#12173049)
      What makes you think that your teachers read the papers to began with. At least smaller papers, I know that all they do is skim for keywords.
    • Re:Hmmm (Score:5, Insightful)

      by MoralHazard (447833) on Friday April 08, 2005 @12:11AM (#12173240)
      Hey, if it was an undergrad math class, like Calculus 1, how would you feel about automated grading? It's possible, trivial even, to write a program and have a way for students to solve problems (showing their work, of course!) in a machine-readable format. Then you can have a computer grade the homework and report a summary to the TA, who looks at what students are missing and deals with it. This isn't that different from the way things currently operate, because the TA manually grading homework is just using a mechanical process to check the students' work, anyway.

      What's the point of the class? In Calculus 1, the point is to learn concepts and methods that allow you to perform basic operations, as proven by your ability to work out problems on homework and tests. You're not asked to be creative or anything--that comes later, in 300 or 400 level classes or graduate work. First, you have to learn the basics.

      I imagine sociology isn't that much different--at least, it wasn't in Poli Sci when I was in college. First, you have to learn a bunch of basic facts and rules and concepts, and demonstrate that you have a know them. You should be able to talk about them, define them, and answer questions about them. Anybody who's being creative in a freshman sociology class is ahead of the game.

      And don't give me no shit about "I spent hours making it, you should spend hours reading it". That's like the .sig that says "I don't use comments--if it was hard to code, it should be hard to read." The fact that an 18-year old punk takes hours to craft an essay (that the professor could do in his sleep) doesn't make the effort more valuable.

      I mean, shit--it took me DAYS to write my first couple of C programs in CS 101. Does that mean that the professor is shorting my education if he takes 10 seconds to grade it?
      • Calculus (Score:5, Funny)

        by John Seminal (698722) on Friday April 08, 2005 @01:16AM (#12173513) Journal
        I can say this much, I have never had any other professor, outside of the Chem or Physics department, grade my papers like a math professor. Most of the humanities professors just skim over. But in my Calculus class, it was possible to turn in homework and get negative points. For example, you have a problem 1.0 + 1.00 = ?. You write 2. First, half a point off for not figuring in significant digits. Another half a point off for sloppy handwriting. And the full point off for not showing your work. Problem worth one point, your score is negative one point. In some cases, it was better to not turn in anything at all.
        • Re:Calculus (Score:3, Interesting)

          by Flyboy Connor (741764)
          1.0 + 1.00 = ?. You write 2. First, half a point off for not figuring in significant digits.

          Hmmm.

          1.0 falls in the range [0.95,1.04].

          1.00 falls in the range [0.995,1.004].

          Worst cases:

          0.95 + 0.995 = 1.945, rounded off to 1 decimal makes 1.9.

          1.04 + 1.004 = 2.044, rounded off to 1 decimal makes 2.0.

          Isn't the number of significant digits after the decimal point indeed zero?

  • Structure (Score:5, Insightful)

    by CHESTER COPPERPOT (864371) on Thursday April 07, 2005 @11:38PM (#12172999)
    "The program analyzes sentence and paragraph structure and can ascertain the flow of arguments and ideas."

    So it measures structure and argument.

    How's it going to measure creativity of thought? Are we going to just pump out logic machines from colleges?
    • At first, I thought, yeah this is crap. But you can still challenge your grade. And maybe that's the whole point. I wouldn't be satisfied with a paper I spent hours being graded by a computer, and neither should his students.
    • Re:Structure (Score:5, Insightful)

      by Pxtl (151020) on Friday April 08, 2005 @12:51AM (#12173417) Homepage
      Given the abysmal failure of most people to demonstrate any mental flow of logic and ideas, but who have tons of very creative thoughts about reality, I think I'd appreciate more people like that. But maybe I follow too much politics.
    • Re:Structure (Score:4, Insightful)

      by El Micko (118401) * on Friday April 08, 2005 @01:14AM (#12173502)
      If his essay marking program is the real deal, then open it up for scrutiny. Allow "experts" to see what he's doing, under an NDA, of course... Lets put his reputation on the line as opposed to the education of the students he's neglecting.

      He's a sociology professor, and he's managed to write a natural language parser that can actually decipher meaning, and mark the relevance of the content of the essay against the question.
      <cough>BULLSHIT!</cough>
  • by SparksMcGee (812424) on Thursday April 07, 2005 @11:39PM (#12173004)
    The only way that a computer program can possibly analyze a paper fully as a paper is if it read it as a human. heuristic algorithms, however sophisticated, just aren't enough for things of this sort of importance--after all, the profs are paid to grade (well, them or the grad students). It doesn't seem like it would be too difficult to just throw in keywords and make sure that you use proper syntax in order to fool this thing, albeit the prof says keywords alone aren't enough. I find the claim that his program can "analyze argument flow" quite dubious. I'll stick to getting my several grands' worth out of my courses, thank you very much.
    • by MoralHazard (447833) on Thursday April 07, 2005 @11:56PM (#12173149)
      This is a common misconception about algorithms. Effective methods don't have be consciously engineered in every step by a human designer, even when applied to extremely complex questions that traditionally require human judgement.

      It's possible to train computer programs to translate text between languages by feeding examples of good and bad translations to pattern-recognition algorithms, which start with simple rules. Most of these models are similar to neural-net machines, which is in turn based on the fundamental theory of how animal brains (including human brains) operate. You don't design and code an algorithm, you train the machine by example, with some human-assisted trail-and-error.

      This often works because that's how human judgement works: we learn just about everything by example and trial-and-error, and we're VERY good at it (look at what millions of years of evolution can accomplish!). This isn't to say that a trained neural net machine is "intelligent" or "conscious", just that solves problems by the same mechanism that a human brain does, albeit in a much more limited fashion.

      Of course, the effectiveness of a trained machine is limited by how big a computer you have, and how well you train it. Re-creating the complexity of the human brain in software with present-day techniques and equipment would be impossible (neural net software is VERY memory intensive when it gets complex). This may change in the future, but that's another debate that I won't get into.

      I'm not saying that this professor's software actually works or not--he could easily be full of shit. I'm also not saying that you can't game one of these machines the same way spammers game Bayesian anti-spam filters: use trial-and-error to figure out how to trick the machine consistently.

      In fact, I'm assuming that a canny student could steal the software and do exactly that. After all, the human brain is a much more powerful learning machine than the program, and could probably outsmart it in the same way that people can outsmart rats.

      But then again, this is a socialogy course, so his students probably won't think of it on their own.
    • When I was in 8th grade I had a horrid science teacher who never read anything that anyone except students he saw as "problem students" wrote. Those who were not on that list turned in nursery rhymes as homework and got A's. Not only was it plagerism, but it was ever off-topic.

      I fear that my experience was not unique. I wonder how well I would do if I turned in a chapter of Moby Dick or Les Miserables.... Or maybe a section of Physics and Phylosophy by Heisenberg.
    • The only way that a computer program can possibly analyze a paper fully as a paper is if it read it as a human.

      Why not? What universal principal or physical law states this?

      I probably wouldn't disagree with you if you said that the current state of technology can't emulate humans here ... But to say it will never happen, no matter what?

      I think John von Neumann once said --- "If you can tell me exactly what it is that a machine cannot do, then I will build a machine to do exactly that!".

  • Intresting (Score:5, Insightful)

    by Patrik_AKA_RedX (624423) <patrik.vanostaeyen@NosPAm.gmail.com> on Thursday April 07, 2005 @11:39PM (#12173009) Journal
    Now if we could code a program to write those essays, the whole process would be automated. Press enter for another A. If you know how the program grades the essays, it should be possible to write a program that generates essays that comply perfectly to those rules, right?
    • Re:Intresting (Score:5, Insightful)

      by superpulpsicle (533373) on Thursday April 07, 2005 @11:46PM (#12173060)
      Why don't they just make a program to print out a bachelors degree. Save me time, money and effort.

    • If I read the article correct, he enters in certain keywords and than gives them each a weighted importance. You would have to know what keywords he's using and their weight to make a program to write the essay for you.
      • Re:Intresting (Score:3, Insightful)

        by d474 (695126)

        "You would have to know what keywords he's using and their weight to make a program to write the essay for you."

        That wouldn't be impossible. You run pattern recognition algorithms on transcripts of his lectures, handouts, readings, and on his own work on the subject (disertations, research, publications). Then it's a simple matter of identifying the phrases, keywords, and assigning the appropriate weight to them. You could add some variability to the distributions and churn out papers for the entire clas

  • Undergrad professors are usually not too excited about teaching these 18 year-old pizza-faced dorks. The problem is that the kids would rather be out drinking and screwing rather than debating the intricacies of pre vs post agrarian culture in the Southern States and the relationship between that and race relations as they exist today.

    So more power to him. He is unlikely to be getting anything better or more insightful than a parroting of what he has already delivered in his monologues to his class. Sam
  • Lazy bastard.
  • by cbreaker (561297) on Thursday April 07, 2005 @11:44PM (#12173046) Journal
    Sure, you could be great at grammar and sentence structure. You could be an ace at using proper english.

    But how would hidden talent and creativity be found? How will the teacher know if his students are actually trying hard to write their papers when all he does is check the thing with a computer program?

    It's a really terrible idea and I think it's really cheezy. Ohh, he saved some time. So does that mean he now gets paid less? Does this automation get the students a discount? Yea, right.

    If I'm going to put a lot of work into writing an interesting paper about something, I want someone to read it.
    • This is school we're talking about. Being talented and creative are dangerous and not recommended. Trying hard only means that you wasted more of your time than the next guy. If you want to write something and actually have it read, try the intarweb. Or write a book.
  • by crmartin (98227) on Thursday April 07, 2005 @11:47PM (#12173062)
    Look Tom Landauer's [colorado.edu] work at University of Colorado.

    It makes more sense than you'd think: it turns out that knowledgable essays in a particular domain cluster statistically in useful ways. Yes, it does mean that something like Molly Bloom's Soliloquy [answers.com] wouldn't necessarily score very well, but then if you didn't know it wsa a Nobel Prize winning classic, would you think it was well written?
  • by Chris Pimlott (16212) on Thursday April 07, 2005 @11:47PM (#12173068)
    What does this say about the field of Sociology? :P
  • by spin2cool (651536) on Thursday April 07, 2005 @11:48PM (#12173073)
    FWIW, this prof is at the University of Missouri (in Columbia, MO).

    Gee - you'd think the submitter could RTFA...
  • Regurgitation (Score:3, Insightful)

    by dcclark (846336) on Thursday April 07, 2005 @11:50PM (#12173090) Homepage
    It works by scanning text for keywords, phrases and language patterns.

    Which is to say, this prof is asking students to regurgitate data. Given, a certain level of base knowledge is necessary in any class and topic, and regurgitation (aka parroting) is an easy way to check that base knowledge. If a paper is assigned on a particular topic that they've been studying, then this sort of program can easily check for base level ability to spit back key words and phrases.

    But, I seriously doubt that the class is ONLY about that base knowledge -- or that the program can reasonably check for anything more. I've had classes where the prof or graders did basically the same thing that this program does (i.e. check only for key words, phrases, and patterns they want to see), and I have little respect for those profs.

    If you don't want to put even a basic amount of effort into checking a paper, don't assign it -- find some better way to check students' progress.
  • by Japong (793982) on Thursday April 07, 2005 @11:51PM (#12173095)

    This one is just nuts. Why on earth am I writing essays which are going to be marked automatically by a machine? It's bad enough that scantron cards have found their way into subjects where they're totally irrelevant (a multiple-choice test for a university level Shakespeare course?), this is just another reason why post-secondary education has become increasingly less complete.

    If he's allowed to use a machine to save him the effort of reading an essay, I should be able to use a machine so I don't have to go through the effort of writing one. Trust me, as arduous as it is to read a 20 page essay on the relative merits of liquid rubber concrete compound fasteners, writing it takes a lot more effort, a lot more time, and it damn well deserves to be read by the professor who assigned it.

  • Just wanted to point out that the software was created originally for the purpose of qualitative coding. Grading essays is one of several other applications it has proved capable of addressing.
  • by wintermute1000 (731750) on Thursday April 07, 2005 @11:52PM (#12173107)
    I was required to take a lot of writing classes for my college (and still haven't finished them all) and I've observed the quality of my writing go up appreciably since I began school. However, the reason I've become a better writer is because my essay graders write copious comments about where I'm going wrong in my papers and what I should do to improve - and they read the next paper I write for the class with those things in mind, and tell me whether I've improved sicne the last one.

    The article didn't say anything about what kind of feedback the program provides, but I can't imagine it's anywhere near as helpful as the paragraph-long evaluations of my logic, style, and structure, which I got back with every paper I ever turned in, and I'd be impressed but surprised if his program took each student's previous weaknesses into account in the course of the evaluation. In writing, practicing can only do so much - the real help is in constructive feedback, and I just can't imagine where these students are getting it if not from the human graders of their papers.
  • by Golgafrinchan (777313) on Thursday April 07, 2005 @11:54PM (#12173131)
    The company who administers the GMAT (Graduate Management Admission Test) has been using a computer grader for the analytical writing portion of the exam for several years now. They call it the e-rater. Both a human and the e-rater grade every essay.

    According to ETS [ets.org], the e-rater agrees with the human grader 98% of the time.

  • by rainwalker (174354) on Friday April 08, 2005 @12:02AM (#12173179)
    From TFA, which apparently no one has read yet:
    "The final papers, which he does read, are usually much better as a result of Qualrus, too."

    There you go! For the reading and comprehension impaired, here's a summary of what's actually happening, which even the reporter didn't get:
    1. Students write a draft of their essay, which they then upload via a Web form to this program
    2. The program gives them a score on various parts of their essay, giving them valuable feedback on what needs to be improved.
    3. Students improve the pieces of their essay that the program suggests.
    4. Students submit the final draft to the professor, who reads and grades each one by hand. Due to steps 1-3, the quality of the final draft is much higher.

    This sounds like a great thing to me. Wish I had something similar for my students. I don't have the time to read through dozens of drafts for every student. Too bad I'm not in sociology.
    • by VeryProfessional (805174) on Friday April 08, 2005 @12:21AM (#12173296)

      From TFA:

      The computer-generated scores count for about a third to a quarter of students' final grade for Brent's class.

      There you go! Make sure you RTFA very carefully before accusing others of being reading and comprehension impaired.

      • From the sound of it, I think that the "third to a quarter of students' final grade" is like credit for a homework assignment--doing the first draft and submitting it to be machine-graded. The other 2/3 to 3/4 is probably the grade on the final essay.

        I have a Spanish class, and in it, we have to work machine-graded WebCT multiple-choice and fill-in-the-blank exercises. It counts for some credit as our homework, but the lion's share of our grade comes from our manually graded written exercises and our tes
    • Newsflash! (Score:3, Funny)

      by pnatural (59329)
      1. Students write a draft of their essay, which they then upload via a Web form to this program
      2. The program gives them a score on various parts of their essay, giving them valuable feedback on what needs to be improved.


      This just in! University English professor discovers the POSIX toolchain. Novel misuse of cat, awk, and sed and friends expected. Film at 11!

  • by d474 (695126) on Friday April 08, 2005 @12:40AM (#12173368)
    So since the professor's time is worth about $36.00/hour and he spends 200 less hours on reading papers...

    200 hours * $32.00 = $7200

    He teaches about 84 students...

    $7200 / 84 = $85.71 refund for each student. It's party time!
  • by d474 (695126) on Friday April 08, 2005 @12:46AM (#12173398)
    Okay Slashdot Editors, time to fork out the $$$ to get some auto-moderating going on in these threads! Wait, can this grading program test for humor? No? Fuck it then.
  • by d474 (695126) on Friday April 08, 2005 @12:51AM (#12173413)
    If this professor's analysis can be "simulated" by a computer program, then he was obviously not doing a thorough enough analysis to begin with. I know plenty of professors that would laugh at the idea that a computer program would be able to "calculate" emotion, nuance, subtle sarcasm, humor, insightfulness, etc...

    This professor should be fired.
  • by IntelliTubbie (29947) on Friday April 08, 2005 @01:05AM (#12173475)
    "But Professor, my original essay was really good! I just had to add a bunch of crap to get past the lameness filter ..."

    Cheers,
    IT
  • A While (Score:4, Interesting)

    by rm999 (775449) on Friday April 08, 2005 @01:12AM (#12173497)
    "How long before he's replaced entirely by his own program to cut down on staff costs?"

    I would say a long time. A program that tries to understand natural language requires some sort of "intelligence," a quality that humans definetly possess and computers, up to now, definetly do not.

    AI still mostly consists of certain hacks to trick other people into thinking the programs are intelligent - basically attempting to fool the Turing Test. This can often produce great results and can be very useful, but almost never replaces a human in complex tasks (such as natural language processing).

    The difficulty arises because humans cannot easily (or perhaps possibly) comprehend their own intelligence. It seems so natural to read a sentence and make sense of it, but when it comes time to program a computer to do it, most people try to emulate the behavior of their own comprehension. This may trick some people, but the simple nature of the programs cannot possibly be as powerful as an actual human.

    The best solution, in my opinion, is a closer study of neuroscience and how it can be applied to silicon (or how new technologies need to arise to emulate the complex neural structure of the brain).

    I know that people are starting to use computers to grade standardized essays, but there (currently) must always be a human checking the results because of the small number of unforseen cases that the hacked algorithms cannot do a good job. After all, the programs do not "understand" anything that is written. That is why I postulate it will be a long, long time before computers can truly emulate humans.
  • by bombadier_beetle (871107) on Friday April 08, 2005 @02:06AM (#12173700)
    Announce to the class that they have two options. If they want their paper to be computer-graded, the maximum they can get on the paper is a B. Only hand-graded papers can earn an A - but they will be judged on original thought, well-chosen sources, and total structure. If they don't meet those criteria, they will be penalized (and as such, would score lower than the computer would give). That way, students who aren't confident in their capability for truly excellent writing and novel thought can choose the computer grader for a safe B, while the truly thoughtful and creative students can be justly rewarded.

    Wait - what the fuck am I saying? ALL college students should be graded by the non-computer criteria I just listed, and those who can't do (or at least attempt) that kind of work shouldn't be in college.
  • by Vellmont (569020) on Friday April 08, 2005 @05:07AM (#12174291) Homepage
    has gone right into the shitter, at least at the University of Missouri. This program was actually approved for school wide use? What the hell is wrong with the University of Missouri? I find the whole thing just reprehensible. I guess at least Ed Brent has finally defined exactly the qualities of a "good paper".

    I'd be interested to put the Gettysburg address, MLKs "I Have A Dream" speech, major works of literature, etc through Mr. Brents meat grinder and see what grade they get. The whole thing reminds me of that scene in Dead Poets Society where they try to measure the "greatness" of a poem using trumped up terms like "importance" and "perfection". Sprinkle in a little computer wizardry, and suddenly you've got a mysterious, unbending, rule machine.

    Frankly this kind of thing just disgusts me. I'm no romantic, but you can't analyse how good a paper is based on some algorithm. It's like the idiots who try to analyse a songs potential through computer analysis.

The clash of ideas is the sound of freedom.

Working...