For CS, there are a number of programming goals that can be graded automatically, and that is used in several courses - both in Udacity, Coursera and the ML course mentioned before.
You just have to make the student respond to an interface - i.e.: setting an object or variable to the answers or outputting them in a particular way.
For example, in the NLP course the first exercise involved using regular expressions to extract e-mails from web pages. You had some starter source code and a set of pages included with the exercise. You just fill in your code, and run it locally to test it against the local set. Then you submit your code, and the grading software runs it against another set of pages, from where it gets your score for true positives, false positives, and misses (it also used the set you already have, but that had a lower percentage, I think).
It is not perfect (you can't easily measure the cleverness of the answer or the quality of the code, just how well it works), but it is very nearly free.
Given the average quality of the CS courses out there, I think that this system is very acceptable, and bound to get better with time.