The PDF was an interesting read, though I agree that the money they are dishing out is pretty paltry for all the free review they are trying to garner. Furthermore, I think they are taking platform neutrality in the wrong direction by locking the idea in to the x86 architecture.
But about how it would work, they are basically enforcing strict limits on how the code can be structured. The limits are designed to make the code easily analyzed. Anything that falls outside the strict requirements is rejected. It doesn't work for antivirus because they have to deal with any code that comes in without restriction.
As to why it doesn't work for OS... There is no reason the basic concept wouldn't, aside from the performance penalty and increased code size. (Though further compiler optimization could minimize or eliminate some of that).
However, if you want to go that route of making an OS do it, you might as well pick up a decent modern RISC architecture, because you're already breaking compatibility with any past program for any OS on the x86 CPU. Most of what they are doing is basically taking something that is standard on RISC and shoehorning it into the CISC architecture of the x86. Namely that instruction boundries can be reliably tested for jumps. They enforce that by requiring jumps only to 32 byte boundries, and then verifying each 32 byte block for correctness. Combined with disallowing self modifying code and eliminating the stack completely, all code that executes can be properly analyzed ahead of time.
The concept looks sound to me (Experience working low level with x86 architecture) but the security still relies on the implementation. Off the top of my head I can think of several ways to break the sandbox depending on how it is implemented. However the PDF is quite short on the details to evaluate the implementation. Namely, what exactly qualifies as an allowed x86 instruction, and for the syscalls that are checked, what the check is, not to mention the potential for bugs in the syscall handler for what would otherwise be valid calls, and even potentially the state of the OS or process when the protected code is executed.
Overall, I don't think this is the right direction for the web platform. Theoretically interpreted byte code should be more secure because it doesn't do anything that the interpreter doesn't explicitly allow (Javascript, Java, Flash, etc) and we see where that got us.
