Let me try to explain it:
Java (The language and vm) does in general allow code to do anything. Mess up the system, call native methods and so on. In this it is no different from other languages such as c++ and php, which is why no one are running c++ og php directly in a browser*.
But sun wanted to be able to run Java code in the browser, so they made an security manager, which in principle allow you to specify exactly what parts of the java standard library a program should have access to. So you can restrict file access, to specific sub directories. Control thread and process control, class loading and so on.
So each call to a potential dangerous method first goes to the system manager to ask for permission, and if permission is granted the rest of the method is executed. And here is part of the problem: Default for all methods is access granted if there is no call to the security manager.
So for every method in the Java library, they had to determine if it could me misused, and it it could be misused, they had to insert call to the security manager to verify that the user have access to use this method in this way based on the arguments to the method.
And just one method which don't verify its argument as good as it should is enough to grant access.
The problem is made far worse by the combination of the java class loader which loads classes dynamicly, and the java reflection api which allows dynamic class loading and method calling. Verifying that code can't use the reflection and class loader to bypass the security system is really difficult, as proved by the many security problems.
So when people say the problem is in the java browser plugin, they are not exactly right. The problems are in the java security manager, but I think the browser plugin is the only component which uses the security manager. Our server code for example don't need a security manager to control exactly what our code have access to, because the code is written by guys we trust, and the rest of the security is handled by the system.
*Or at least should not. Active X is Microsofts version of the "Try to run untrusted general purpose code in the browser project". And its security record is as bad as javas.