Yes, you are correct that everything is legal except that which has been specifically forbidden, and that you are only liable for your own actions.
Here is the problem: copyright infringement (reproducing or distributing without authorization) has been illegal in the US for more than 200 years. When someone watches a video on YouTube (for instance) it is YouTube, and not the person who uploaded it, who is distributing without authorization and is therefore liable for copyright infringement. YouTube would not be liable for the actions of someone else (whoever made the copy on YouTube in the first place), they would be liable for their own action of distributing copyrighted material without authorization.
So the DMCA carved out a special provision in copyright law in order to allow services like YouTube to exist. You will not be held responsible for distributing copyrighted material as long as you remove the copyrighted material when you are informed that you are not authorized to distribute it.
And you can't say that these service are being held liable for the actions of others. They are the ones who say 'we will distribute whatever anyone uploads, no questions asked'. That is their action, not someone else's. Look at it this way: suppose you ran a consignment shop, and offered to sell whatever anyone gave you. Do you think you would get away with saying 'not my fault' when some of the things you were selling turned out to be stolen or illegal? No, you would not.