You have a good point when you say that allowing all non-commercial use of copyrighted works is almost the same as abandoning copyright for the masses. And this is what the Pirate Party wants.
If we do not allow non-commercial use, somebody has to check all private communication to check if copyrighted works are infringed. Would you like some government entity to eavesdrop your private communication with your local journalist, lawyer, doctor, priest, or secret lover? We pirates think that a society where private communication is impossible cannot stay democratic in the long run.
The alternative - no monitoring of private communication - is almost as bad. This way people can break the law with no risk. Many people have a strong incentive to break the law here: Free access to cultural content. But if a majority of the citizens regularly break the law, they also loose respect for the law. Lack of respect for the law is extremely dangerous for our society, as it is based on people respecting the law.
In many countries there are equivalents to "fair use", allowing non-commercial use of copyrighted works. But these exceptions in copyright law are under attack and almost every adjustment of copyright law these days seem to remove some of the rights to non-commercial use we citizens have.
For example: In my country before 1995 I could legally mail a copyrighted mp3 to a journalist. Today this is illegal. I do not want our government (or anybody else for that matter) to monitor my private communication with journalists. If such communication is monitored, the principle of source confidentiality of journalists would be broken, and nobody would dare to tip journalists about for example government abuse.