Of course nut jobs, communists, professional agitators want to be free as well.
Freedom would imply the liberty to ignore these people. We're it becomes dangerous is when people say that it would also imply the liberty to beat them up, or even kill them. A moral compass would prevent this, but not everybody can be trusted to have a moral compass. There comes the choice: should there be a distinction between people with and without a moral compass - is there a need for a government that locks "bad" people up? I think so, but as soon as you give the rights to do so to a government, it will expand its authority. This means putting more people in jail for questionable actions. It will invent laws that first slightly bend, then outright contradict what was its goal. It will start spying on the people it says it wants to protect.
As long as people disagree with each other, they will team up with other people that agree with them. That is called a government and the purpose of that government is to oppress the ones that don't agree with it. After a while, nobody is being served by that government anymore.