You seem to think that without government authority, power will not be exerted against you, and not against those least able to fend for themselves. This is not the case. Those who have wealth and power will always seek to gain more at the expense of anyone else, and through whatever means they have available. This extends to backroom deals, monopoly power, insider trading, unsafe working conditions and violence. Private police forces and strike breakers.
Government, restrained by a separation of powers with checks and balances over each other, should serve as the check against the power of the wealthy over those who have no wealth and therefore no individual power.
Unfortunately currently the power of government has been bought by the wealthy. What's needed is a restructuring to reduce the influence of money in politics. But the answer is not to remove the power of government. That power vacuum will simply be filled by the power of the wealthy, and the poor and middle class will have no defense against their rapaciousness.
This is the perpetual blind spot that libertarians just don't seem to understand. Without government restraining wealthy sociopaths, they will kill or enslave you, through "legitimate" means or otherwise. You have this fantasy that either you will be in charge or that everybody will just get along, but they won't, because there are evil people in this world who are only restrained by the collective action of government by the people.