There's more to it than that. These libertarians all circle jerk each other over a revelation that money equals debt and inflation is a hidden tax on the people. It likely all started with the zeitgeist movement, which is merely an extension of the wild ramblings of Acharya S.
What all these new age libertarians fail to realize is that for most of history the world ran exactly how they are advocating. The invention of "easy credit" isn't a genius conspiracy perpetrated on the people by shadowy unknown figures, but rather an attempt to empower the common man with privileges like land ownership, and starting his own business. Banks, or anyone for that matter, wouldn't lend you hundreds of thousands of dollars to buy property if it had to come out of their own pocket. This means the only people able to buy land, would be those with the cash on hand, or more accurately, the rich. History is full of elite aristocracies of business' and land owners that existed actual tyrants over the common man. I don't think anyone really wants to go back to that times.
That being said, the banking system certainly isn't perfect. Allowing a private entity to have so much control over our money supply is probably a bad idea. At the very least the central bank should probably be government controlled and not for profit, with the sole ability to lend money through brokerage arms. Private banks would thus have to become brokers for the publicly owned central bank. Of course, given the titanic industry that is private banking, it would be a massive and messy undertaking to make a change like this.