People are attacking libertarians over it because they don't know the difference between anarchist, minarchist, and libertarian. ... Libertarians would be, rightfully, more divided on the issue but one thing is certain that [they] should be generally be constitutionalists. Strictly speaking, libertarians want the USD to be backed by something tangibile (like gold).
Apparently you don't know either. Hint: these categories are not mutually exclusive. There are libertarian anarchists, libertarian minarchists, and even some (rather confused) people who are neither anarchist nor minarchist and yet like to think of themselves as libertarian. Some (but not all) constitutionalists would fall into the third category, bordering on minarchist but with significant exceptions.
The defining characteristic of libertarianism, the Non-Aggression Principle, doesn't have anything to say about the USD being backed by gold or anything else. There's a problem with the way in which paper currency replaced gold in the U.S. (via confiscation), but that's ancient history by this point. The N.A.P. does, however, conflict with mandatory deposit insurance (like the FDIC), KYC/AML requirements, money-transmitter licenses, etc. The consistent libertarian will favor unrestrained voluntary trade without third-party interference.