Unfortunately, that opinion is a great part of why things are in the current mess. It presupposes that if you take away the money, then the "extras" will be curtailed and only the core "important" parts will be left. But this blind faith that the cretins and crooks that are currently in charge, will willing give up their cash cows and "do what's right" is so incredibly naive as to be almost unbelievable.
Blindly "tightening the purse strings" leads to those parts of government that are good and useful to be sacrificed first, while the partisan and corrupt parts better defend themselves and their budgets. So, instead of a progressive nation of healthy, happy, nutritionally fed, employed, well educated citizens in a nation focused on freedom, scientific and technological advancement, we have become the secretive spymasters and bullies of the world, looking for the next war to line the pockets of the oligarchs, while the bigoted, ignorant masses fight from paycheck to paycheck, if they can find a job, until they die from easily preventable disease, if they survive the worst infant mortality rate of any first world nation.
"Tighten the purse strings" has meant killing the National Aeronautics and Space Administration, killing the National Institutes of Health, killing the Center For Disease Control, killing the Food & Drug Administration inspection program, etc. ad nauseam. Meanwhile the secret budgets, the crony protected waste, the bureaucracy, swells and continues unabated.
Instead of demanding that the money be taken away, we should be demanding that the places where the money is being mis-spent be stopped, or at the very least that the places that lead to a better society are better funded, in the hopes that doing so requires funds to be reallocated from those things which are wasteful. It has been a tremendous coup by the oligarchs to get people to focus on the dollars, not what value are they getting for the dollars. Government of a large advanced nation by its very nature will involve sums of money so large that the average person will be staggered to the point that most wont even comprehend just how big the amount is. This inevitably leads to the uninformed, most radical knee-jerking among the mob to scream at the size of the number, not at any analysis of how it should be spent.