Very few people argue for more taxes than necessary, or more government restrictions than necessary. In that sense, we're all libertarians. We differ widely on what's "necessary".
I am perfectly content to define everyone who would prefer less taxation and fewer government restrictions as libertarian. I think in a broader and practical sense that is an accurate description. But the fundamental underpinning of libertarianism is the ideal of a society that only uses government force and coercion in order to counter force and coercion.
And although therefore libertarians traditionally view just defense, police and the courts to be absolutely necessary to the function of government most libertarian leaning leaders are usually just seeking modest reductions in taxation and spending as a gradual approach to reverse long term trends.
Since the libertarian viewpoint is based on freedom, prosperity and reducing the use of force and violence in society then it isn't good to go cold turkey on government programs which could result in violence in society as people's lives and livelihood's are disrupted. Merely shifting the use of coercive force from government to individuals isn't a net gain. When it regards dismantling social programs I think it is up to libertarians to demonstrate that non-governmental charities and individuals are up to the task of that transition before we phase any of these programs out completely unless we are talking about merely shifting resources to demonstrably more effective programs.
What I do disagree with those that are critical of libertarianism is on the ideal of a libertarian society. Whether practically achievable or not to its fullest extent, to me the ideal of libertarianism is an inherently good value to strive for. And it is a given that this assumes a healthy society where charity and free will have picked up the slack in areas of social good. To me the libertarian ideal is simply like saying better parents don't beat their children, or we should treat others as we wish to be treated. The point is that you have some ideal to strive for and a system of government to perfect.
And conversely that having a society which is based on the growing use of coercive force against its own people to achieve very narrow partisan political gains is a very unhealthy path to be on.