Not only are there legitimate criticisms about the interpretations of measurements, but even of the measurements themselves. Prior to the 1920s, most stations around the world providing temperature measurements weren't staffed with trained personnel recording the data. In many cases, the measurements weren't taken at any regular intervals, but rather when someone had free time in between performing other work to do so. In fact, it was often a janitor or other completely untrained staff member recording temperatures because it wasn't viewed as a critical task and being with a degree or two was good enough. This began to change around the 1920s and 30s, especially as radio and television began to take hold and the sciences around climate and weather began to develop.
Accurate instruments didn't even become available until the late 1800s, so any direct measurements taken prior 'til then are highly suspect regardless of who was in charge of taking them. This makes most of the direct measurements prior to ~1930 extremely limited and any direct measurements prior to ~1880 just about utterly useless. It isn't until the 1960s that you really start getting measurements useful to a discussion about 1C of variability in climate worldwide. Earlier climate data is even worse, as that primarily comes from proxy measurements. Two problems there: 1) the proxies lack the precision to be useful in a discussion of 2-3C of climate variation and 2) the proxies don't agree with one another, nor do any of them agree with direct measurements.
Statistical smoothing is used to work around this, and that's where we get into what you stated about problems with interpretations of the measurements and the way in which mathematical principles are applied. You can blur your way out of very minor errors and largely leave the data intact, but you can't do so when your error bars are orders of magnitude greater than the trend you're seeking. At that point, by the time you've blurred away the errors, any reliable data hidden in there has long since been turned to mush.
None of this is to say we shouldn't be working hard on climate science. Rather, it's to say we need to do better work on the subject and stop pretending we understand our world's climate or its history. None of this is to say we shouldn't be working hard on shifting from technologies that pollute and poison our environment to cleaner and better technologies. Rather, it's to say we shouldn't jump to absurd doomsday conclusions and take radical actions like geo-engineering "solutions" to problems we can't yet say for certain even exist.
Good science is honest about its faults; about what it does know, what it should know, and what it can't know. Good science starts with good data, strict methodology, open presentation of work for review, and the ability to accept valid criticism. Unfortunately, climate science has been tribalized into an "us vs them" situation unlike virtually any other science. There are great debates in science about the validity of things like string theory, quantum loop gravity, etc, but those arguments are based on the underlying math and how well it explains what we've actually observed thus far in our universe. The debate around climate science is mostly crap. It's two sides screaming "WE'RE RIGHT, YOU'RE WRONG!" at each other. I'm not surprised that there's a group that will always refuse to believe humans have any impact on the climate of our world because we have a group (and it's largely the same group) that refuses to believe evolution is a real thing despite the fact that we can observe it happening in front of us. No, what surprises (and saddens) me is that there's a group that believes in AGW with a religious fervor on par with the worst of the zealots. Questioning the data or the science behind their dogmatic beliefs is an attack on their very being, and they respond by spewing hate. It's unfortunate, because there are extremely important questions to answer and there needs to be real work done on answering them rather than merely confirming beliefs.