Both sides can make their claims. But unless someone can do a proper experiment with a control planet, and make that experiment repeatable while you're at it, its all speculation. Not proper science.
You can't be that dense. By your reasoning, just about every aspect of science is "speculation".
Almost all non-trivial physics models are simulations. This includes everything from CFD's to weather and climate models. These simulations are built upon physical equations the describe the phenomena. These models are run against KNOWN CONDITIONS to see if the are accurately modelling the phenomena.
In the case of climate models, the models are initialized with pre-industrial conditions (with various small tweaks to the initial conditions to create what is known as an ensemble). Then the models are run forward to present day to see how well they modeled the KNOWN conditions that happened over that time period. And, not surprisingly, the climate models do a pretty good job. Keep in mind, these models are not STATISTICAL models. These are PHYSICAL models, i.e. modelling the actual physical dynamics of the earth's climate.
And even then, the models are just tools. The research used to the develop the models are based upon real world observations (historical as well as current). And this research has been ongoing since Fourier first proposed greenhouse gas theory back in 1824.
Speculation is someone saying "The moon is made of cheese!". Science is someone showing objectively that it isn't. Idiocy is looking at the science and disregarding it as nonsense since it goes against your belief that the moon is made of cheese.