Sorry, but the Sun has everything to do with climate change when combined with the variable orbit geometry of the Earth around the Sun.
This is absolutely true -- over millions of years. It does not explain the warming trend in the past century. Your mode of argument is like saying "all will eventually die of old age, therefore automobile accidents don't kill people." There can be more than driver of climate change, and the timescale over which a driver of change operates is very important. Even if car accidents are less likely to kill you than old age, the fact that they kill you at 19 years old rather than 90 makes a big difference.
Four degrees C rise over 100,000 years is no bit deal for the human race. The same change over a century is a very big deal. Not species extinction for humanity by any means, but massive economic dislocation. Imagine the western US as much more arid than it is now; it could mean the end of agriculture on the Great Plains.