I have never deliberately clicked on a banner ad. I certainly have seen banner ads before, so it's impossible to say that they have had NO influence on me, but I can't remember ever seeing an ad for any of the companies that I actually buy things from (gibson, asus, newegg, amazon, etc). Even if I did, for anything other than commodity items with no practical difference between them (deodorant, cola), I would base my decision off of some research, not the contents of a banner ad. Honestly, if you make an ad that is really that good, I'll see it on youtube anyways (e.g. that Old Spice "now I'm on a horse" ad).
One could argue that I would generate impressions for the sites whose ads I view, but I would contribute no click-throughs, thus lowering the click-through rate. You then could reply by saying that I'm obligated to click some of the ads, to which I would respond 1) fuck you, no and 2) then that unless I actually purchase something, I'm just watering down the statistics again.
At the end of the day, advertising is about ROI, and inflating impressions or click-throughs doesn't make a damn bit of difference.
Plenty of companies (Amazon and Newegg) get plenty of my money over the internet, and if you can't figure out how to do that, then I'm comfortable with you going out of business.