It's not about the concentration (absolute or relative), but the effect. If someone started to double the concentration of O2 in out atmosphere from 21% to 42%, I would call that pollution because of the large number of negative externalities (ie. due to the impact it would have on forest fires). If someone increased N2 concentration from 78% to 89%, we'd all have a harder time getting to oxygen we need to function, so continued willy-nilly dumpin of N2 into the atmosphere would be pollution.
Actually, changes to low concentrations of greenhouse gases has a larger effect than changes at high concentrations. In a simplistic picture where the transmissivity of the atmosphere in a certain band depends on a single molecule, you can write the transmissivity of the atmosphere as I = 2^(-t/h). Where I is the % of the light that makes it out of the earth's atmosphere, t is the thickness of the molecule, and h is the level at which that molecule blocks half the light from making it out of the atmosphere.
Let's assume that 200ppm will block out half the light. At 280ppm, 38% of the light made it through. At 390ppm, 26% of the light makes it through. At 480ppm, 19% of the light makes it through. By 5%, basically no light makes it through, which means that from 280ppm, 480ppm is halfway to 5% and at 390ppm (where we are now) we are a third of the way to the effect of 5%. In actuality, every molecule has multiple absorption bands, and I'm sorry I don't have exact numbers handy. But, this exponential dependence is one of the reasons why methane, which is normally at a much lower absolute concentration, is a more potent greenhouse gas than CO2 (the other reason being that the ocean won't gradually reabsorb it).