Yes this is entirely the case. I would expect that the number of photons produced to fall off as a steeply falling power law. There will be vastly more UV photons than x-rays and vastly more x-rays than gammas. Until one gets to really high energies where the photon is showering and the shower is producing lots of additional ionizing particles, it is a good approximation that a single photon will damage only one atom or bond.
If there were a significant flux of x-rays that would already be pretty bad and it wouldn't really matter how many gammas were also there, because the x-rays are already ionizing radiation and they could possibly significantly elevate risks for affected persons. So while a single gamma might not be devastating, if that were accompanied by a large flux of x-rays, bad things would still happen.