Well yes, much in the same way one infers the presence of a stream of electrons from an electrical charge or the Big Bang from the CMBR, relative proportions of hydrogen, helium and lithium in the Universe and the red-shift of distant galaxies. Even a particle accelerator like the LHC at CERN does not in fact directly image subatomic particles. For chrissakes, what you "see" isn't a raw image, but is heavily processed by your nervous system, beginning right at the retina itself, then by the optic nerve and then by visual centers in the brain. In other words, what you "see" isn't actually the photons that the physical structures of the eye captures.
Lots of science is inference, seeing as many phenomenon cannot be directly observed. If you're saying inference is somehow questionable, then you're basically calling all form of observation questionable.