Do we not have to worry the most when the faint objects do *not* move at all, between pictures? Then they are heading straight for us.
Incorrect, we are judging their movement as compared to the background stars, which are (relatively) fixed in position while we move. An object on collision course with us will not be heading straight the position we are at when it is observed, but it's trajectory will have to carry it into the path of our orbit, at the same time that we occupy that same point. If an object appears in the same fixed position as the starts around it, it is either too distant to concern us, or it is in a concentric orbit at the same angular velocity as us and not an issue (although this would be an extremely interesting discovery).
Even something on a perfect spiral trajectory on the elliptical and matching our angular velocity would be detectable due to observations at different times of the night (as we rotate around the Earth's axis) and the different Doppler shift compared to much more distant stars.