How long would it take for someone to walk from the southernmost tip of Africa to the southernmost tip of South America?
A long time, right? Has anyone done it?
But over generations, humans moved little-by-little, farther and farther out, and given a hundred thousand years or so, there were humans all over the face of this planet. It's not like one person left Africa and made the trip by themself in one day. No, but humans multiplied and spread out.
Given the universe's age, more billions of years old than we can comprehend, even if we can't bet on a traveler making a non-stop streak across the sky to our backyard, why is it unreasonable to assume that they might have learned to live in space, and might have moved further and further out, little-by-little, and that they could be within a lifetime's travel of Earth right now?
The fact that you're unlikely to meet some form of life from another planet is in no way an indication that you're unlikely to encounter something whose great great great* ancestor was from said other planet.
We're so busy searching for signs of life that we don't realize that a sufficiently capable civilization might be able to exist where such signs of life don't exist, in the same way that we can exist in a plane or on a submarine. For all we know, the first bacteria on Earth could have been scraped off of the "boot" of some interstellar traveler visiting the Earth as the cosmic equivalent of the Grand Canyon. They could have been nearby before we were even here.
And we're searching for intelligence, even though it's much more likely that we'd find simple life akin to bacteria? Has a scientist ever considered that if we do find bacteria and we set out in that direction, by the time any of us get there there might be intelligent life there? And has anyone ever considered that somewhere out there some extrasolar travelers might have made the same bet regarding Earth?