(quoting Douglas Adams)
“It is known that there are an infinite number of worlds, simply because there is an infinite amount of space for them to be in. However, not every one of them is inhabited. Therefore, there must be a finite number of inhabited worlds.
There's your problem right there! (Yeah, I'm sure Adams was perfectly aware of this, he was, after all, writing a comedy novel.)