Well on Earth, it took 3 million years for life to go from very simple single cell organisms to primitive animals.
Their hypothesis says that rather than the occasional rock like Earth, the environment may have been favorable on quite a few planets. Well, "quite a few" being "just about infinite" in our concept of numbers.
Meteorites bombard planets. We know that. We believe planetary collisions happen. I don't think we've observed that yet. So pieces of planets are ejected on a fairly regular basis. Earth seems to do a pretty good job of keeping our chunks attached
If the idea was not that life evolved relatively close to our planet's life timeline, and with a virtually infinite of source planets in relatively close proximity to each other, it would be more feasible to consider that panspermia could be real.
It will be quite a while before we can even begin to test for something like that. Well, unless someone drops by with a starship and 2 billion years of intergalactic research. Or more likely, someone does build an Einstein-Rosen bridge. I'm sure if we are the first to do it, it will be militarized, to be the way to win any war, anywhere. Who needs nukes, when we can just open a bridge from an enemy stronghold to the photosphere of any convenient star for a few seconds. Hmmm, considering that, maybe someone has made it, and realized the potential for harm.