That would require the Earth to be very very special indeed, and I just don't see it.
Not at all. For example, I just generated a random number between 1 and 1e9. It was 869,502,332. By your logic, therefore, that number must have been very, very special. But no, it was just really improbable and that number happened to come up.
It may very well be the same case with life. Life could just be extremely improbable, and Earth just happened to be "the number" that was picked. This is what the Anthropic Principle is all about. Our perceptions are colored by the fact that we're here, so we think, "Since the Earth is not special, therefore, other planets must have life like Earth." It might just be that Earth was the lottery winner.
I said this in another post, but I'll say it again: The best evidence against life being common is the fact that it only happened once on Earth. It's fairly conclusive that all life on Earth has a common ancestor. If abiogenesis were easy and common, it wouldn't just stop once it happened one time, it would happen continuously over the billions of years since it happened for us. But it didn't.
And honestly, life on Earth being completely unique in the universe isn't that hard for me to believe when I look at the utterly insane complexity of cellular machinery. But again, extreme improbability doesn't matter when we're deal with the anthropic principle. We don't sense how long it took for intelligent life to pop up, just like we didn't sense the 13 billion years until you and I were born to think about all this.