Once we become a space-faring civilization, this rarity value attached to non-Earth rocks will seem very quaint.
What a strange way of thinking of it ... so we have a Venn diagram, and then it's Earth and non-Earth. That's a little too simplistic.
In your scenario, we'll have Earth, Mars, Venus, Alpha Centauri, Vulcan, Ceti Alpha V, and what have you. But they'll all be boring because they're "not Earth".
They may not be universally valuable, but like people collect souvenirs, they'll have some sentimental attachment. Or they'll be sufficiently rare as toe have a degree of uniqueness.
Even if we were space faring, a rock from the furthest planet in the universe is worth more than the one you're standing on, because it's harder to get another one.
The reality is, there's a relatively small amount of material we call "non-Earth" which we can access. You're right, they are deemed special because they come from someplace else, and not everybody can have one.
But you'd have to be a space faring species who can instantaneously travel anywhere in the universe to say that rocks from further away and harder to get won't have some cachet to them.
It's not like we'd become a space faring civilization (assuming we ever actually do before we go extinct) and suddenly all sense of distance and place of origin disappears.
But you sounds jaded about space rock to an extent that seems to imply you figure you'll have access to rocks from the entire universe within a few weeks. Right now, here on planet Earth, in any meaningful sense of the word ... "space rocks" are very rare, and to people who really want them, quite valuable.
But it will always be true that the further away it's from, and the harder to get it is, the more value people will assign to it.