I'm sure that a makeshift homebrew directional antenna could sniff these signals out in a suburban or office park area just fine.
Even in your high-rise case, the signal could be narrowed down to a small subset of all devices. Processing the sum of a few signals to pick out "words" shouldn't be too hard, either.
Not to mention, I said three times the distance to frigging PLUTO. Presumably, attackers could get within 100m of most targets. I don't think you realize how much easier that is.