There's filters in between. If you're in a modern and well maintained building, that's okay (you'll get the smell though, but you won't get smoke or contact highs, lol)
Unfortunately, modern and well maintained buildings aren't the norm in a large part of the world. I lived in a "super luxury" 3 years old building right on the steps of MIT for a few years and they had to make the building smoke-free because of the way the heat loop had been built (there was filters for the A/C and air in-take, but for the heat, things were connected).
The actual air that was being pushed around was filtered and purified, but weed smoke would go through the system backward when they were off.
And even if you're not talking about that, only the most well built places are completely sealed from each other. Walls often don't have sealed joists, and in warmer areas won't have insulation in between (aside for noise insulation), so air and smell will slowly go through, if strong enough.
Its also a pretty big problem in older wood properties in cities where bribing the inspector is common.