That is just called annoyance, not health risk. The brief few seconds outdoor smell exposure you are talking about is probably something unmeasurably small, like 0.00000000001% the exposure of actual smoking. Once you walk into the building, you probably are breathing a zillion times more contaminates from paints, perfumes, plastics, carpets, wood preservatives, cleaning products, etc., continuously for many hours, day after day.
It's just an annoyance to you, perhaps, but a life-threatening health issue to me and a subset of people with serious respiratory illnesses like athsma and COPD. My airways constrict when exposed to smoke. Paints, perfumes, plastics, etc. are just fine - I have no problem with them. Smoke for some reason is a trigger and people like me are the reason smokers are exiled to the great outdoors. You're welcome.