Sure you do. In real life, I interact with four sets of people:
1. My friends, who I get to choose, subject only to the limitation that they also have to choose me as a friend.
2. My coworkers, which I don't totally get to choose, but I can choose a profession that is inaccessible to people who aren't above average.
3. People I exchange goods and services with, like retail cashiers, repairmen, waitstaff, etc.. They can run the full gamut, but I honestly don't have to spend a lot of time with them. Unlike school, where you had to spend ~30ish hours a week with the same people whether they are jackasses or angels.
And with things like amazon.com and slef-checkout at grocery stores, even this level of interaction is being reduced to mostly skilled tradesmen or airport security.
4. My family, who I didn't choose (except for a hypothetical future spouse, or technically if I adopted a kid I guess).
Basically every other interaction is either an exception (eg. random police stop, construction workers redirecting me because they're busy fucking up the sidewalk I need to walk on for whatever reason) or by my own choice.
Besides which, the social skills to talk to smart people aren't completely independent of the social skills to talk to not-so-smart people, honestly. There are some differences, but smart people just aren't that special.