(See? I used per se, so I'm... oh never mind...)
Intelligence and being highly observant are great skills both in society and from an evolutionary/survivalist standpoint.
But in a society I've found it brings up two downsides:
Guilt, because your intelligence allows you to avoid pain or achieve a higher level of comfort in society. You weren't "superman" you just made rational choices based upon your understanding of how the system works and now your friends and family are suffering because they didn't and you want to help them which requires more energy and effort or you can't which means your intelligence has limits and all you can do is watch them suffer.
Stress and anxiety. Once you figure out that you can problem solve and improve your quality of life it's natural, like any athlete, to grow and push your boundaries. But intellectual pursuits aren't as cut and dried as physical ones - It's easy to know that you can only bench press 200lbs and that's what you need to work on - Less so when you're trying to solve problems like familial and social discord but nobody will listen or trying to improve your company's fortunes by making proper investment choices. More to the point, I'm an engineer and there's nothing more frustrating trying to solve a problem you've encountered with your design that YOU pushed for, can't figure out why it's not working, might not work AT ALL and the boss is breathing down your neck (oh and the company is on the line). There's plenty of days I've driven by a building crew and daydreamed about just running the earth mover or driving a dump truck.
In an Agrarian society - in a pre-industrialized world these issues just didn't come about for intellectualism - Partially because it wasn't as much of a survival skill. (And that's probably why steampunk is so romanticized today)