There are biological foundations to this issue. I used to work with researchers at the Yerkes Primate Center who were working on this very problem. The idea is that males want to ensure the paternity of their children if they are going to expend resources raising them. So they are protective of their sex partner. Females want to ensure that they have the help they need in providing for their offspring, so they desire a faithful partner. These needs drive the species toward monogamy.
At the same time, females want to enhance the odds of successful offspring by having more than one mating partner. (Partners who would be notoriously bad as long term spouses are particularly attractive ... the bad-boy alpha male) Males also want to enhance their chances of having successful offspring by having partners that do not require them to stay around and provide for the kids. So both genders have an incentive to secretly violate the monogamous bond.
Therefore you see a continuum of activities along these lines within and among human societies. One of the researchers reported that as many as 1/3 to 1/2 of all children in the pre-industrialized societies she examined were the result of illicit affairs. Most of these societies frowned on infidelity much more than we in the west do.
Enter game theory and genetics, which argue for a balancing act between the two competing needs, with different people taking different strategies and feeling different drives. Another researcher at Yerkes gave a talk about the "seven year itch". She had evidence that suggested a biological basis for the lagging emotions of marriages a few years along - with parallel evidence from other species. You know that "he's just not romantic any more" trope that is trotted out to explain a flagging libido and attraction to other partners? She had a theory that this was an instinctual result of changing hormones affecting the brain. The end result was to drive a woman to seek out other sex partners in order to ensure genetic diversity in her offspring.
So the answer is yes, it is complicated.
BTW, I'm in your camp. I would never consider cheating, and have a few decades of experience to back up that characterization of my own proclivities. But I do recognize that this is not everyone's experience. And I've been close enough to a few people who took a different path to know that it isn't just culture or upbringing that makes for fidelity.