The real point is the tremendous risk his behavior created for the company. Diddling a subordinate alone creates a mega-millions liability. Another affair with an exec at a customer company is another liability.
The challenge, of course, is that when you're in that position, everyone at the company is your subordinate, so if you want a relationship, your choice are execs at other companies or a subordinate. The guy is married, so he shouldn't have had another relationship at all, but if we ignore that moral question for the moment and pretend that he was single at the time, you've just eliminated pretty much every kind of relationship he could realistically have, and that bothers me a bit.
IMO, when someone in a position of authority over someone does something like that, the right questions to ask are who initiated the relationship, whether the other person felt pressured to accept, whether it happened suddenly at the start of the job or after years of spending time together, whether he took steps to protect her from retaliation (e.g. talking to the board about it), etc.
Of course, whether women felt harassed or not, there's still a bright-line problem; he is married, and shouldn't have gotten involved with someone else. But that's a moral concern, not an ethical or legal one. Is that adequate reason to force him to step down? Depends on the company and the image they want to convey.
The real question in my mind is whether the people wanting him back are women or just men. If only men are complaining, then that means the women have concerns, and we should err on the side of caution and listen to the women. If both are complaining equiproportionally, then we should listen to them.
... with the caveat that they all just lost a lot of money, which might bias their attitudes more than just a bit, and those biases should also be taken into consideration.