If you have term limits, there's another problem: since the candidate is not going to get elected again, then
1) In his last term, he won't care if he's acting against his constituents' wishes, since he'll never get elected again anyway.
2) In his last term, he'll pander to special interests. This is enabled by point 1) but is also a separate problem, since he'll want to put food on the table once he leaves office and doing so will get the special interest to hire him.
Of course, you can fix point 2 by making it illegal for him to be hired by anyone he affected during his term. For any significant position this ensures that only the rich ever take office, which is its own problem.