The two major parties used to be collections of disparate interests that came together to win elections, rather than classical left/right-wingers. If you nibble around the edges you can find that this still holds true with a few issues (abortion and the 2nd Amendment are good examples), but by and large the trend has been for the elected officials in each party to pander to the left or right on all the issues, lest they alienate the primary electorate and find themselves out of a job.
Primary Elections are dominated by the true believers, while General Elections require massive amounts of cash, cash the candidates are most likely to obtain from the aforementioned true believers, so is it any surprise that they cater to them once they win office? Rinse and repeat, election after election.
Joe Q. Public, i.e., the "most" people you're referring to, doesn't religiously follow politics. He doesn't watch MSNBC or Fox, or read Daily Kos or Red State. He has a belief structure that might favor one party over the other, but he doesn't tune into the process until a few weeks before the General Election, at which point it's really a beauty contest in the vast majority of House districts. Joe Q. Public essentially picks the side that he thinks is going to do the least damage, then prays the pendulum doesn't whack him in the head as it swings from left to right and back again.