I've traveled to Asia 4 times and have visited areas that get very few foreigners. I get the stares, sometimes no one will sit next to me on the train, once I was even denied an advertisement being handed out on the street when the woman looked up and saw me.
People who are different attract attention. It's a natural human reaction. I've never been insulted (to my face) and I take as many precautions as I can to do what is socially acceptable.
P.S. - If you want a unique experience, be the only white guy at a Japanese onsen.