So by analogy, if I every time I saw you I grabbed your crotch[1] you shouldn't get offended, because I never meant to offend you.
Probably not. I mean, it'd be a bit weird, but I'd set the boundary, explain that it made me uncomfortable, and expect it to not happen again. If it continued it would be something other than offense.
Honestly, I don't get offended very much because I consider "offense" to be intricately tied with intent. If someone spat at me or flipped me off or something, I'd take it as it was intended. There's lots of explicit ways various cultures have to indicate "I intend to offend you", and in my culture, those are two pretty good ones. But other cultures have different ones, and they generally don't apply to outsiders. A Frenchman wouldn't be offended if you shied away from a "faire la bise", unless it was your cultural norm as well. The Japanese have a very complex business card etiquette. Someone doing business in Japan regularly should make a point of getting to know this, but a one-off instance of a Japanese person interacting with someone from a different culture shouldn't be offended if somebody doesn't realize that just taking the card and pocketing it after a glance is considered offensive. Indian culture has a thing about shaking with the left hand since that's the ... "wiping hand" (gotta do something if you have no toilet paper). These are generally amusing tidbits shared later (by either party), not tense moments.
It's hard enough to get offense straight when it's entire cultures - when it's random individual's sensitivities it becomes impossible. I will behave in a way I believe to be appropriate - if it offends you, the onus is on you to realize that I didn't mean to offend, because that's how interpersonal relationships work. I will make a good faith effort to avoid doing that in the future - if it's reasonably accommodable - to avoid repeating it with you, but I expect you to forgive my lapses since this stuff is hard (especially for me).
Not to put too fine a point on it, but there's a lot of people who can't tolerate other people behaving in ways contrary to their exact wishes. We generally call them "two year olds".
I can't imagine being offended more than a few times a year. It sounds exhausting - that is, with my definition of offense. But since you're explicitly saying that intent doesn't count, clearly we disagree. I'd call your definition more like "mild irritation". And I don't think I can (and have so far failed to) really interact with you, or people like you - I mean that 100% seriously. I don't mean to presume to tell you when you or anybody else should be offended, but I think you, bible-thumpers w.r.t evolution, and the anti-gays have a lot in common when it comes to offense thresholds...