I work for a non-profit that does, among other things, employment counselling. As such, most of our staff members are extroverts and very touchy-feely (which is good in their positions). However, as the lone IT person and an introvert, I'm not a big fan of team building activities and what some have described as 'emotional circle-jerks'. I'm a fairly humble person, and get easily embarrassed at employee recognition events, and as such tend to stay away from them. I'm quite happy to do my job and interact with the rest of the staff as I choose - I'm very fortunate that our management doesn't require attendance in most of these activities.
So when it comes to lunches, I tend to do the same. I'll often just eat at my desk as I work, or maybe go out for lunch with my wife or a friend with our schedules happen to line up. If I'm feeling more social than usual, I might take lunch with other staff members, but those days are few and far between.