I'm sorry, and I mean no offense, but that's not evidence. The problem with parents who tell these tales about how peanuts are like kryptonite to their kids or they're allergic to X in food is also he reason why we shouldn't base public policy on anecdotal evidence (there's another comment below about someone "who knows a family with a son who...")--so please don't take this as if I'm targeting you specifically or questioning he veracity of what you're relating; I'm just pointing that this is isn't how we gather evidence on public health issues and the stories told by parents shouldn't form the basis of public health policies.
The thing is, in the scenarios you're describing, you have a son who is quite allergic to nuts, I'm going to guess because he had something with peanuts actually in it at some point, or came into contact with the oil, and after that happened a couple of times with an allergic reaction, you figured out he was allergic. And people at the school and around him basically know this, too.
So now, when your son doesn't feel well, on a field trip, or at school, everyone looks around for the nuts. And lo and behold, you're next to a peanut farm. Or a kid at the table is having a PB&J. Or you find out his playmate had peanut butter pancakes that morning, or a snack made in a facility processing pine nuts. Or whatever. And you have your "explanation."
Except that you don't actually know how frequently your son is exposed to "peanut dust" or "contaminated surfaces" or whatever, and doesn't have a reaction. Maybe he's allergic to something else, or maybe not. Or maybe it goes down exactly as you suspect. The problem is that in the absence of a controlled study, we just can't tell. And while it makes sense (maybe) for you to just be on the safe side with regard to nuts, it doesn't make sense to make rules, regulations and laws with significant costs for others without that peer-reviewed, study-based justification.
Anyway, I hope people take this as the call for more information and for better study of the public health implications of allergies that it is, and not as an attack on a dad and his son, which it certainly isn't intended to be.