Ah, I know this one. In fact, you want the negotiations to be in your native language, and make the other party use their non-native language. This puts you at an advantage.
If they're calling you a dumbfuck to your face, you need to drop them. If you're suspicious, bring in a speaker of their language and tell them he's part of your team. He stays silent the whole time and listens. He'll tell you what they're saying. Again, advantage: yours.