Given Go is a mainstream language without anything unusual about it, and given that's pretty much well known, I'd say most programmers wouldn't consider it a barrier. The programmers that do? Probably the people who aren't going to contribute to an open source project in the first place.
Why do I say this? Well, because you either love programming or you don't. If you do, then yes, open source is interesting to you, and no, you're not going to be put off by having to use a language you're only 90% familiar with (because, like I said, for non-LISP/Prolog/etc programming languages, you're already 90% familiar with them), you'll consider that a feature, not a bug.
What might put a programmer off contributing to a project because of the language is if the language is unpleasant or a chore to use, not if the language is not something they've used before. But Go isn't that either.
I'm a developer too. I've been in this profession for nearly 25 years, and been programming since I was 10 years old. If something can be modified and the source is available, I tend to play with it, regardless of the language. I really suspect most of us are the same way. Those who aren't... well, do you think they're really interested in open source?