I love C#. I program in it every day. It's plenty fast, and it's a great language.
However, there are two reasons I would suggest looking to another language.
First, the hottest market for gaming right now is mobile. While it's possible to compile C# for iPhone or Android using Xamarin (along with Windows and OS X), it's not exactly a native experience.
Second, C# (like O-C, C++, etc.) is a general programming language -- it's not in any way specific to the domain of game programming. So, while it's *possible* to design complex games in any modern language, you're probably going to spend *way* too much time dealing with silly stuff like tracking graphics resources and animation loops and simulated physics. You'll have a higher chance of success if you use a language and platform that is more game-specific out of the box.
I would suggest looking into Swift -- it'll give you access to the lucrative iOS market, it's C-like, and it has features that are game-specific. Sure, it's a new language it doesn't compile to Android, but by all accounts it looks like a great language with first-class support for gaming, so you can focus less on infrastructure code and more on the game.