I love C#. I program in it every day. It's plenty fast, and it's a great language.
However, there are two reasons I would suggest looking to another language.
First, the hottest market for gaming right now is mobile. While it's possible to compile C# for iPhone or Android using Xamarin (along with Windows and OS X), it's not exactly a native experience.
Second, C# (like O-C, C++, etc.) is a general programming language -- it's not in any way specific to the domain of game programming. So, while it's *possible* to design complex games in any modern language, you're probably going to spend *way* too much time dealing with silly stuff like tracking graphics resources and animation loops and simulated physics. You'll have a higher chance of success if you use a language and platform that is more game-specific out of the box.
I would suggest looking into Swift -- it'll give you access to the lucrative iOS market, it's C-like, and it has features that are game-specific. Sure, it's a new language it doesn't compile to Android, but by all accounts it looks like a great language with first-class support for gaming, so you can focus less on infrastructure code and more on the game.
Another option would be HTML5. Depending what sort of game you're looking to build, Javascript and HTML5 may be just the ticket, and there are a number of libraries that can abstract away browser differences and assist with the plumbing needed to make a game run.