I draw the line at mechanical ability. If video games required none, you or I could probably compete just as well as anyone else on the stage, but that's clearly not the case, especially if you've ever played some of these games. Just because these individuals have limited their athletic prowess to the hands does not put it in the same realm as chess even though it's closer to it than it is to rugby.
To further make the point, are bowling, golf, darts, billiards, or auto racing sports? None of them require much in the way of athleticism, yet they are all considered sports and have professionals who can make a living engaging in them. Which of those, if any, are also on the chopping block?