I liken it to motor sports. Its one level of abstraction from direct physical involvement like most of the major sports. Driver masters the skill of driving which is used to control a car for racing instead of racing themselves. Players control an avatar for competing instead of themselves directly. I find eSports intrigueing but driving to be insanely boring so your mileage may vary.
I'd argue that level of gaming definitely requires a type of physical skill, though it'd be closer to badminton or chess than that of driving. Its about reaction time, quick twitch reflexes, and teamwork/strategy.
I agree on the classification though, and think the term sport is the root of the issue. What makes these things exciting isn't usually the sport action itself, but the fact that the sport/league is set up in a way that enables competition. These are a competition, as is poker, diving, football, and hell even debate, and should be called competition. I think the use of sport in its place is simply because competition is a mouthful and most competitions are sports.