Adults and older kids enjoy those games, sometimes because we appreciate the realism of the simulation, or because we appreciate the ways in which the simulation defies our notions about reality (you can kill without repercussions, or you can crash a car and not die, or you can jump and then start flying.)
Once a kid is capable of assembling his own Lego block creations, inserting pentagonal pegs in pentagonal holes and stacking unstable wobbly objects in perfect balance, any additional fine motor skill or coordination is not going to provide that much more of a real life advantage, IMHO, and is taking away from a whole bunch of other skills that they also need, like how to treat people and animals for mutual benefit (empathy), and how much they can get away with before they get yelled at (what's appropriate behavior according to their parents).
So I would limit video game playing of any child under 3 to a minimum. And above 3 then the freedom is much greater.
In my opinion, I'd say before age 3, no VR type games for my son and after that age, it would be depending on how much exercise and other activities he is taking part of.
With that said I worry more about my son playing *too much* with model trains. And it just goes to show that children at early ages (especially boys, but some girls too) will always have strange little obsessions that always seem to detract from getting a well rounded education, whether it is telephones, video games, remote controls, treating inanimate objects like their "babies" and so on.
It leaves a parent wondering what is the cause and what is the effect, are kids who love video games "born" with an interest to sedentary but mentally stimulating activities and are they just playing video games as a consequence? will they start obsessing about telephones or counting things if you turn off the TV, yet another sedentary but mentally stimulating activity? There is a little bit of truth to that you know...