Full disclosure: I work for NVIDIA on cloud gaming.
I was as skeptical as you about the latency. In this interview. Phil Eisler talks about 200ms of XBox + TV latency that people live with every day. (See page 2) If that's our target, then that's pretty doable, since with strategically located data centers you can get the network latency down to 20-30 ms.
In the work we're doing, we're actually focusing more on hitching in the game than latency, since the latency isn't as big a deal if you're say in the Bay Area where one of our test clouds is. Heck, I played Trine2 from the east coast and it was very playable. I wouldn't play BF4 across the country, of course. :)
There are plenty of other risks with the idea, but I wouldn't put latency at the top of the list.