I think it is very simple to replace service jobs with robots. And this is happening a quite a fast pace.
In my country (Netherlands), nearly all banks have gone entirely online. They are not just "focusing more on the Internet", no, they have just closed nearly all their offices. Paying taxes has been completely automated for the vast majority. Offline retail is having more and more trouble keeping up with online retail (which, in many cases, is handled almost entirely by robots!). It has been ages since I've been in a real shop! Train personnel is being replaced by access gates and security cameras at a rapid pace, even the checkout employees at supermarket are mostly a thing of the past; customers just scan their groceries themselves. In the medium term, we can also expect all transport personnel (taxis, truck drivers) to be replaced and then it's just a matter of not too much time before the entire process from growing food on the land to it ending up in your fridge is handled by robots.
In fact (again, in my country) in the medium and long term, it does not really look like we're transitioning to a service economy at all. We already did that in the 50s to 80s and are now transitioning into the next phase. This can be seen very clearly during the past 30 years: employment in nearly all professions is declining, except for IT, recycling, sales, care, medical care and recreation (those are the main categories used by our national numbers-agency). In other words: apart from some jobs that are on the rise because they're just new, we're quite quickly moving to those jobs that cannot be replaced by robots.
I think this means the service economy will very soon be a thing of the past. Instead, we're moving towards a "care&joy economy", in which most jobs are about the one thing robots cannot do (very well): being humans.
Note that in the long run, employment rates have been pretty constant. I - like you - am not worried that robots will cause major unemployment; no other type of automation has done so in the past. What I am worried about, is that robots will enable further concentration of wealth on a level and at a speed never seen before. This old-fashioned problem, once popularized by one Karl Marx in the time of steam-engines is even now already gaining major traction. Economic and social inequality are increasing rapidly and will continue to do so.
This worries me because, contrary to popular believe, economic and social equality are the best guarantee for economic growth; the healthiest (but not necessarily the biggest) economies are characterized by notably high equality amongst their participants. (Also see http://www.ted.com/talks/richard_wilkinson.html)
Robots won't take our jobs. However, their owners will increasingly not (be able to) spend their (ever growing stack of) money in ways that keep the economy up to speed. If we don't solve this problem before it occurs, it will be the end of western society as we know it.