It really depends on the load.
I don't see them taking over totally anytime soon but for some tasks they will be good enough.
sql back end of any kind forget about it, not even worth discussing anytime some.
They could need hardware accelerators to even be really useful on anything but the lightest loads.
Even middle-ware processing, not soon.
This is an area where accelerators could shine.
Front end web servers, This is the place they will show up and shine.
A lot of power is wasted waiting to serve a heavy load that only comes a few hours a day.
The small standby power of ARM is a good start. Get the i/o up and add some accelerators.
Hook them up to a fast fabric and go.
File servers are a real possibility many NAS already are running them, but they don't scale yet.
They do have the potential to do so however if the r&d is spent on them.
Many raid controllers are nothing but lower end arm cores running firmware, put 5 or 6 of these on a single chip with some fast 64 bit cores to run the OS and a 10GB network controller on a fast fabric and you would have the basis of a killer server.
Routers, many already are ARM and they will scale farther up the food chain in the future.