I've seen drivers mentioned a few times as an issue with USB serial adapters, but something I haven't seen mentioned yet is the fact that these drivers are usually (or always?) tied to the OS, which means the device can't be used until the OS is running and the driver is loaded.
From the client perspective this isn't a big deal. Eg., when using a USB serial adapter on a laptop to connect to a switch or modem, you're already going to have your OS up and running on the laptop. However, from the server perspective, this can be a major problem. If you use this on a server that, say, doesn't have a native serial port, nothing can be redirected until after the OS loads the appropriate USB driver. So, POST/BIOS messages, bootloader options, initial boot messages (eg., dmesg for Linux) are all unavailable. This severely limits the usefulness of USB serial adapters.
I think this problem needs to be resolved before USB can ever be considered a viable replacement, though personally I hope it does. It seems like it should certainly be possible; HID device (keyboard and mouse) support is offered by most BIOSes by enabling the "Legacy USB" support option, so obviously it's possible to talk to USB devices at this level. They just need to settle on a standard protocol for serial communication that can be implemented in a similar manner.