ARM are low performance chips, that's why they have the room to improve. Intel server chips crush ARM in performance. Let's take a typical hyperconverged server my employer uses with 8 six core Xeon processors, how many of your USB powered ARM servers will it take to equal that? A row of racks?
The question is though....in some situations....is it better to have a monster multi xeon machine that sits there idling 98% of the time vs an ARM based server that may only be idle 10% of the time while consuming a hundreth of the power that the xeon system does?
Both the multi xeon machine, and the ARM machine have their purposes in the server area. It's about picking the best tool for the job. One size doesn't fit all.