I don't think the xServe and xSan sold very well. After all, had they been runaway successes, I doubt Apple would have discontinued them. No public corporation is going to throw away easy money "just because". If the server products were really thriving, and Apple *really* wanted to just be all about consumer hardware; they'd have just spun the business off into a subsidiary, like they have in the past with FileMaker and Claris.
And really, Mac OS X doesn't make much sense as a server anyway, Unix underpinnings and their token OS X Server development notwithstanding. The big advantage OS X has over other Unix/Linux OSs is a GUI that doesn't suck and the availability of a good number of commercial consumer software products. Neither is relevant to a server. And a GUI and most of the other OS X features are a liability for a server. A server should have exactly and only the packages it needs to perform its specific job, and nothing more. That's not Mac OS X.
Perhaps Apple could publish some RPMs to be installed on RHEL, CentOS, or Amazon Linux with customized services targeted at the "classroom full of Macs" they used to cite as the use case for OS X Server... CalDav, CardDav, Wiki Server, Open Directory with non-sucky directory templates, and so on. Combine this with an easy-to-use client that'd run on a Mac to manage these services and you have a server offering that makes sense for Apple. But dedicated Mac server hardware always was a really odd duck out and never made a whole lot of sense. And I say that as someone who likes Apple, owns many of their products, and practically grew up with their computers. All the way back to that first Apple ][+ I can hardly recall a time when I did *not* have Apple kit. But for a server in any kind of production environment? As much as I like Apple and hate, for example, Dell; I'd choose CentOS on a pair os R220s (for redundancy) over Apple's offerings... even the xServe when it was available... every time.
Add to all that the fact that many businesses these days... especially the startups that are open to Apple products in general (It's not like you'll see many Macs in an SAP building or on a Lockheed Martin campus.)... don't even want to own their own rack mounts or datacenter space in the first place; but would prefer to to just spin up infrastructure in AWS and avoid the headaches of owning and maintaining hardware. Why would Apple... or anyone, really... want to jump into the 19" pizza box market now?