Xenix was $500, which was the standard price for major software back in the day. That's what WordPerfect cost, etc. Yes that was insanely expensive in inflation-adjusted dollars however the productivity gains over a typewriter made it absolutely worth it. And Xenix was super cheap (comparatively) because Microsoft did a bulk buy from AT&T which drove the cost of their licenses down. AT&T was not a monolith trying to maximize profit across the whole corporation - the licensing division at AT&T did not want the hassle of selling jelly-bean software so they set the price high. Xenix was pretty popular for what it was -- common office applications were ported over Xenix and you can still run those Xenix applications on Linux today (in theory -- I am sure the iBCS code in Linux does not get the love it did 20 years ago).
Minix was never a commercial operating system, though very late in its life it did get used for some specific use cases such as the Intel Management Engine. One reason Minix wasn't popular in the 1980s was because it wasn't licensed to be. Two, because it was designed to replicate Seventh Edition Unix for educational purposes. If you've ever tried to port over software to Minix you've found very quickly that calls which were quite standard and expected by the late 1980s (e.g. select, or named pipes, sockets, or the curses library) were not implemented at all, or were implemented in a perfunctory way which didn't match the contemporary Unix API.
I don't know that Coherent ever had much in the way of applications, you'd really be relying on it being compatible with other unices.
I think the bigger reason Unix never took off in the PC mass market is that there was no reason for it. In law firms, there is an obvious advantage in document sharing and centralizing backup and version management, and it was easy to quantify productivity gains that justified the cost. (I.e., lawyers billed by the hour, but secretary cost was covered by the lawyers' billable hours.) For most users, this was not the case -- if you needed a heavy duty Unix system for CAD or whatever that's what you'd buy. Otherwise MS-DOS did the job. There were always tons of "make the PC more industrial-grade" products like DesqView but the reality is that very few people needed that stuff until Microsoft brought it to the masses with Windows.