I'm well-aware of the advantages of SmartOS, actually. I am in the process, however, of migrating my dedicated server from one system to another (I'm upgrading the hardware, while staying with the same hosting provider). In doing so, I've made the difficult decision to move *away* from SmartOS, and back to GNU/Linux, for the following reasons:
(1) Despite promises to the contrary, compiling most C/C++ FOSS is *not* easy on SmartOS. Also despite promises to the contrary, a vast amount of FOSS that I need is *not* available in SmartOS's repositories, nor in any SmartOS equivalent of Ubuntu PPAs. 99% of projects need extensive source-level patching, awful environment variable hacks, symlinked binaries, edited configure scripts, or some nasty combination of the above. Some extremely complicated projects like PhantomJS are nearly impossible to compile on a system that is not the Linux kernel, with glibc as the C library, libstdc++ as the C++ library, and gcc 4.x as the compiler. I cannot use any alternative programs for some of these use cases, though, and PhantomJS isn't the only one -- I simply don't have the time or willpower to spend weeks fighting horrendous build environments that are opaque to diagnosis to do something that I could accomplish with "aptitude install phantomjs" on Debian or Ubuntu or Devuan or Mint or... you get the point.
(2) Although the kernel and core system was sound, I experienced inexplicable random crashes of a 4 GiB kvm guest running Windows Server 2012 on SmartOS. I tried fixing this numerous ways by updating the host OS, updating kvm, looking at logs, reducing the amount of RAM assigned to it, etc. -- but after about 24 hours of uptime, the VM just crashes (on the host side). I don't experience this with KVM on the Linux kernel. And for various reasons I can't *not* have a Windows VM for certain limited use cases on my server. It's a multipurpose box and it needs to be able to do a lot of different things. I don't have the money to buy a dozen different boxes each filling its own little niche.
(3) To run the aforementioned programs that are infinitely resistant to compiling cleanly on SmartOS, I ended up firing up a paravirtualized Linux kernel (CentOS 7) on top of SmartOS. This ran well enough, but it just felt *unclean* to need to run a UNIX on top of a UNIX, when all I'm doing is running FOSS programs. Although there is that one program that's binary-only which absolutely *does* require Linux...
(4) I tried messing with lx branded zones, but could never get it to actually do anything useful except print error messages. I of course googled those error messages, asked about them in IRC, and the like; but the most I could get was someone saying "Huh... that's strange." No offered solution. lx branded zones have a lot of promise if they can emulate an actually modern GNU/Linux distro such as CentOS 7 or Ubuntu 14.04.2 or Debian Jessie, but until/unless they can do that, with few or no gotchas, I really can't be bothered to mess with them in an alpha/experimental state.
(5) lxd to the rescue! Canonical (officially the "Linux Containers project") is working on a daemon called lxd ("lex-dee") which brings sanity and proper isolation (through use of already-existing kernel resources) to the lxc project. The `lxc` command that comes with `lxd` operates very similarly to `vmadm` in SmartOS, and the guarantees that you expect for isolation in SmartOS are pretty much true in lxd guests as well. At their core they just use mainline Linux kernel features; the difference is that lxd actually uses these facilities *in a smart way* to isolate guests. Docker on the other hand, encourages the guests to be friends with one another. Yuck.
- I had problems with SmartOS in actually using it, despite a promising and rock-solid stable base system. It needs a lot (and I mean a LOT) of work for compatibility with existing FOSS and/or better support for lx zones based on modern (recently-released) Linux distros.
- I still have ZFS on Linux and it works great.
- KVM isn't broken on Linux like it is on SmartOS, so I can have my Windows VM and not constantly have to `vmadm start $WINDOZE` when it crashes.
- Ubuntu's lxd is a viable isolated container solution for me on Linux, using the mainline Linux kernel.
- By using native Linux, I don't have to virtualize a Linux kernel, so any software I want to run that happens to depend heavily on Linux will "just work", and I likely won't even have to compile it thanks to everything under the sun being in a PPA.
- Rebootless kernel updates :D
P.S. - my use cases for this server are varied and diverse, from gaming to music streaming to file hosting to VPN to IDS to .... well, you name it. "But it's got nginx in the package repo!" isn't enough package support for me.
Sorry, SmartOS, but it's not ready to meet my needs, and Ubuntu Server 14.04.2 definitely is.