You're barking up the wrong tree. I don't think you remember how things were before PulseAudio.
PA was created to solve all this mess. PA basically handles everything and provides interfaces for everything, so finally pretty much all apps can talk whatever protocol they like, and work. And audio can be reconfigured as you plug and unplug devices.
Was it unreliable for a while? Yes. But there is still nothing better. The kernel doesn't mix audio. You need a daemon by design, and you need something PA-like to provide a modern level of functionality. The only way to do without PA is have the kernel implement all that, and as far as I know, the kernel devs don't want it.