The problem with an external adapter, and with lightning connector headphones, is that the DAC has now been moved from inside the phone to the external dongle or headphone.
For audiophiles, this was standard practice (to use an external DAC) even with the 3.5mm jack, because the iPhone's internal DAC was limited to 24-bit/48kHz. But now with cheap dongles and cheap headphones, the built-in DACs could be quite inferior for audio quality, or if decent, raise the cost (rather than one decent DAC in the phone, now every connected headphone/dongle has to provide it's own DAC and draw power from the phone).
I don't like Apple removing the 3.5mm jack, but now that I use Sennheiser's as my primary headphones at work, I no longer use the 3.5mm audio jack and use an external dock to charge my phone and drive my headphones (still have to turn the volume to near max to drive them). I think the market for external DACs with added bonus of phone battery charging capability will be on the rise (see Oppo HA-2 and the like). Incorporating such features into a phone case even better.