To my dispointment, a number of my live sound colleagues have blown up iphones by plugging in via mini jack to XLR, and forgetting to switch of phantom power. They should know better, luckily they play dumb and tell Apple it just stopped working, and get replacements.
That should be the least of their worries. Plugging pre-amped headphone level output into mic level inputs isn't good for the mixer either. At least use a passive direct box that can match impedance and attenuate the signal. Better ones do re-amp, balun and 48V stripping, but even a $25 direct box is better than blowing your mackie/motu/whatever, and having to software gate a horribly clipped signal.
Fortunately, most mixers made after the 1950s have these magic devices called "Input Gain", which can match the sensitivity of the input channel to an extremely wide range of signal levels, including so-called "line levels". Portable device headphone outputs can almost always be adjusted to be well within this range. So, unless you ignore that red "clipping" indicator on your input-strip, signal level mismatch isn't a problem. And if you're sourcing from a phone, any impedance mismatch (which probably doesn't matter anyway) is not going to be the limiting factor as far as "fidelity".
As far as the phantom power goes, well, that's hardly the phone's fault, now is it? Don't use cheap boards with "global" phantom power!