It's not that 3.5 mm jacks are perfect and impossible to beat. They just happen to be good enough for most people, usually only becoming unreliable after the device has reached the end of its useful life. In terms of reliability I'd put them about on par with Micro-USB jacks; those can also easily experience forces they weren't designed to handle and will then become unreliable. I don't know how much force Lightning jacks can take.
The main beef many people have is that Apple removed the 3.5 mm jack without supplying an adequate alternative. All options Apple has offered are problematic in some way:
Lightning headphones: Few manufacturers offer these so availability and choice are severely limited. It might be straight-up impossible to obtain Lightning headphones that have all desired qualities (form factor, sound, price etc.). These headphone are also incompatible with any non-Apple device. I didn't bother researching prices but I also suspect that Lightning headphones are a bit more expensive as far as the low-end market is concerned. Also, the only port capable of charging the host device is occupied, which is impractical when - for instance - using the phone while working in an area where it's easy to keep plugged in. Does not allow the host device to be connected to a car stereo that only has a 3.5 mm input port and no Bluetooth support.
Bluetooth headphones: Usually markedly more expensive than similarly capable regular headphones. Choice is limited, especially with in-ear monitors. Limited battery life can cause reliability issues and requires additional maintenance. RF interference and spectrum congestion can affect performance. Pairing might not always work well or might be lost during operation. Use of Bluetooth headphones drains the host device's battery faster than use of wired headphones. Sound quality is dependent on which audio profiles the host device and the headphones support. Also does not allow the host device to be connected to a car stereo that only has a 3.5 mm input port and no Bluetooth support.
3.5mm headphone adapter: Having a dongle attached to the host device makes it less handy and takes up additional space in one's pocket. One might accidentally unplug the dongle while pulling the device out of the pocket. Less control over how exactly the cable is positioned in the pocket (since many people are not going to keep it dongle-up) makes it easier to accidentally kink the cable. Having one more thing to lug around means one more thing that can be lost. Again the Lightning port is occupied.
The downsides of regular 3.5 mm wired headphones are well-known: The jack is moderately fragile and may become unreliable over time. Depending on the precise dimensions of the plug and jack the plug might become easily dislodged from the jack.
The 3.5 mm jack's problems are mainly of a reliability nature. The other options' problems are often about convenience and the inability to do things that used to be possible. There's the main beef: Apple's alternatives are all inconvenient to acquire and/or use or require workarounds to do things that used to be easy to do (such as plugging the device into a car stereo's aux port while powering the device from the cigarette lighter port). Unlike when Apple ditched the floppy drive for USB sticks or ADB for USB HID the change comes with a noticeable reduction in capability and ease of use. That is something worth complaining about in my opinion.