I take issue with your seemingly contradictory points. At first you say "some 'accents' are simply lazy nonsense" and then you answer yourself by saying "different people hear things in different ways," which mostly boils down to your first language. I'll concede one argument to you and say that, as someone who at least tries to mimic the "accent" of the original language when speaking in tongues, I don't understand the math teacher I had one year in college. He insisted on saying the word "path" as "pass" every single time. As this was a class in graph theory, the word path came up a lot. :(
This whole thing reminds me of a class in linguistics I took. The teacher showed a video of a bunch of Americans in a submarine that was sinking. At the end of the video, the German dude on the other end of the emergency communication channel said, "So... what are you sinking about?" German lacks the "th" sound that English has and to the German speaker, the "th" phoneme translates to the nearest sound, "s," turning the word into "thinking" in his head, but it comes out as "sinking."
Your original language controls very heavily how your perception works. There was another example of a tribe of people somewhere that count up to "many." They had 1, 2, 3 4, 5 or something and many. You show them 10 or 100 and they simply call it "many." Their notion of counting ends at "many."
As someone who had a hard time in college with some accents, I really don't see this is as wrong when communication is a vital part of the job description, such as teaching or being a call agency of some sort. If you settle on a language, you need to have an accent that at least somewhat resembled the typical speaker. There are horror stories across the US, I'm sure, of colleges with student teachers with accents so thick you might as well not go to class. (And don't get me fucking started on students that ask said teachers in their native language and then receive an answer in that same language, without them ever explaining what the fucking question or answer was.) If you're saying "pass" and you mean "path," maybe it's time to get some accent training.
English has a wide variety of sounds but if someone came up to you and started talking in a language that uses clicks (they do exist), you would have an incredibly difficult time distinguishing one click from another, even though to the native speaker, they are the difference between "path" and "pass."