I agree completely, even though I'm part of the group for whom the browser is much more toy than work. Here are some pet peeves I've had with Firefox:
1) inconsistent UI design across platforms.
The browser was supposed to be cross-platform. I've found that phrase is quite broad. That can mean it compiles in different environments, that the code is general enough that it can be compiled to run on different hardware, that a version can be run on any major operating system, and that the user experience will be consistent across operating systems. They did a really good job with the first three points, and dropped the ball on the last. At one point I had similar versions of Firefox on Windows, Mac, and Ubuntu boxes, and all were slightly different. They were trying to conform to user expectations for each given system, but it made Firefox a headache for people who moved between systems. The good thing about browsers is that they're your gateway to the internet, and the good thing about the internet is it's as platform agnostic as anything we've ever made. It's probably only irritating to a few people like me, but it's seems pointless that the UI for Ubuntu would have different colors, different icons, different menus, and different text for the same piece of software. Is there really any point to giving "options" to Windows users, but "preferences" to Ubuntu users?
2) inconsistent UI design across versions on the same platform
Every time a major update came out, they changed the icon set, and changed enough of the underlying CSS markup for the interface to break a good number of extensions. Most of the time, the changes were so superficial that there was no reason for them to be made. Later changes were more dramatic, like the star system for bookmarks that they cribbed from Flock, but those also tended to break other things than display. More than anything, it tended to piss a lot of us off.
3) Feature creep/Bloat
Firefox started to pick up a lot of features that just don't need to be in most people's browser. For example, the Google-driven anti-phishing whitelist and/or URL-checking service. Sure, you could disable these things with preferences, but the code is still there, the CPU cycles still need to be spent to check the preferences, and you still have to take the time to lookup and disable them. Pre-fetching is another feature of dubious value. Other features, like "ping" died on the vine. (For those who don't remember, Mozilla argued that the ping feature was valuable because advertisers were going to track users no matter what. For the good of the internet, the ping feature would allow webpages to embed a silent callback address, allowing advertisers to track us in the most efficient, low-traffic way possible. Isn't it great how Mozilla looks out for us?)
There are more services now than I care to keep track of. As an experiment, go to about:config and filter based on "http". Did you know that the browser may interact with all of those sites? Do you know WHEN it interacts with all of them?
Most of the services they add are unnecessary for the majority of users. The extensions system is the crown jewel of Firefox, and there's no reason to bless dubious features like safebrowsing and force them on everyone, while very popular extensions can be broken at the drop of a hat. Firefox should have official extensions, and removing dubious features should be as easy as disabling an unwanted extension. Most AV systems already interface with Firefox this way.
4) No respect for add-ons
The typical open-source movement "f*** you" is "if you don't like it, why don't you change it yourself." It's somehow reasonable for everyone to learn to code in addition to their real job, rather than ask the people whose real job IS coding (most Mozilla contributors are paid) to justify themselves. Also, ignore the fact that Firefox is a huge moving target, that addons theoretically have to be updated as often as the browser, and that developers have to support multiple firefox versions (because they can't give half their customers the middle finger with every new release like Mozilla can).
But it's more than just their tendency to break addons: addons are the best thing about Firefox, the ONLY distinguishing element, and they never seem to acknowledge that, much less make use of it.
For example, each new version and system had different, slightly tweaked default themes, but you can't find any of those themes in stand-alone form. You can't easily carry your version 3 theme forward to version 7, or get an "Ubuntu feel" on your Windows install. They keep adding new niche features to try to distinguish themselves, and make the entire codespace larger for everyone. But an extension mechanism exists SPECIFICALLY to let developers built niche features without bogging down the code. Extensions are good enough for mere mortals, but not the Mozilla developer elite. It might be good enough, of course, if they could make the extension system more stable. As it is, "turn off extensions" is the best diagnostic tool they have, and hardening the system to guarantee extension stability is more work than they actually want to do.
These are all philosophical problems of what the browser should be doing. This doesn't even touch on how it should do what it does WELL. Memory management, threading, and sandboxing are all important. But they're all under the hood. So long as they're done right, the user doesn't even know anything's being done at all. Getting Firefox to a level where that's true is important, but it won't fix the bigger issues that drive most people away.