There have been a lot of moving pieces to the Firefox project.
* First, the code base was meant to be universal, with options to support a range of OSes and hardware. This involves a lot of not-so-sexy problems like word size, address space, endian-ness, etc. that don't come up a lot anymore. It also has it's own home-made (pseudo)thread system, because some of those supported OSes did not have nativemultithread capabilities. Since this is at the base of all code, things higher up have a lot of clunky support code.
* To make this all work, everything is modular with defined interfaces. This takes more memory, but makes for a more stable yet versatile design.
* Building on that, since this chrome layer was never hard-coded anyway, it makes it easy to re-write with extension code. In fact, the extensions can do damn near anything by interfacing with gecko modules directly (or by building their own).
Now the problems:
* The Gecko engine is basically functional, but was never designed to do a lot of the goofy crap modern Web API standards are pushing forward. (Web API is basically aiming to make your browser into a universal interface for your entire computer. That way "web apps" have access to all the resources they could possibly need... though, sadly, almost no thought seems to have been given to making it secure.)
* Gecko is also way too bulky to ever work on a mobile phone, which is how a lot of the web is being consumed now. They had to fork and rewrite code to get out a workable android version.
* The extensions that did more than change the display meant that you either couldn't change internal interfaces, or that doing so would break everything. The way the chrome code was written meant that changing it would often break theme add-ons. Most of the time this just leads to angry users and people who refuse to update. But it also leads to a lot of bugs filed and support requests that ultimately stem from badly-coded, out-of-date extensions screwing everything up. Mozilla mostly considers this a perception problem, since the browser seems to be more broken because of the extensions.
* Firefox had a lot of preferences. These could all be edited at runtime, and took up a lot of memory with the properties themselves, and with the branch logic needed to support some of them.
* Firefox has been very leaky on a per-tab memory basis.
So where have they been going:
* legacy OSes and architectures aren't supported. Not enough horsepower to actually make the modern web work on, say, BeOS.
* Mozilla threads are deprecated in favor of real multithreading done through OS API calls.
* They've gone to a Google Chrome-like model for extensions, where there's a general extension API. They're open to expanding it however is necessary to get extensions with large market share -- such as NoScript and Firebug -- to work, but at some point we can expect that there are things they just have no interest in adding in.