It _is_ a problem on your average desktop. In fact, it's a problem on your high-end desktop.
I switched to Chrome, as of Firefox 9. I worked at Netscape in the 90s, so it was not a switch I made lightly, due to my Mozilla loyalty. The difference between the two (on multiple machines, across multiple versions, with multiple operating systems) is that on a system with 12gb of RAM and one to two dozen tabs (just averaging, here), Firefox grinds to a halt. It starts hanging/beach-balling. Eventually, it gets to the point where every action you take results in the OS saying the application isn't responding for ten to sixty seconds. Click a tab, hangs. Scroll. Hangs. Type in a text box. Hangs. Type in a new URL? Hangs. Click on a link? Hangs. Same type of surfing and extensions on Chrome? Never hangs. Never crashes.
All the "we don't have memory leaks" or "we have memory leaks, but *here is why*" or "well, it's because of evil extensions!" in the world is irrelevant, when _it_just_works_ for your competitor. Whatever the explanation or justification for Firefox's experienced problems, the counter is that _Chrome_works_.
I figure I gave Firefox enough time to get its shit together. It had problems in 3x (2x, too, probably - but that was so long ago I don't really even remember what my 2x experience was like). So I waited for 3.6. Then I waited for 4x. Then I waited for 5x. Then 6x. Then 7x. Then 8x. Then 9x. After like 8 major point releases over four years, I've stopped waiting and moved on.