Seriously, I've literally been chasing this for years to no avail. If it is just some sort of super-aggressive caching policy, then I'd refer you to the comment by AC below, who hit the nail on the head: "Sufficiently aggressive caching policy is indistinguishable from memory leak."
Well, that depends what tool you use to gauge memory usage. I use both the "about:memory" page and the app working set size in Process Explorer. The two don't often agree with each other. In particular, even though about:memory shows that most memory has been released, the working set still hovers over 1GB. Sometimes you have to sit and wait a while before the memory pooler (or whatever) gives memory back to the OS. However, through various tricks, I can indeed get both to reduce to levels similar to a clean startup. It's well known that Firefox will cache many, many pages in memory "just in case" you hit the Back button, and will even do so after closing a tab/window. Yes, it may be dumb design, but I'm not convinced it's due to leaks.
Slow and jerky scrolling, delayed mouse click events, etc etc.
I've seen this on XP 32-bit when memory usage goes beyond ~1.8GB, including parts of the display not drawing correctly, and even rainbow graphics corruption. I presume its because internally the browser is nearing the application memory limit on a 32-bit OS. On 64-bit, the same version of the browser does not have these problems. The biggest problems I've noticed on 64-bit are the pauses (cycle collections, of course), and sometimes the text caret disappears. Otherwise the browser is perfectly stable and never, ever crashes.
Honestly, if Mozilla doesn't fix this in the next rev or two I'll be switching to Palemoon or Opera or something
I've been using PaleMoon as my primary browser for about 2 years and it's been great... until version 27. Version 26 had absolutely none of the memory hogging or leaking problems of Firefox, and would run perfectly around 300MB or less for days. It was fantastic. Then version 27 was retrograded to a newer version of Firefox, and now it's just as bad, gobbling up a gig of memory and never letting go without using the "tricks". The fact that PaleMoon 26 worked exactly as it should and 27 "leaks" memory is part of the reason I believe it's a configuration issue. Most of my work has been comparing PM 26 an 27 to track down the problem. Alas, there's 40K additional files in version 27 and most of the settings for memory management are now "-1" (for fully automatic) instead of hard values like in version 26. It's proven very difficult figuring out how the code actually works. I wish I were a more experience application developer, and not just a web designer.
A word of warning: many web frameworks test Firefox by brand name, and not whether your browser is Mozilla compatible. As a result, a LOT of pages will show up blank or missing huge chunks of content when viewed in PaleMoon, regardless of how you configure the user agent mode. Alas, life with PaleMoon is very frustrating, as there are way too many web pages that refuse to adhere to standards and will not work properly. It's the same situation with all "unpopular" browsers. Apparently, Chrome, Firefox, and Edge are the only browsers worth supporting these days, and standards-compliance just isn't fashionable anymore.