It sounds like you're asking for an option like "Firefox, please pretend my computer only has 512MB of RAM" that affects all types of caches in Firefox. Or a slider that's like "Make Firefox as fast as possible [------|--] Leave as much RAM for other apps as possible". I don't think such a thing exists.
You can install RAMBack for a "clear all in-memory caches" button. In combination with about:memory, it can help you tell the difference between healthy caching behavior and memory leak bugs. Unfortunately, it skips sqlite-based caches, which are some of the largest.
You can disable Firefox's "Block reported attack sites" and "Block reported web forgeries" to save maybe 15MB of RAM. On the other hand, it's nice to have that extra defense against zero-day attacks.
You can disable Firefox's URL history to save maybe 35MB of RAM, but then you'll lose purple links and the awesomebar (unless you use bookmarks extensively). There used to be a way to tell Firefox to only keep a few weeks of URL history, but I can't find it now.
You can lower browser.sessionhistory.max_total_viewers from "8 if there's enough RAM" to 1 or 2. This controls how many navigated-away pages Firefox will keep in memory with their complete state (rather than just scroll position and form data). I'd advise against lowering it all the way to 0, because if you accidentally click a link or close a tab, restoring the page from bfcache is not only faster but also significantly less likely to lose page state (especially on AJAXy pages).
If you use session restore, you can instruct Firefox to only load restored tabs once you switch to them, by setting browser.sessionstore.max_concurrent_tabs to 0.