you seem to know what you're talking about. can you explain this idea of memory compression, and what the heck the new activity monitor means? the old one made sense. Pie chart, showing free, available, and active. Now it's apparently using up all my memory I have 8 GB but it shows a line chart with a small amount of "memory pressure".
Sure -- I'll try to explain it the best I can. I won't make any specific judgements as to whether the new controls are better than the old, except to point out that there is more useful information in the new that wasn't present in the old. You're still perfectly welcome to prefer the old pie chart :). I'll try not to stray too far into the esoteric; if you need more details on a specific subject here, feel free to ask.
First a bit on the theory of memory management in general. In most modern operating systems like Mac OS X, each application appears to get it's own memory space, starting at '0' and running up to 0xffffffffffffffff (a fancy way of saying the addresses go from 0 to somewhere in the neighbourhood of 1.84*10E19 bytes of memory). To make things easier to deal with, the operating system breaks these up into chunks 4096 bytes in size called a 'page'. Now 1.84*10E19 bytes is probably way more memory than you have available on your system, but that's okay -- while conceptually an application can use any of that memory space for pretty much anything it wants, the operating system keeps track only of which pages have actually been allocated to each application. This system is called 'virtual memory': each application has its own virtually memory space to play with that doesn't interact with he memory of any other application. This is the value that shows in the "virtual memory" box in the activity monitor.
Now of course, you have real, physical memory in your machine, and you don't have a separate set for each application (in a physical sense -- you don't have one set of chips for Safari, and another set of chips for iPhoto, for example). The real memory has to hold the virtual memory somehow, and be able to map from one to the other. The operating system keeps a structure known as the Translation Lookaside Buffer that keeps this mapping for pages stored in physical memory. So it might have a bunch of entires for Safari, saying that the page consisting of what the application sees as memory area starting at 0x0000 and going to to 0x0FFF are stored in memory location 0x40000000 (the 1GB mark), the page of what the application sees as memory area starting at 0x1000 and going to 0x1FFF are in location 0x40096000, etc. In fact, the pages can be all over the place, and not even in order -- the operating system keeps track of all the used memory pages for the application wherever they are stored in memory. The amount of physical memory you have shows in the "Physical Memory" box of the activity monitor.
If you don't get all that, don't worry -- the main takeaway is that these pages can be stored in memory, and the operating system tracks of them when they are. Because we work with all of these pages, however, the operating system can also store them someplace else. Prior to Mavericks, this was always written to disk in the "swap" file (also sometimes known as a "page file"). This happened when memory pressure gets higher than the operating system can handle in RAM alone; that is, programs are asking for more virtual pages than the operating system can fit into real memory. To try to make room for new requests without unloading applications, the operating system will periodically go through the list of pages if memory pressure is high, find the least-used pages (you might have some application running that you put into the background and haven't touched in hours, for example, or applications which have reserved pages for things such as documents you haven't looked at in hours, even if you've otherwise used the application itself), and write them to disk. This is known as "swap". The pages of course are still there -- if memory pressure decreases, the operating system can load them again. Alternately, if you need to use the memory, the operating system will load it again, writing some other old, unused page to disk. The size on disk of all of these old pages sent out for long-term storage like this is displayed in the "Swap Used" box in the activity monitor.
When memory pressure is really high, this can get really ugly really fast. Everyone has hard drives that are massive compared to the physically available RAM, so the system can use this system until the hard drive is full. For many people, however, the system will become nearly unusable long before this point; if that "least used" page is one that hasn't been used in just a few milliseconds, and will be needed again in a few milliseconds (that is, all the pages have been used very recently and are expected to be used again), you can wind up in a situation where you're trying to write out some not-very-old pages in order to load some also not-very-old pages, and you're going back and forth doing this again and again and again, the the speed at which you can do so is slower than the need for the pages, slowing your system to a crawl. You've probably seen this before -- you load too many tabs in Safari and have a bunch of other stuff running, and all of a sudden the hard drive is running all the time and the system is unresponsive until you unload some tabs or programs and wait for the system to catch up to you (in effect, reducing the memory pressure). This phenomenon is known as "thrashing".
As an aside, modern operating systems take a shortcut when allocating memory to applications that request it. The memory page is typically only actually created when you put something into it. Empty pages are recorded by the system, but aren't given any real physical existence until you put something into them. This means you could (conceptually) ask for more pages than the system could ever handle -- an applications entire memory space in fact -- and no real memory pages will be used until you put something into them. This is the reason why you can sometimes see the "Virtual Memory" value exceeding the amount of RAM, and still have no swapping happening.
That brings me to "memory pressure" which you asked about. This is basically a measure of how responsive the system can be to give out new pages of memory when an application needs them. When it's a low, green line you effectively have no pressure -- the operating system can give pages immediately with little or no work to an application that asks for some new pages. When it's tall and red, the pressure is very high, with very little real RAM available to give out, lots of reliance on reading and writing pages to the hard drive swap file, and general slowness. You can think of this as the state where the programs you're running (including the operating system) has asked for way more RAM than you have, and the system is having a hard time rearranging the pages in order to make room for new requests. As the memory pressure gets higher, the probability of thrashing also increases.
Now, in Mavericks Apple introduced a new system they call "Compressed Memory". This subsystem basically sits in-between the point where the system looks for old pages to write to the swap file, and the actual writing of those pages to disk. Instead of writing those pages straight out to disk, the operating system finds pairs of pages and runs them through a compression algorithm, and then stores them as a single page in memory. In effect, you take two pages and jam them into one page, giving you an empty page to work with. Mavericks tries to do this with as many pages as possible instead of writing them out to disk, because it can do this operating a whole lot faster than writing all that data to disk, which is comparatively slow. The amount of memory the system is using to hold compressed data is listed in the bottom-right "Compressed" box in the activity monitor. Looking at mine right now, it says I have 392.3MB compressed. This is equal to over 102 million pages of data that the system has brought back available for application use by compressing those pages I haven't used in a long time. If memory pressure were to increase, more memory would be compressed. If pressure continues to increase, those least-used compressed pages will be swapped to disk (in compressed state).
The key to performance, of course, is not to write anything to swap if it can be avoided. Swap slows things down. Conceptually memory compression also slows things down, but it's so slight and imperceptible at the level it typically occurs at you'll never notice it as a user.
"App Memory" is the amount of memory actually in use by applications and the operating system itself. Memory pressure will be green so long as this value is less than your Physical Memory. "Wired Memory" is a special subset of memory that cannot be compressed, swapped, or written to disk -- typically because it's too core to the operating system (imaging if the operating system paged out to disk the code used to read pages from the disk! You'd be stuck in a paradox and the system would stop working. Much of the OS kernel is always flagged as "Wired").
Which leads me to the last value in the application monitor -- File Cache. It's well known in virtual memory circles that unused memory is wasted memory. Memory is very, very fast, and if you're not using it for anything at all, you're not gaining the benefit of its speed. For this reason, the operating system keeps a File Cache in memory where it stores bits and pieces of files it thinks you may need in the near future. It acts as a sort of predictive read-ahead of your hard drive: if you need a file in the cache, it will load in the blink of an eye without spending much time hitting the hard drive, making the system seem really, really fast. Modern OS designers use a file cache system that will use as much available memory as possible in order to benefit from this speed increase as many times as possible -- the larger the disk cache, the greater the probability you'll need something from that cache, and the faster the system will perform. As such, the File Cache claims nearly as many pages as are available in order to increase this probability, and give you a faster experience loading files (note that some number of pages is usually left free in order to have somewhere to swap to and from as the need arises, but it doesn't have to be particularly large: maybe just a few MB of space)
The magic of the File Cache is that, since it's typically a read-only copy of what's actually on your hard drive, if an application needs more memory than is available, the operating system can simply drop the pages int he cache with very little effort. They are never compressed or swapped (which would defeat the purpose -- why would you write data to the hard drive that already exists on the hard drive?) -- they just get dropped if an application needs more memory (perhaps by loading a new application). In this way, the File Cache will basically grow and shrink to fit into whatever free physical memory your system has at the time. This is why you may see your "Memory Used" value getting very close to the "Physical Memory" value, but with the Memory Pressure still down in the green and with no swap used: if you get into this situation and something asks for more memory pages, the OS will simply drop some of the disk cache pages and reassign them to the application.
For this reason, it's actually expected that Mavericks will always appear to be using all of your memory. Trying to fight this will actually hurt your performance. The truly important things to keep an eye on is the "Memory Pressure", as this is a much more exact guide to what you should expect performance wise as you use more memory. The old chart was great for people who know the ins and outs of virtual memory (like me), but it often led to users trying to do weird things to try to "clear out" RAM in order to see more free space. You have to trust that the operating system can do this faster and better than you can, but just seeing a pie chart showing all of your memory in use doesn't help. The "Memory Pressure" on the other hand is a much better measure, as only when it gets into the higher values should you start to look into doing something (such as unloading unused applications, browser tabs, documents, etc.).
I hope you didn't find this TL;DR -- I know it's long -- the system is unfortunately very complicated, and you could spend an entire university semester dealing with the subject in depth. When I taught 3rd year Operating Systems at a University I'd usually spend at least a week on this subject, and even then you'd just crack the surface and give as much background as you find here (albeit with some more algorithmic details). If you did find this TL;DR, here are the quick takeaways:
- Swapping to disk should be avoided as it slows your system down. Its main benefit is to allow you to use more memory that you physical have. Try to keep this value low.
- Mavericks tries to help you keep the swap size low by compressing pages before it has to write them to swap (with the hope the saved space means it never has to write to swap). And if it does eventually have to write to disk, it writes out the compressed pages, making swapping roughly twice as fast as it was prior to Mavericks.
- Try not to manually manage your memory. It's both okay and expected that the OS will try to use all of it. It does so to try to speed up your computer. If a program needs some of this used space, the OS will give it back immediately.
- The old memory pie chart is great for the uber-techies who understand virtual memory, but doesn't paint a useful picture if you don't. Memory Pressure is much, much more useful to laypeople, as it gives you an indication of when you should actually start to worry about reduced performance. When the memory pressure is low/green, the OS has lots of memory it can dole out at a moments notice (even if the Memory Used value is close to the Physical Memory value). When it's yellow, you'll start to see some performance degradation if you don't unload some stuff. When it's red, performance is already suffering and you should unload some stuff right away.
- Virtually Memory is your friend! Try not to second guess what the OS does with it -- it's smarter and faster than you are. You only need to take action if the Memory Pressure creeps into the red (or if it's yellow and you know you're going to need a bunch in short order). If you find you're frequently in the yellow or red, a RAM upgrade would be of tremendous help (and conversely, if it's always green a memory upgrade probably won't help you much.
Congrats if you made it to the end. That's my "short" version of a very complex topic. I hope it made sense and is useful to you.