Well, for the last month I've had my Mini which just sits there as an iTunes server run out of memory. Never happened before.
Trying to watch a bit with Activity Monitor, the kernel_task balloons up over 4Gb, Finder shows as non-responsive, the File Cache is only around 1Gb -- and "Compressed" is huge. Free memory the last time I caught it was about 16Mb out of 16Gb.
Given its role, I expect the File Cache to grow -- but either it isn't or Activity Monitor isn't reporting it as such, as it only shows a few Gb in normal operation and doesn't show at all when the problem state is hit. And even if it did eat all the RAM for caching -- if it can't shrink it down without hanging all the user processes then that's certainly a bug.
Otherwise, my assumption is that something is leaking in the kernel over time so the system can't find it to clean it up, everything else gets shrunk/compressed as it can and the reclamation hogs the processor in the worst case. Again, haven't seen this before Mavericks and based on some support threads, I don't think I'm alone.