Re:This is spot on -- I did some benchmarking, too (Score 1)

Memcached is an excellent suggestion -- especially since it is a distributed cache. Of course, there is overhead; some time ago, I did a series of tests using ApacheBench, trying to establish just how big the performance penalty is.

In purely local tests (i.e. ignoring network overhead) performance dropped 40% with the introduction of memcached. Over a rather slow 10 Mbps LAN, the performance degradation was only 10%. Note that memcached was still local to the server in the second series of tests -- only the request from ApacheBench went over the LAN.

Much of the memcached overhead was due to marshaling; in the non-memcached version, all objects lived entirely in memory. Increasing the number of objects in memcached 10 times resulted in a massive 70% performance drop for the LAN-based tests.

So, if you cache few objects (or plain strings), and there is little communication between the machines in your server farm, memcached performance will be close to pure in-memory performance. On the other hand, if there is lots of local I/O to handle a request, and you maintain a complex set of objects in memcached, you will take quite a hit.

The above is not the fault of memcached -- it is just that when designing distributed systems, you are actually trying to reduce the amount of communication inside the cluster. This is similar to multi-threaded design, where you must try to reduce the number of threads and especially their interaction with each other (I covered this in one of my articles for O'Reilly).

In my system (FlightFeather) I do maintain quite a lot of in-memory state to improve performance. For example, a special subclass of the float type (this is in Python) helps create a fast session cache. In addition, the operating system helps when you are using files directly, by maintaining a cache on its own. The most important thing, however, is to try to generate static content for frequently accessed material. The authors of memcached already know that static content is "boring, easy" (PDF; see page 30). If you want reliability and performance, boring and easy is where you want to go :-)