code_martial's Journal: Beating the hell out of a "server"

Today is the last day of a very interesting two month... what do I call it... adventure?!

At my workplace we had the requirement of a Linux (RedHat 8.0) box with jBoss 3.0.0 running on it and support for 60 real users through 12 user accounts. There was supposed to be a complete EJB application developed and deployed by users of each of the 12 accounts.

What would a decent hardware configuration for such a setup be? I'd say -- Athlon XP 1500 or above, 512 MB DDR RAM and a 7200 RPM UDMA enabled IDE Hard Disk. What we had available was a Compaq Deskpro with a 5400 RPM 9 GB IDE Hard Disk, 256 MB of PC100 SDRAM and a Celeron 500 MHz CPU!

Keeping the "server" up and running itself was an ordeal and an adventure. I had anticipated some of the issues that could've cropped up during the two months of usage. So I set the limits on RSS mem usage and max number of user processes in limits.conf. This helped to keep the server responding quickly to user actions.

However, troubles started cropping up rather soon. Two weeks into active development and I got a call informing me that the server wasn't allowing any ops, complaining about too many open files. A little googlizing saw me bump up the file-max limit in /proc/sys/fs/file-max from 8192 to 65536 and a set hard limit of 32768 open files for jBoss in limits.conf. Phew! Problem solved. Out of curiosity, I did lsof | wc -l on jBoss's processes. That thing had over 19000 files open!

As the deadline approached (it was the same for everyone) the load on the machine started increasing and jBoss started acting up. It would refuse to deploy or undeploy people's stuff for extended periods of time, unless restarted. This made me write a cron job to restart jBoss daily at 03:00 hrs. This quickly proved to be too little and the cron entry was modified to restart jBoss every two hours!

Though regularly restarting jBoss did seem to help the situation, sometimes things came to a point where the swap usage exceeded physical RAM and things began to crawl. I had to restart the whole system three times -- twice on the day of the demo itself.

The greatest PITA for the users, and for me too, was the fact that there was only one server log file for all the accounts. Poor guys had to continuously run tail -F and still it was a nightmare trying to locate their own messages in the barrage of output that scrolled past. Betcha, even The Matrix code never scrolled as fast! One group of smart ladies had put tail -F server.log | grep groupname in a shell script and they took care to include the word groupname in thier messages, so they managed to save themselves from a lot of trouble.

At the end of it all, everyone was cursing jBoss for the crap that it was -- I guess not many of them realized what hardware they were using!