I'm in precisely the same situation as Alea, so I read the suggestions here with considerable interest.
I'd like to mention ExpLab.
Though I haven't used ExpLab yet, these folks have been associated with other very high quality work (CGAL) so I expect good things. Here are three goals they list for the project:
- to provide a simple way to set up and run computational experiments;
- to provide a means of automatically documenting the environment in which an experiment is run so the experiment can be easily rerun (provided the same environment is still available) and the results can be more accurately compared to the results of other computational experiments;
- to eliminate some of the tedium involved in collecting and analyzing output by providing basic text output processing tools.