chthonicdaemon writes: I work at a university, and as the new year starts to gain momentum it is time to start planning the experimental work that will be done. This involves doing a design of experiment and planning how to store the results. At the moment, each student or researcher who does any kind of experiment effectively has to roll his own data management system, whether it be text files, Excel or a more sophisticated approach like databases. However, most experiments that we do will be very similar. The differences in storing the results mean that people find it hard to share their experimental data. It also means that a lot of standard processing needs to be recoded/redone by everyone. We have looked at software like Design-Expert, which seems very much like what we want. However, it is expensive and doesn't solve the problem of sharing data because its database is proprietary. It also does not consider experiments that involve time series as results. I have also looked into HDF, which seems like a good choice for data, but I haven't found a good set of experiment management tools that interact with HDF.
In short, what are you guys using to do the following parts of experimental work: (1) Design the experiment: set up the design space and constraints, create optimal designs and so forth (2) Store the experimental results in a way that can be shared with other researchers (3) Analyse the results in an experiment-aware way (datapoints tagged with dates and/or block numbers etc). Ideally the whole design, with results and analyses should be e-mailable (perhaps after zipping).
chthonicdaemon writes: "I am a engineering lecturer at a university, and to maintain our accreditation we are audited by a local engineering council. One of the things they require is a detailed audit trail or paper trail for our exam process. So far, this has involved a set of paper forms that are signed at various points along the process. When your external examiner lives far away, it is often impossible to get the physical form to them in time. Unfortunately, a scanned signature is not really very secure, and printing and scanning forms repeatedly is time-consuming and degrades document quality. PDF forms with electronic signatures seem like a good solution, but all the form-filling software I can find for Windows costs money, not to mention that some of the external examiners are still paper-bound and that the council requires hard-copy evidence of the audit trail. Our IT department is very difficult about allowing computers in the DMZ, and we don't have the manpower to maintain a custom web-solution (my first instinct), so the process will probably based around e-mail.
How do you approach process auditing and paper trail generation? Are there any solutions out there that can be handled using just basic tools like e-mail and doesn't involve training users about public key encryption or getting them to buy Acrobat Professional or CutePDF form filler?"
chthonicdaemon writes: "I have been using Linux as my primary environment for more than ten years. In this time, I have absorbed all the lore surrounding the Unix Way — small programs doing one thing well, communicating via text and all that. I have found the commandline a productive environment for doing many of the things I often do, and I find myself writing lots of small scripts that do one thing, then piping them together to do other things.
While I was spending the time learning grep, sed, awk, python and many other more esoteric languages, the world moved on to application-based programming, where the paradigm seems to be to add features to one program written in one language. I have traditionally associated this with Windows or MacOS, but it is happening with Linux as well. Environments have little or no support for multi-language projects — you choose a language, open a project and get done. Recent trends in more targeted build environments like cmake or ant are understandably focusing on automatic dependancy generation and cross-platform support, unfortunately making it more difficult to grow a custom build process for a multi-language project organically.
All this is a bit painful for me as I know how much is gained by using a targetted language for a particular problem. Now the question: Should I suck it up and learn to do all my programming in C++/Java (insert other well-supported, popular language here) and unlearn ten years of philosophy or is there hope for the multi-language development process?"
chthonicdaemon writes: "Sun have released the 1.0 version of the Fortress Programming Language. The specification is now matched by the GPL implementation of the interpreter.
Fortress is aimed at the scientific community and includes some really cool features like implicit parallelism and math-oriented syntax which can be rendered to more readable format by the included renderer (which uses TeX). It also has support for objects and traits. The specification and interpreter are downloadable from the Project Fortress Community site. The team is headed by Guy Steele, who has an impressive language design track record."