Comment Some technical info for slashdotters (Score 5, Interesting) 61
I wanted to give some info on the technical aspect of getting this to work that might be appreciated by slashdotters.
You can read about the Blue Waters hardware profile here. Our simulation "only" utilized 20,000 of the approximately 700,000 processing cores on the machine. Blue Waters, like all major supercomputers, runs a Linux kernel tuned for HPC.
The cloud model, CM1, is a hybrid MPI/OpenMP model. Blue Waters has 16 cores (or 32 depending on how you look at it) per node. We have 16 MPI processes going and each MPI rank can access two OpenMP threads. Our decomposition is nothing special, and it works well enough at the scales we are running at.
The simulation produced on the order of 100 TB of raw data. It is easy to produce a lot of data with these simulations - data is saved as 3D floating point arrays and only compresses roughly 2:1 in aggregate form (some types of data compress better than others). I/O is a significant bottleneck for these types of simulations when you save data very frequently, which is necessary for these detailed simulations, and I've spent years working on getting I/O to work sufficiently well so that this kind of simulation and visualization was possible.
The CM1 model is written in Fortran 90/95. The code I wrote to get all the I/O and visualization stuff to work is a combination of C, C++, and Python. The model's raw output format is HDF5, and files are scattered about in a logical way, and I've written a set of tools to interface with the data in a way that greatly simplifies things through an API that accesses the data at a low level but does not require the user to do anything but request data bounded by Cartesian coordinates.
I would have to say the biggest challenge wasn't technical (and the technical challenges are significant), but was physical: Getting a storm to produce one of these types of tornadoes. They are very rare in nature, and this behavior is mirrored in the numerical world. We hope to model more of these so we can draw more general conclusions; a single simulation is compelling, but with sensitivity studies etc. you can really start to do some neat things.
We are now working on publishing the work, which seems to have "passed the sniff test" at the Severe Local Storms conference. It's exciting, and we look forward to really teasing apart some of these interesting processes that show up in the visualizations.