Same sig since 2003....
Slashdot videos: Now with more Slashdot!
We've improved Slashdot's video section; now you can view our video interviews, product close-ups and site visits with all the usual Slashdot options to comment, share, etc. No more walled garden! It's a work in progress -- we hope you'll check it out (Learn more about the recent updates).
Link to Original Source
Better yet, it appears to have been introduced in 2011, and known since March 2012."
Link to Original Source
The OP needs lower-priced spot instances, which are intermittently available and designed exactly for this workflow.
Here's how to utilize lower-priced spot instances for scientific computing:
1. Set up one long-running, low-cost instance (a small is fine) that creates a distributed queue using Amazon's SQS, and adds jobs to the queue corresponding to each "unit" of the relevant computational problem of interest. New jobs can be added using a command line interface, or through a web interface.
2. Create a user start-up Bash script for the spot instances that runs your main program -- I prefer using Python and boto for simplicity. The main program should connect to the SQS queue, and begin an "infinite" while loop. Inside the loop, the next job off the queue is pulled, containing the input parameters that define the "unit" of the computational problem of interest. These input parameters are fed to the main algorithm, and the resulting output is uploaded to Amazon S3. The loop continues.
3. Any time the queue is empty or the spot instance remains idle for ~5 minutes, the spot instance then auto-terminates using EC2's command line interface.
4. Finally, just write a simple Python script to pull all the results off S3, combine & analyze them, and export to another useful format.
You'll also need to set up your spot instance price threshold, and make sure the queue has jobs to run. That's it, it's fairly simple.
Emergent OOism -- that everything is an object, including the variable types -- can provide continual surprises of what is possible, even to veteran programmers in other languages. As you were developing and using Python, Guido, what was your favorite surprise? What was now easily possible using Python that would have been very difficult with another language (at the time, or even nowadays)?
Mine: a dictionary of lambda functions for parsing text, and writing a custom MapReduce capability for AWS in 372 lines.
That just means he's ignorant AND old.
I 100% agree.
And f2py is the perfect way to combine the advantages of Fortran + Python.
It beats SWIG any day.
I write lots of numerical codes for solving engineering problems, and most of the time I use Fortran, even though I have written some in C as well. Ultimately, the easier memory management and advanced array support is worth using a language considered by many to be strange and unusual. Of course, this is only for numerical analysis. Most other things I write in Python.
Go ahead. Argue. I dare you.
v same sig since 2002. v
I'll 4th that.
Wow, nigh universal consensus on Slashdot. Next, it'll be snowing in April in the Mid-Atlantic! (erp, nevermind)
The Road to Reality : A Complete Guide to the Laws of the Universe
by Roger Penrose
Likely the most serious math book you will find in a retail, consumer bookstore. An excellent read and essential to truly understanding modern physics.