Submission + - Ask Slashdot: Scientific Computing Workflow for the Cloud?
diab0lic writes: I have recently come into the situation where I need to run cloud computing on demand for my research. Amazon's EC2 Spot Instances are an ideal platform for this as I can requisition an appropriate instance for the given experiment {high cpu, high memory, GPU instance} depending on its needs. However I currently spin up the instance manually, set it up, run the experiment, and then terminate manually. This gets tedious monitoring experiments for completion, and I incur unnecessary costs if a job finishes while I'm sleeping for example. The whole thing really should be automated.
I'm looking for a workflow somewhat similar to this:
- Manually create amazon machine image (AMI) for experiment.
- Issue command to start AMI on specified spot instance type.
- Automatically connect EBS to instance for result storage.
- Automatically run specified experiment, bonus if this can be parameterized.
- Have AMI automatically terminate itself upon experiment completion.
Something like docker that spun up on demand spot instances of a specified type for each run and terminated said instance at run completion would be absolutely perfect. I also know HTCondor can back onto EC2 spot instances but I haven't really been able to find any concise information on how to setup a personal cloud — I also think this is slight overkill.
Do any other