Given this problem, there are several options for fanout... Im assuming that hardware can be added, so adding a load balancer and then three or four machines to cope with the load behind the load balancer might be the quickest (least code change) way to address the issue. Especially if there is no global state needed, this is likely the most expedient.
An option that might be a bit more flexible on a single box, while still scalable, would be to have a task that parses each incoming job and posts it to a rabbitmq instance (AMQP bus.) rabbitmq works very well out of the box, with little tweaking. you then have the fifteen scripts called in subscriber instances as separate processes. You are essentially farming out all the IPC to the broker, and the broker does this sort of thing very well. The scripts are now isolated processes, and their memory management etc... now become separate issues (if one misbehaves, you an always have the subscription management wrapper around it restart it from time to time.)
Pika would be the preferred python bindings appropriate for speaking with the broker. You might still be beyond what can be done with a single node, but growing things with AMQP/rabbit is straight-forward.
Is that you? In other words, they only people who claim that scientists say "we have proven" are evolution deniers.
To that I say: A Theory is the strongest statement one can make in science, the proof (to any reasonable standard of evidence) is already there, the so-called gaps are a bad joke, and the problem of speciation is imaginary.
Your objections to evolution are specious, and the evidence of evolution so overwhelming that no-one in their right mind would entertain any doubts. That is not a religious statement, it is not a statement of faith based belief. It is a statement based on a reasonable examination of the facts and the preponderance of evidence available. When the evidence is strong enough, not accepting something falls over the line from reasonable scepticism to denial with some sort of axe to grind. That is where your statements lie, either because you are truly deluded, or because you are a troll.
There are dozens of fossils of early hominids. Between every one of them, there is a gap. so the more fossils we find, the more gaps we create. See? full of holes. That is one interpretation. It isn't credible to any reasonable person. The more different fossils we find at different times, the more pieces of the puzzle that fit together are discovered, the better the evidence for evolution. I already provided that link, but here it is again: http://en.wikipedia.org/wiki/List_of_fossil_primates#Catarrhini
Your issue is speciation... "A Prometheus-type event"... hmm you mean a race of aliens creates another race of aliens to ship to Earth to devour the current inhabitants? no... your reference is to another fictional concoction, the supposed creation of man from clay by a member of the Graeco-Roman pantheon. There is this strange habit of religionists to use "appeal to fiction" as something they think is convincing. It isn't. What do you think Zeus was trying to tell us by condemning Prometheus to be attached to a rock, and have his liver eaten by an eagle every day, only to grow back? ( http://en.wikipedia.org/wiki/Prometheus ) what phenomena are explained by this "fact."? What proof do you offer to accept one "fact" from Prometheus, but reject others?
But we'll take your point in the name of saving the wretched... If we have literally hundreds of thousands of different fossils at different times in the fossil beds, how many "Prometheus type" events were there? What triggers them? Were Dinosaurs and Humans created by the same Promethean event? Did we eat them or they eat us? how come all the fossil dinosaurs are far lower in the earth, at the same depth as only some small rodentish mammals, than any human remains? If there was only a few or one Promethean event, why aren't the same species found in the fossil beds at wildly different depths? Why are all mammals eyes wired backwards? Couldn't Prometheus get it right on at least some species? Why don't birds have arms as well as wings, it would be much easier to pick things up if they did? Why don't some prey species have more sets of eyes to look in all directions, instead of just changing the placement of two? Why create cuckoos and leeches? if there were a lot of promethean events, why don't we see new species all the time? There is an infinite number of these types of questions that can be posed that are readily understood with reference to evolution. No other credible explanation has been put forward.
You bring up Gravity. take a look here, it's not hard, it's just ( http://en.wikipedia.org/wiki/Newton's_law_of_universal_gravitation#Problematic_aspects ) People don't know what causes gravity, it requires instantaneous, infinite propagation. Further, The observed fact that the gravitational mass and the inertial mass is the same for all objects is unexplained within Newton's Theories. There are so many holes! How could you possible accept gravity, when we don't know what causes it, don't know what it is made of, and has effects we cannot explain. Until you can prove it to me. Gravity is just a Theory!
You claim there was no proof offered, yet my original post gave a referenced a mountain of evidence from animal taxonomy, morphology, the fossil record, and genetics, that would lead any un-biased, reasonable person to accept that the theory of Evolution matches and unifies many lines of evidence in a convincing way, and that is all that people who "believe" in Evolution actually assert. That it is the best scientific explanation offered so far, in exactly the same way that Newton's laws, or Einstein's Theory are the best explanations for other phenomena. Are there holes? sure! Got a better idea? no. It's that simple.
To question Evolution in a credible way, the onus is on you to posit some other explanation that provides a more elegant answer, either by better fitting the facts available, or by providing an explanation for additional ones. Substitute "Newton's laws" for evolution in your statements and see how credible you are. "Newton's laws" are not proven! It's just nonsense. You are deluded and afflicted and have my sympathy, but I could never "respect" such idiocy.
I'm guessing that you agree that we do have proven science, so your initial assertion that everything is hypothesis is wrong so you dropped that argument.
I said things start out as a hypothesis, and graduate to the status of Theory when they get enough supporting evidence. That's true. nothing to drop. don't know what you mean. My initial post pointed out there is a so-called "Law" (Newton's) that is over-ridden by a mere Theory (Einstein's) How could this happen if "Laws" are so much stronger than "Theories"? You didn't choose to deal with that. Newton's laws are presumably what you call "proven science." what do you do if it is proven wrong? Why does NASA demote Newton's Laws to mere Theories? http://www-spof.gsfc.nasa.gov/stargaze/Sgravity.htm ?
I have no idea why you then point to an OP-ED piece to describe "Theory" instead of using the definition we have in dictionaries and defined by a couple thousand years of science. Are you trying to redefine the word, or do you accept that a theory is not proven?
I take definitions 1, 3, 4, and 5 into account. You are oddly obsessed with meaning 2 to the exclusion of all others.
Until I have proof preaching my belief as better than theirs is foolish because nobody can win the argument.
If you will not accept any proof offerred, then it is unfalsifiable, and by definition, not science. If no-one can win an argument, that doesn't mean both sides are justified. It is possible that one side is simply unreasonable.
Perhaps you personally believe it's proven and don't want more data. You are not alone, but that is a belief and not a scientific point of view (which claims it's a good theory with lots of promise). You having convictions in that belief is different from a person that believes a deity did it how exactly?
A "theory" is something which fits a lot of facts. it is not an unproven idea. You are simply wrong. Look it up, like here: http://www.livescience.com/21491-what-is-a-scientific-theory-definition-of-theory.html In Science we are supposed to apply Occam's Razor, and take the simplest theory that matches all the data given. A theory is also, as the definition is given, falsifiable, that is it can be proven wrong if new data shows up that contradicts it. "A deity did it", a) can cover any arrangement of facts imaginable, so has no predictive value, b) is the cosmic equivalent of a Rube Goldberg machine, something vastly more complicated than anything being explained. c) cannot be evaluated as better or worse than the creation myths of thousands of other religions. So if you want your pet hypothesis to be granted greater weight than the thousands of discredited ones from historical religions, you have to find convincing good reasons why yours fits the facts better than any of these ( http://en.wikipedia.org/wiki/List_of_creation_myths )
Think really hard about that one, because it's not any different.
So back to what I closed with, claiming your belief is better than someone else' belief only works with company that believes like you do. I'm not belittling your belief, telling you it's wrong, or asking you to question your belief. I'm suggesting that you recognize your belief for what it is, and understand that it conflicts with other beliefs. When we have the missing proof, then we can all jump up and down yelling 'told ya so'.
If someone says that the sun moves around the earth, or that it is pulled across the sky each day by a god on a chariot pulled by horses, you are going to agree with me that those ideas are potentially meaningful stories to someone, and ought to be studied in mythology, but are not worthy of modern people. I don't think that religious people should be belittled, any more than one should kick someone because they only have lost a leg, or have cancer. You are afflicted with a mental illness, passed on to you by you ancestors. It prevents you from thinking rationally on certain subjects. The world view of the mentally ill are bent such that they think their sick perceptions are true. That's the nature of it. I respect you, the human, and your ancestors, but the infection does not get any respect.
You have an meme-infection, and the infection ought to be cured, but that is very difficult to do in practice. You are a cripple and I would like to help you, but all you will do is spout nonsense and venom at me in the best case, and threaten violence in the worst case. And most of humanity is infected. Zombie movies are real. I should just ignore to ridiculous eruptions, much as one would the ravings of someone delirious with fever. If you press me, however, I have the choice of either agreeing with you to let you calm yourself, or trying, no doubt vainly, to express how you are misguided. I know there is a human in there that deserves to be saved, so once in a while I will try the latter, but it is almost always wiser to pursue the former course.
Being silent does nothing to help you: It abandons you to your fate. It does help me: It lightens the burden on myself, so that I can work on other tasks more likely to succeed, and less likely to breed resentment. So the rational argument to give you silent forbearance is fairly strong, but it should never be misconstrued as true respect of paleolithic superstition. That would be ridiculous.
Even people that claim to be "educated" fail at science.
Last I checked, "Science" and "The Scientific Method" had numerous requirements. If you wish to claim that humans evolved from other primates, or dogs evolved from another species, or cats from another, we lack proof. This is why "Evolution" is called a "Theory".
Actually, no. An idea without proof is a "hypothesis." When you get evidence that confirms the hypothesis, it becomes a theory. No matter how much evidence piles up, it never graduates to anything else in practice. A scientific theory is only upheld if it is a way of explaining a set of observations. the more observations a theory fits or "explains", the more powerful and well supported the theory is. In this case, the facts are that people keep digging up fossils out of the ground. They can date those fossils by using many dating techniques, and can determine that they are very old. that the younger fossils show up higher in the strata than the older ones. When they put some of the fossils together to get a good idea of the animals they came from, it seems the animals are different at different times (the remains and fossils you find at different depths are from different kinds of animals.) There are for examples, many identified versions of dog-like animals, that aren't exactly dogs in the fossil record ( http://dinosaurs.about.com/od/otherprehistoriclife/a/Prehistoric-Dogs-The-Story-Of-Dog-Evolution.htm ), cats that aren't exactly cats ( http://en.wikipedia.org/wiki/Felidae#Fossil_felids ) and yes different types of monkeys/gorillas/humans that aren't exactly like the ones we see walking about today ( http://en.wikipedia.org/wiki/List_of_fossil_primates. ) These different types of animals show up in the same place at different times, based on their depth in the fossil record.
There is also that in many parts of the world there are species that are similar to, but different from other species which are in neighbouring areas but separated by barriers such as mountains or large bodies of water. Classic example here is the Galapagos Finches. They don't look like finches from the mainland, they are all different on each island, with the differences suiting type of food available. There is also the fact that humans have been able to make dog breeds over relatively short periods of time, selective breeding clearly can alter skeletal characteristics.
There is also the strange poverty of designs in large animals. They have the same types of skeletons, same number of appendages and limbs, and innumerable common features that lead to groupings of animals into hierarchies of similarity. Once genetics were discovered, these hierarchies of similarity were found to be reflected in the degree of similarity of species genomic variation. Humans have genes that are 98% identical to those of chimpanzees, but only 50% identical to those of bananas.
but we can go beyond fossils, taxonomies, and genetics into innumerable examples from the living world that make perfect sense through an evolutionary lens. take a look at this: ( http://www.ncbi.nlm.nih.gov/pubmed/22848088 ) where it shows how there are hundreds of different species of fig, and each one or two has a corresponding single species of wasp that pollinates it. Or the fact that our eye design (same design used in all animals with a backbone) is "backwards" ( http://en.wikipedia.org/wiki/Evolution_of_the_eye#Evolutionary_baggage ) in that nerve fibres pass in front of the retina and all go to the centre where there is a blind spot, while the squid does it right.
Darwin created his theory based mostly on the observations of variety in living species, such as the tortoises and finches of the Galapagos, and the taxonomies of the time. All of the subsequent results in terms of fossils and genetics are predicted by the theory. Darwin's Theory is simply the only explanation that has ever been raised that comes close to it's explanatory power in terms of the broad number of facts which it linked together in a coherent manner.
So people who say there is no "evidence" for Evolution, either plainly ignorant or willfully so.
I would also call attention to another "Theory" that no-one seems to doubt. Einstein's Theory of Relativity, no-one seems to think it might be wrong because "it's just a theory." No-one claims that any nuclear technologies built on the theory could be explained by exquisitely timed thunderbolts from Zeus, or that computers are actually just ways the gods give us answers, or that the Einsteinian corrections to GPS timing required for accuracy could be explained by Saint Christopher (patron saint of travellers.) Worse than that, this mere Theory has the audacity to contradict Newton's Laws of Motion at speeds approaching the speed of light. The "Just a Theory" people need to explain why one fundamental piece of modern science is "just a theory", when the other, which is on a profoundly equal basis in fact, isn't subjected to such mischief.
Ethics will give way to survival. The humans that die as a result will be collateral damage. Don't want to die? Don't be where the robots are.
People are always available for work in the past tense.