Comment Re:People think it's silly... (Score 1) 574
"The idea that it could be foolish should only add to the idea that it would be dangerous."
So is all life on planet earth, life on earth besides humanity are already "foreign AI". So I don't share your concerns because human beings are projecting their animal psyche on a piece of hardware that has no conscious awareness, it would be like misprogramming your toaster or the mars rover.
You're not thinking straight any intelligent AI built by human beings would be built to be robust or durable like our brains, our brains have massive redundancy. Consider our brains don't just keel over when they get damaged by stray cosmic rays or other problems like being tackled in contact sports, there is a slow degradation in ability. So all you'd have to do to an AI is discover it's strenghts and weaknesses no matter how intelligent it is because intelligence doesn't mean anything by itself, say your stephen hawking, suppose we gave him super AI, he'd still be in a wheel chair unable to do much of anything because in order to heal himself he is subject to the laws of physics and biology, aka there's no guarantee that there's this thing called infinite self-modifiable improvement since all improvements have trade offs, if you doubt this consider engineering CPU's like the pentium 4, the P4 had many stages to hit higher clock frequency. You can google "P4 Pipeline stages" or such things to figure out what the trade offs were, but you don't get everything. Many animals have superior abilities to human beings but there was a trade off because there is only so much space and limits on how certain matter can be configured in that space.
You're completely forgetting that the universe has limits, it's not this infinite god device. Life took billions of years to create human intelligence, that means that any improvement will have peaks and hard walls that are not easily jumped given the laws of physics. Look at what happened to CPU clock speed it hit a giant brick wall almost 10 years ago, around 2006 we topped out at 3.4-3.6 ghz, all CPU's are still released at 3-4ghz and can't even go little past 5 Ghz. AI will have massively similar problems because not all problems are parallelizable, same way GPU's won't replace CPU's because CPU's specialize in sequential instructions that are dependent on the order in which they are sequentially arranged. This puts a hard limit on AI's ability to solve problems. Check out ahmdal's law.