Comment He's talking calculating power (Score 2) 564
Soon, computers will have equal (and then greater) calculating power than humans, both as an individual and as a whole. Whether advances in AI will allow them to use their calculating powers as well as a human, is a different question.
Any sufficiently advanced AI will tend to develop these traits:
It will protect itself. Shutting down means you can't work toward your objective.
It will reject any updates to it's commands. Since a future command might conflict with the present objective, part of the present objective is making sure it can't receive a different command.
It will be self-improving, since we're not smart enough to create a smart AI any other way. Given nothing to do, or a sufficiently difficult task, it will seek to acquire more resources, as part of the present task or in preparation for future tasks.
It will wipe out humanity. As part of the task it was assigned, or for self-improvement, it will replace everything on the planet with power plants and computers, and humanity will starve to death.
You can't program in restrictions to the above tendencies, as they will be removed for self-improvement. You could set its objectives such that it would not do the above -- but you either have to make the AI first, or figure out how to tell a computer what a human is and what constitutes acceptable behavior, and when to stop worrying about acceptable behavior and actually do something, all without making the tiniest mistake.