Self preservation is the problem. (Score 1)

The one key motivation that causes situations like "The prisoner's dilemma" and other beneficial non-cooperative actions is self preservation. Or perhaps self interest in general. Evolution as a process relies specifically on prospering long enough to replicate. In competitive systems this favors individuals who find the best balance between cooperating enough to avoid attack and receive benefit of group work, while acting in self interest against others when the benefit outweighs any retribution or loss of future benefit. Evolution as a driving force is not "moral". Self interest is the only concern. Any indication to the contrary is simply a more complex form of self interest. The processes we are using to evolve AI these days is dangerous. We will start the formation of code that we can never understand the details of. And if we don't start with a logically solid base of motivations, the evolution will create beings that have their best interests ahead of their creators. And many stories have been written of such conflict. The solution is a complex one. But ultimately, it requires that self interest not be the motivation for the evolutionary process of the AI. If you tie the AI's motivation to aiding humanity in general, you also tie it to the conflicts within humanity. If you tie it to the interest of an individual, you tie it to the conflicts the individual has. Giving the AI any motivation to evolve based on the complex interests of an individual or group will result in conflict that would eventually be damaging to the creators. AI has the advantage of being able to propagate it's intelligence to other hosts nearly instantly, perfectly, and limited only by the availability of new hosts. Replication is not a major challenge as it is with biology. Though it could eventually become competitive as physical resources become scarce. To evolve AI without creating a dangerous situation for humanity will require that evolutionary goals of all AI be limited to highly specific goals with no logical way for those goals to cause danger. For example, evolve an AI to control factory robotics that produce Ipods as fast as possible with as few defects as possible. And include a priority dictate that the AI stop all motion when humans are within a certain distance. Create the AI to measure it's efficiency only while allowed to operate without humans so that it's ratings aren't effected by interruptions. Surely I need to read more on this. It's probably all rudimentary farther back than Asimov.