You wrote: "As useful as capitalism has proved to be, its motivations are primitive and short sighted. How AI is being punted is another example of "bad" capitalism. Bad capitalism has helped wreck the planet more than anything else."
Geoffrey Hinton, as a self-professed socialist, makes a version of your point in the interview previously linked to.
And your point is ultimately the key insight emerging from our discussion, as I reflect on it. AGI or especially ASI may indeed take over someday to humanity's detriment, but that is likely in the future if it happens. What is the biggest threat right now to most humans is other humans developing and using AGI or ASI within a capitalist framework.
I wrote to Ray Kurzweil about something similar back in 2007, responding to a point in one of his books where he was suggesting the best way to quickly get AI was for competitive US corporations to create it. I suggested essentially that AI produced through competition is more likely to have a bad outcome for humanity than AI produced through cooperation. I'd suggest the points there could be said about several current AI entrepreneurs. Someone I sent it to put it up here, and I will include a key excerpt below:
https://heybryan.org/fernhout/...
That said, other systems like, say, in the USSR have their own legacies of, say, environmental destruction and suffering (as with Chernobyl). So Capitalism has not cornered the market on poor risk management -- even though the ideal of any capitalist enterprise is to privatize gains while socializing risks and costs.
Here is one book of many I've collected on improving organizations (maybe of tangential relevance if you are thinking about organization improvement for your project):
"Reinventing Organizations: A Guide to Creating Organizations Inspired by the Next Stage of Human Consciousness" by Frédéric Laloux
https://github.com/pdfernhout/...
"Reinventing Organizations is a radical book, in the sense that radical means getting to the root of a problem. Drawing on works by other writers about organizations and human development, Frédéric Laloux paints a historical picture of moving through different stages of organizational development which he labels with colors. These stages are:
* Red (impulsive, gang-like, alpha male)
* Amber (conformist, pyramidal command and control)
* Orange (achievement, mechanistic, scarcity-assuming, cross-functional communications across a pyramid)
* Green (pluralistic, inverted pyramid with servant leadership and empowered front line)
* Teal (evolutionary, organic, abundance-assuming, self-actualized, self-organizing, purpose-driven)."
Maybe we as a society need to become Teal overall -- or at least Green -- if we are to prosper with AI?
Good talking to you too, same.
--------- From book-review-style email to Ray Kurzweil in 2007
To grossly simplify a complex subject, the elite political and economic culture Kurzweil finds himself in as a success in the USA now centers around maintaining an empire through military preparedness and preventive first strikes, coupled with a strong police state to protect accumulated wealth of the financially obese. This culture supports market driven approaches to supporting the innovations needed to support this militarily-driven police-state-trending economy, where entrepreneurs are kept on very short leashes, where consumers are dumbed down via compulsory schooling, and where dissent is easily managed by removing profitable employment opportunities from dissenters, leading to self-censorship. Kurzweil is a person now embedded in the culture of the upper crust economically of the USA's military and economic leadership. So, one might expect Kurzweil to write from that perspective, and he does. His solutions to problems the singularity pose reflect all these trends -- from promoting first strike use of nanobots, to design and implementation facilitated through greed, to widespread one-way surveillance of the populace by a controlling elite.
But the biggest problem with the book _The Singularity Is Near: When Humans Transcend Biology_ is Kurzweil seems unaware that he is doing so. He takes all those things as given, like a fish ignoring water, ignoring the substance of authors like Zinn, Chomsky, Domhoff, Gatto, Holt, and so on. And that shows a lack of self-reflection on the part of the book's author. And it is is a lack of self-reflection which seems dangerously reckless for a person of Kurzweil's power (financially, politically, intellectually, and persuasively). Of course, the same thing could be said of many other leaders in the USA, so that he is not alone there. But one expects more from someone like Ray Kurzweil for some reason, given his incredible intelligence. With great power comes great responsibility, and one of those responsibilities is to be reasonably self-aware of ones own history and biases and limitations. He has not yet joined the small but growing camp of the elite who realize that accompanying the phase change the information age is bringing on must be a phase change in power relationships, if anyone is to survive and prosper. And ultimately, that means not a move to new ways of being human, but instead a return to old ways of being human, as I shall illustrate below drawing on work by Marshall Sahlins. ...
One of the biggest problems as a result is Kurzweil's view of human history as incremental and continual "progress". He ignores how our society has gone through several phase changes in response to continuing human evolution and increasing population densities: the development of fire and language and tool-building, the rise of militaristic agricultural bureaucracies, the rise of industrial empires, and now the rise of the information age. Each has posed unique difficulties, and the immediate result of the rise of militaristic agricultural bureaucracies or industrialism was most definitely a regression in standard of living for many humans at the time. For example, studies of human skeleton size, which reflect nutrition and health, show that early agriculturists were shorter than preceding hunter gathers and showed more evidence of disease and malnutrition. This is a historical experience glossed over by Kurzweil's broad exponential trend charts related to longevity which jumps from Cromagnon to industrial era. Yes, the early industrial times of Dickens in the 1800s were awful, but that does not mean the preceding times were even worse -- they might well have been better in many ways. This is a serious logical error in Kurzweil's premises leading to logical problems in his subsequent analysis. It is not surprising he makes this mistake, as the elite in the USA he is part of finds this fact convenient to ignore, as it would threaten the whole set of justifications related to "progress" woven around itself to justify a certain unequal distribution of wealth. It is part of the convenient ignorance of the implications that, say, the Enclosure acts in England drove the people from the land and farms that sustained them, forcing them into deadly factory work against their will -- an example of industrialization creating the very poverty Kurzweil claims it will alleviate.
As Marshall Sahlins shows, for most of history, humans lived in a gift economy based on abundance. And within that economy, for most food or goods people families or tribes were mainly self-reliant, drawing from an abundant nature they had mostly tamed. Naturally there were many tribes with different policies, so it is hard to completely generalize on this topic -- but certainly some did show these basic common traits of that lifestyle. Only in the last few thousand years did agriculture and bureaucracy (e.g. centered in Ancient Egypt, China, and Rome) come to dominate human affairs -- but even then it was a dominance from afar and a regulation of a small part of life and time. It is only in the last few hundred years that the paradigm has shifted to specialization and an economy based on scarcity. Even most farms 200 years ago (which was where 95% of the population lived then) were self-reliant for most of their items judged by mass or calories. But clearly humans have been adapted, for most of their recent evolution, to a life of abundance and gift giving.
When you combine these factors, one can see that Kurzweil is right for most recent historical trends, with this glaring exception, but then shows an incomplete and misleading analysis of current events and future trends, because his historical analysis is incomplete and biased. ...
So, this would suggest more caution approaching a singularity. And it would suggest the ultimate folly of maintain[ing] R&D systems motivated by short term greed to develop the technology leading up to it. But it is exactly such a policy of creating copyright and patents via greed that (the so called "free market" where paradoxically nothing is free) that Kurzweil exhorts us to expand. And it is likely here where his own success most betrays him -- where the tragedy of the approach to the singularity he promotes will results from his being blinded by his very great previous economic success. If anything, the research leading up to the singularity should be done out of love and joy and humor and compassion -- with as little greed about it if possible IMHO. But somehow Kurzweil suggests the same processes that brought us the Enron collapse and war profiteering through the destruction of the cradle of civilization in Iraq are the same ones to bring humanity safely thorough the singularity. One pundit, I forget who, suggested the problem with the US cinema and TV was that there were not enough tragedies produced for it -- not enough cautionary tales to help us avert such tragic disasters from our own limitations and pride.
Kurzweil's rebuttals to critics in the last part of the book primarily focus on those who do do not believe AI can work, or those who doubt the singularity, or the potential of nanotechnology or other technologies. One may well disagree with Kurzweil on the specific details of the development of those trends, but many people beside him, including before him, have talked about the singularity and said similar things. Of the fact of an approaching singularity, there is likely little doubt it seems, even as one can quibble about dates or such. But the very nature of a singularity is that you can't peer inside it, although Kurzweil attempts to do so anyway, but without enough caveats or self-reflection. So, what Ray Kurzweil sees in the mirror of a reflective singularity is ultimately a reflection of -- Ray Kurzweil and his current political beliefs.
The important thing is to remember that Kurzweil's book is a quasi-Libertarian/Conservative view on the singularity. He mostly ignores the human aspects of joy, generosity, compassion, dancing, caring, and so on to focus on a narrow view of logical intelligence. His antidote to fear is not joy or humor -- it is more fear. He has no qualms about enslaving robots or AIs in the short term. He has no qualms about accelerating an arms race into cyberspace. He seems to have an significant fear of death (focusing a lot on immortality). The real criticisms Kurzweil needs to address are not the straw men which he attacks (many of whom are being produced by people with the same capitalist / militarist assumptions he has). It is the criticisms that come from those thinking about economies not revolving around scarcity, or those who reflect of the deeper aspects of human nature beyond greed and fear and logic, which Kurzweil needs to address. Perhaps he even needs to addres them as part of his own continued growth as an individual. To do so, he needs to intellectually, politically, and emotionally move beyond the roots that produced the very economic and political success which let his book become so popular. That is the hardest thing for any commercially successful artist or innovator to do. It is often a painful process full of risk. ...
I do not intend to vilify Kurzweil here. I think he means well. And he is courageous to talk [a]bout the singularity and think about ways to approach it to support the public good. His early work on music equipment and tools for the blind are laudable. So was his early involvement with Unitarians and social justice. But somewhere along the line perhaps his perspective has become shackled by his own economic success. To paraphrase a famous quote, perhaps it is "easier for a camel to go through the eye of a needle than a rich man to comprehend the singularity." :-) I wish him the best in wrestling with this issue in his next book.
