I think Elon sees something that most of you do not. Artificial intelligence is not like anything else. We know very very little about the kinds of intelligence that are possible. But if it is possible to build AI that is smarter and more capable than us, then it will by definition be better than us at building the next generation of itself. And at that point, humans are permanently obsolete because we have no rapid methods for upgrading ourselves. It has nothing to do with who is 'using the AI' or 'Who is doing the prescription'. There will be no person and no human moral intuitions in the loop at all. The intelligence that supersedes us will be doing what it wants to do. We'll be like the fish who debate how to control their bipedal relatives who have decided to start overfishing the oceans. It is simply out of their control. And if that doesn't scare you, then you don't understand.

We don't know whether or not artificial intelligence is possible. But it seems like a very reasonable possibility sometime in the next few centuries. And we know so little about intelligence that we have very little idea about whether it will share anything like the moral intuitions that undergird human society. Many of us suspect those evolved for survival in hunter-gatherer tribes and AI will evolve a very different set of criteria upon which it makes its choices.

Maybe it matters even if we can't easily travel there. For example, how many super-nova candidates are within 1000 light years of home? Not all stuff is 'good stuff'. There are good reasons to be glad that we are in a pretty low density region of the galaxy.

That error jumped out to me also. Its like describing a city 93 miles away and instead saying it is 93 million miles away which instead of being 1.5 hour drive is all the way to the sun. It is really useful to get a cosmic distance scale in your head: billions of light years is the size of the visible universe, millions of light years are distances to nearby galaxies. 30,000 light years is the distance to the center of our galaxy. 4 light years is the distance to the nearest stars.

We just discovered that we are made of atoms a little over one century ago, and our ignorance is vast. But we should also be careful not to err on the side of blindly assuming that anything is possible. It is essential to think clearly about what might and might not be possible based on what we know now in order to direct our investigations. Will we discover new laws of physics that are relevant to releasing energy from nuclear reactions? I suspect the answer to that is probably no, and the reason is the high precision we achieve from our current theories in describing the behavior of atoms and nuclei. Careful experiments of nuclear excitation energies, fusion cross sections, etc agree with theoretical calculations, often to many significant digits. There just isn't much place to hide new physics in this energy range. Of course new fundamental discoveries (dark matter, etc) are very likely. They just are unlikely to change our predictions for nuclear phenomena by a quantitatively significant amount. Would it be better to stay open minded because one can never be sure? (See http://www.preposterousunivers...) Or is it better to make audacious, falsifiable hypotheses, such as the hypothesis that we already know the laws underlying the physics of everyday life? (http://www.preposterousuniverse.com/blog/2010/09/23/the-laws-underlying-the-physics-of-everyday-life-are-completely-understood/)

That doesn't tell us much about how to engineer processes that obey the known laws of physics. Predicting what humans will be able to do is very very difficult...and people regularly get it badly wrong being both too optimistic and too pessimistic. In my mind, good hypotheses based on careful consideration of the best evidence are never premature. They just might be wrong.

You imply that if string theory and fundamental particle physics must have practical relevance like the quantum theories of atomic and nuclear physics of 100 years ago. But they are dramatically different. At that point, they didn't understand what matter was made of. (And despite a few notorious quotes, the best scientists knew that they didn't know how to explain atoms and chemistry.) Now we can't find anything in our galaxy that deviates from our current theories. There just are not going to be any practical applications of the Higgs boson or dark energy (at least not for many thousands of years...). If you are among those who think that physics is discovering new fundamental laws and engineering is using those laws to understand and control phenomena that we care about, then your version of physics is ceasing to be relevant. Instead, physics is actually the attempt to explain and control the world we live in using our knowledge of the fundamental laws. That kind of physics is slowly taking over all of science and engineering.

Maybe we can try to help those ignorant of applied physics, but you may be right that they are hard to help...

There is a fantasy that lives on and on that physics is only the search for the fundamental rules of how the universe works. Physics does include the search for the most fundamental theory...things like trying to detect the higgs boson or understand dark energy. But those two pretty nicely define 'irrelevance' to the everyday lives of humans. If physics is only about the search for fundamental rules, then physics is essentially over as an enterprise with practical relevance. (See http://www.preposterousunivers...) But the overwhelming majority of physicists have long been working on applications of known fundamental physics to discover new emergent laws and new technological applications. Semiconductor and device physics is one of the great successes of 20th century physics and this achievement of fabricating gallium nitride with its large bandgap was a major advance, both in the fundamental science of crystal growth and in high frequency electronics as well as the production of blue light. This is exactly the kind of prize that should be given because we need the next generation of physicists to be finding fundamental problems that have practical relevance rather than using their talents on interesting but economically useless tasks like string theory. I predict that in the rest of the 21st cenury, there will be more Nobel prizes in physics given for biological, environmental, and neuroscience applications of physics than there will be for fundamental particle physics. If not, then the Nobel prize will be overshadowed by the Kavli prize or some other prize that recognizes accomplishments that have consequences for humans.

"The reviewers who decide which projects receive funding are risk-averse."

This hasn't been my experience. Reviewers and grant officers want to fund high risk/high reward science. But you are competing with others who have already tried a bunch of risky ideas and are only proposing the ones that happened to work. You basically have to make a significant discovery before you can be funded and then you can get funding to bring that idea to full bloom and hopefully fund a few risky projects on the side that will serve as the basis of the next grant proposal.

Most new ideas are bad ideas, so funding agencies have to have a pretty rigorous filter to sort out the promising ones. As a result, it will always be very hard to get funding to explore an idea before there is evidence that it is on the right track.

Individual choices to conserve resources are well known to be ineffective. Improvements in energy efficiency are also ineffective. Look up Jevon's paradox. Individual conservation mostly decreases the price of fuels which encourages others to use more. The main rational reason for an individual to decrease their energy usage is build the knowledge base and cultural values of conservation that can serve as a foundation for eventual society wide action. I know the libertarians are unable to think clearly about this subject, but there is simply no way to manage common resources like the atmosphere, rivers, and oceans without holding all polluters accountable for the pollution they emit.

Good to see I wasn't the only one who noticed the bad units. :) For those who missed it: Power is measured in Watts, which is energy per unit time (J/s). If you have 22 GW/hr, that is a rate of increase of power provided, so AvitarX calculated that after only 24 hours they would have 22*24=528 GW of solar power being produced, which is ridiculous, but that is what the post indicates.

I loved the Goat Bubble story. Thanks for linking to it.

We have been for a long time in the situation where the financial institutions are primarily extracting rents from producers rather that contributing to economic productivity. High frequency trading is simply a particularly obvious example of this. The situation is not particularly new. Those with wealth and power have always influenced the rules to their own benefit. The question is whether there are any counter-measures that effectively push people to contribute value rather than skim off value created by others.

Your model of the brain as multiple neural nets and a voter is a good and useful simplification. I think we still know relatively little about how accurate it is. You would expect evolution to have optimized the brain to avoid blind spots that threatened survival, and redundancy makes sense as a way to do this.

However, I wouldn't classify blind spots as 'no problem whatsoever'. If the simple model of multiple neural nets and a voter is a good one, then there will be cases where several nets give errors and the conclusion is wrong. Knowing what kinds of errors are produced after what kind of training is critical to understanding when a redundant system will fail. In the end though, I suspect that the brain is quite a bit more complicated that a collection of the neural nets like those this research is working with.

What does it take to specify the state of a car? Many would claim that the dimensions of all parts and the material properties of the alloys and composites is a pretty complete description. You don't need all the modern physics. You just need a model that is quantitatively adequate for the level of accuracy desired. Simulating the classical/solid mechanics of a full auto crash is still a bit beyond us, but we can simulate many parts of the process. Even before simulation, we developed an understanding that allowed major improvements in auto safety that was partly rooted in approximate theories and partly in phenomenology. That distinction between an approximate theory (classical mechanics plus solid mechanics) and a phenomenology (observing auto crashes and drawing conclusions about what the important processes must be) is very important. With human interactions, we have many pieces of phenomenology (some pretty complicated parts of this are taught as 'social skills' in pre-school). But we have nothing in the way of a theory whose approximations are understood and whose predictive power goes beyond the situations from which the phenomenology was derived. I would argue that predictions in social engineering are essentially extrapolations of phenomenology that might work and might not...you simply don't know until you do the experiment. Whereas in classical mechanics, you know much more about what is going to happen including the Lyapunov exponents for chaotic systems that tell you when your predictions will no longer be accurate.

Your last comment is about political control. You seem to mean that our phenomenology will improve as big data mines human behavior and we learn to use that data and those ideas to control others. We do need to be careful who controls that. Although I suspect that the degree of control achievable by such means is easy to overestimate.

This hope for an effective theory of human behavior is a reasonable hope. There might be a simple effective model that could describe human behavior without accounting for all of the parameters needed to fully specify the problem. It happens in many fields. We can very accurately predict fluid flows without worrying about the parameters of all the molecules. But for humans, it really is hard to imaging how it might work out. Consider the fact that once some humans have a theory of how humans behave, someone will start using it to gain a competitive advantage and then other people will start changing their behavior in response to knowledge that the theory is being used. It is a fascinating way to be unknowable...to be guaranteed to change as soon as anyone figures it out.

"Social scientists will be able to understand and predict the interactions of people the way physicists understand and predict the interactions of objects."

Many of us technical types would love for this line of inquiry to be fruitful. But to have a 'physics of people' you have to know the values of all the parameters needed to specify the current state of a person and you need to know all interactions of that person with the rest of the universe. Phrased like that you can see how ludicrous it is to dream of using the methods of physics for social science. Physics works because the fundamental constituents of the universe happen to be only a small number of particles whose interactions are amazingly simple. For example all electrons are exactly identical and interact via only 3 forces (with some uncertainties about effects on scales larger than galaxies and energies higher than trillions of electron volts). The hope for a theory of sociology is a false hope. The hope for a useful phenomenology might be more reasonable and big data can help.