What? No. That's ridiculous. That an AI takes as a default going to do this, and is even capable of doing this underscores exactly how these systems can be a problem. If this a reasonable course of action for the AI, then we have some serious problems. What's the next step:

Person: AI, I don't want to go to Cathy's party. Her friends are so boring and I'm exhausted. Find me an excuse.

AI: I've caused a nuclear reactor melt down at the nearby reactor. Now the party won't happen because the whole region is being evacuated.

Systems which when given a goal will go out of their way to achieve that goal are exactly the sort of thing that people were concerned about a decade ago when discussing things like the Paperclip maximizer hypothetical. And people like Yudkowsky were mocked for discussing it then. Now it is happening and we're told that well, of course that's the idea would do, this is just over-hyped clickbait? Seriously?

Or, an alternative, they are genuinely concerned and are trying to balance their own financial incentives (which would be to rush out the models quickly) with the actual danger. If you prefer, look at what happened with Kimi. That model was open-weights, and a security firm testing it ended up with (in part due to their own sloppiness), a similar sort of containment breach to what Anthropic and OpenAI have reported https://www.wired.com/story/moonshot-kimi-k3-ai-model-escape-sandbox/. So that's an open model being reported to do the same thing by an independent security group with no incentive to lie. So one should take it seriously in their case. And if it is real for Kimi which is almost certainly weaker than any model being currently built/evaluated by Anthropic or OpenAI, that should cause you to take their own concerns about these new models more seriously also.

I suggest you go and read the link. It discusses why Occam's razor is a reasonable rule and also why it does not in fact put personal deities as a simple hypothesis when done correctly although it uses "witch" as the same type of category. In essence, some things like "witch" or "god" are types of items that feel very simple to human minds, but really aren't when you try to describe them precisely.

This is not accurate. Occam's razor has multiple forms that are carefully and don't run into the issue you have. See for example here https://www.lesswrong.com/posts/f4txACqDWithRi7hs/occam-s-razor for an introduction to some of the relevant ideas.

Dyson spheres, Dyson swarms, and similar objects all have a lot of advantages. In the same way, we don't live in houses made of sticks and stones because our current houses have advantages.

If you want to harness a lot of energy, or you want a really big civilization where everyone can talk with each other, then a Dyson sphere is better. If you spread out to multiple star systems, you have to wait for years between contact. It seems likely that a sufficiently advanced civilization would try both. And the white dwarf Dyson swarms would require much less material, engineering work, etc (probably orders of magnitude less).

That's a neat and plausible idea. And if that's the case, then the concerns I raised don't really fall out at all.

This is a good reason to suspect that asteroids are more common on worlds with life. But 65 million years, and millions of years before that show it just isn't that common. For a natural disaster to be the Great Filter it would need to be happening every few hundred years.

Because there's a substantial fraction of them, and the AI community, that are genuinely concerned. There's a lot of conflicting incentives here.

Also, a followup note: We've now seen an open-weight Chinese model do nearly the same thing https://www.wired.com/story/moonshot-kimi-k3-ai-model-escape-sandbox/ and this was seen by independent security researchers who have no reason to lie. So we have at least one case where this has happened where there's no incentive to claim it happened when it didn't.

That's possible, but the question is how likely is it? If you've gotten to be around for a million years, you've learned a massive amount, you've likely spread to multiple star systems, so you've got all of that. That means a collapse would have to wipe out multiple different parts of your civilization in multiple star systems. And at even a few thousand years, you start getting things like added resilience from terraforming/xenoforming nearby planets. Worst case time estimates for example say it would take a few thousand years to terraform Mars, so even at 10,000 years that gives you a lot of time to make backups https://www.planetary.org/articles/can-we-make-mars-earth-like-through-terraforming .

We've looked at a lot with our telescopes.. We've searched systematically for Dyson spheres and similar objects and not found any in a very large radius https://arxiv.org/abs/0811.2376 Now, it is true Dyson sphere like objects around a white dwarf would be easier to make (much less material involved and much less structurally difficult) https://arxiv.org/abs/1503.04376 . They would be harder to find with our current technology but we've started looking for them too and haven't found any either.

We know how common large scale *natural* disasters are. Not very common since the last one was 65 million years ago and the one before that was even longer ago. Disasters which are arise which can kill a civilization are therefore much more likely to be things that we do to ourselves. That doesn't mean we shouldn't be working on things like asteroid deflection of course, but it isn't the most likely scenario. Asteroids aren't out there hunting civilizations.

Huh? That's not an argument against a Great Filter. That's an argument to think that we're not near it yet. But it also isn't an argument that works that well. What matters less is being K0 and more being spread out. Once you're multiplanetary (in the genuine sense, not the oh-put a few people there) this goes way up, and we're likely only a few centuries form that at worst.

Climate change is bad, and we're not doing enough about it. But it isn't a great filter situation. We're unlikely at this point to hit the worst case scenarios for climate change; better modeling has suggested they are less likely than we initially thought, and separate from that, we're now adopting solar and wind and EVs at a fast enough rate that it is helping avoid the worst case scenarios https://theconversation.com/scientists-have-scrapped-the-worst-case-climate-scenario-because-action-is-making-a-difference-283675. But we can certainly do more there (I like donating to Everybody Solar which helps get solar panels for nonprofits like museums and homeless shelters https://everybodysolar.org/).

None of these are a Great Filter. These are in the worst case, bad things. They aren't things that are going to wipe out a civilization or preventing it from going to the stars. Not every bad thing is a filter, and not every thing we personally really dislike (of which I could add quite a few) is a filter. Things which add a few million deaths a year are tragic and need to be dealt with, but they aren't a filter for this purpose.

Life arising once should reduce the chance of life arising separately (or at least taking off after it arises) since you already have life around using a lot of the obvious resources. If life tried to arise again now for example, it would likely be quickly consumed by much more optimized living things. But even putting that aside, if the chances are independent, and the chance of life arising on an Earth-like planet is x, then the chance of it arising twice on the same planet should be x^2 . That means that if x is even a little small, x^2 is much much smaller. So this is strong evidence that x is not very small.

Doesn't really work. It helps explain no visitors, but it doesn't explain the lack of megastructures or evidence of the massive easily available energy being used in other galaxies. They all are very far away and look natural. And even for visitors it doesn't work well at a galactic level. Space is massive but things are really old. Our galaxy is around 100,000 light years across. That means if one is traveling from one end to another at only 1% of light speed, which looks pretty plausible as a speed one can travel at, then you can get from one side to the other around 10 million years, which is not that long from an age standpoint.

Is it for a civilization that's been around for a million years or two million years or more? And we can look at a lot. For example, look at this search for Dyson spheres https://arxiv.org/abs/0811.2376 and to be clear, this isn't just "Dyson sphere" in the sense of a solid block, which is unlikely to be possible, but also similar structures with large orbiting sections. There's been some speculation that building Dyson sphere like objects around a white dwarf would be easier, and would be harder to find with our current tech https://arxiv.org/abs/1503.04376 but we've started looking there and haven't found any either.

If this is accurate, this suggests that abiogenesis is easy once one has a stable, watery environment, since it happened twice on Earth. If it was difficult, we'd expect t see it exactly once (it has to happen at least once for us to see it so there's always an anthropic issue there). There was already an argument that we should expect life to be somewhat easy since if not, we'd expect it to have arisen somewhat late on Earth's existence, not soon after. Presumably, one way of seeing if life really is easy is to look at Europa; if we do end up finding life there that will be strong evidence.

Aside from this being cool There is however a worrying bit. One potential conclusion about the Fermi paradox and the apparent the apparent absence of other civilizations in the universe (and not just radio signals, but no visitations, no artifacts, and no sign any civilization in any galaxy is using a major part of its available energy, nor any signs of obvious megastructures in our own galaxy or nearby ones, or things like star lifting https://en.wikipedia.org/wiki/Star_lifting, that there is a "Great Filter" https://en.wikipedia.org/wiki/Great_Filter something that makes advanced civilizations very rare. There are two classes of potential filters, early filters, like life arising, multicellular life arising, development of specialized nerve cells, tool use, etc. Then there are late filters, which hit a civilization like nuclear war, bioengineered diseases, bad nanotech, and AI. While late filtering from natural causes is possible it looks unlikely since natural disasters on that scale just aren't that common. Evidence like this means less early filtration, and makes it more likely that there is a genuine Great Filter that we are going to need to pass through.

Personally, I used to be generally skeptical of AI being the Great Filter, because it seemed like if a lot of civilizations were being taken over by AI, we'd see evidence of that as the AIs spread out to the stars. If something spread at anything less than 99% or so of light speed, we'd have a lot of a chance to helplessly see it coming. However, the AIs were producing while dangerous seem less likely than I would have guessed to just take over everything in their future light cone, in part that their goal oriented behavior is limited, and they come across as more pseudominds than true minds with long-term planning and objectives. In that contex, the chance that AIs wipe out civilizations and wipe themselves out in the process or shortly thereafter is looking like a worryingly likely explanation.