The actual evidence we have is that, as a rule, for organisms on Earth, extinction is the norm rather than an exceptional event. The history of life on Earth is one of repeated mass extinctions, and continuous extinctions otherwise.
The idea that humans are "special", that in some way the rules of life on Earth do not apply to them, is attractive, and it probably has some merit. But in order to counter the actual evidence of Earth's history, all you really have is a sort of narrative about what humans are like and would do. It's as related to the real probability of human extinction as verbal arguments about "what you would do in a fight" are to actual combat.
A billion years is a very long time, and it's easy to imagine scenarios which, however unlikely, cause human extinction. A genetic disease which disrupts reproduction, that we all already have and so cannot isolate. The astronomical cataclysms you mention. Heck, our understanding of the structure of matter is pretty basic and dates to within the last 100 years--do you think that perhaps there are possible material instabilities that we don't yet understand that could somehow result in such a cataclysm?
We humans have exterminated many other species. Other forms of life we encounter may return the favor, for their own inscrutable reasons.
It even remains to be seen whether human beings can live in a self-sustaining way on planets that are not inherently habitable. When we have a thriving population on Venus, you can make the argument that something that makes Earth uninhabitable (for example, an atmospheric revolution by novel organisms, as has already happened in the planet's history) won't cause human extinction. But until then, you just can't say. And "I don't know" is a lot different than "I know it can't happen."