You seem legitimately interested in hearing a perspective different from your own, so I'll oblige. Note that libertarianism is diverse, like any political affiliation, so not everyone who identifies with that label would agree with my responses.
Insurance is a common libertarian answer in this situation, and one that makes sense to me.
i guess he could take an insurance policy in theory, but even if an insurer were willing to cover this, the premium if correctly computed would probably be more than he could afford
Then he can't do it. If the fair premium is higher than the benefit of the project, that tells you that the total benefits of the project are lower than the total costs, so the project should not be undertaken. If he thinks the premium is unfairly high, or if nobody will cover it due to the size and uncertainly, he could work to build up significant evidence of the safety of the proposed activity (which is exactly what most people would probably want to see before he started anyway, libertarian or not).
so he would just go ahead and do it anyway.
This seems like an argument that applies to any crime under any political system... but if he did that in a libertarian society, then anyone who was potentially going to be affected would be eligible to defend themselves by preventing him from doing it. Note that people could delegate their self-defense, so even someone who lives far away and/or has a busy life already could still exercise their right of self-defense without undue inconvenience.
what would happen in the real world is, of course, that private interests would have this guy arrested and maybe worse. but that's initiation of force
I don't know of many people who consider self-defense to be initiation of force. Of course people would only be justified in using as much force as reasonable to stop him... initiating an experiment like this would not be carte blanche for dialing up the assassins. (This just follows from current common-law precedent for justifying self-defense.)
libertarians would have to admit that private prisons would still exist in their paradise
Yep. The main problem I personally have with prisons is not their existence but the high number of people locked up for acts of non-aggression (e.g. marijuana possession). Also note that, in the hierarchy of monopoly government institutions that libertarians want to get rid of, the justice system is typically near the end of the list because of the problems inherent in having two people with opposing interests (plaintiff and defendant) jointly select from multiple competing private court/prison systems.
you could say that the entrepreneur is "initiating force" by doing something very risky, but that's a definition which would admit many of the government regulations we have today.
"Regulations" are not necessarily against libertarian principles. After all, most libertarians want to live in stable modern society too, and are against direct aggression such as theft as well as indirect and/or probabilistic aggression like pollution of other people's property or reckless endangerment (e.g. driving drunk or attempting large-scale unproven geo-engineering experiments affecting other people's property). Remember not to confuse us with anarcho-capitalists!
I appreciated your reasonable tone when referring to libertarian principles so I was actually willing to respond, unlike in most slashdot flamefests. Hope you found the perspective interesting at least.