I think operators of all types of power plants should have to pay for the costs they cause to society and which are so far externalized. Even if the payment is only partial, it can make the problematic technology less competitive and help others gain market share.
Mandatory insurance for the risk of nuclear accidents is a step in the right direction, if the insurance sum is realistic. For instance, Germany has mandatory insurance for nuclear power plants but only at a paltry 2.5 billion euros coverage per plant. Needs to be much higher.
For coal and gas-fired plants, I agree that there should be a mechanism for penalizing CO2 and mercury emissions. The EU has introduced such an instrument for CO2, the http://en.wikipedia.org/wiki/European_Union_Emission_Trading_Scheme. There seems to be no such thing in the USA yet.
See also http://en.wikipedia.org/wiki/Kyoto_Protocol where the USA, Canada and Russia are the least cooperative states .
At the same time, the USA are starting to do something about mercury emissions while the EU doesn't yet:
http://www.epa.gov/mats/powerplants.html
Not based on taxing the emissions but on emission limits. Emissions below the limits stay free. Still, it is something.
Gas when extracted by fracking probably needs its own regulation concerning fracking chemicals. And so on...
Which would leave renewables in a better position because they don't have most of the usual risks and harmful emissions. Most complains I hear are about birds crashing into wind turbines and turbines looking ugly (matter of taste, YMMD).