The question is whether to lay down your own life or kill someone to preserve it.
The U.S. invading Japan would've involed a LOT of U.S. soldiers dying. And maybe more Japanese, depending on how strongly they resisted, than the atomic bombings anyway (remember, those were just 2 cities, and we're talking about conquering the entire home islands).
I don't think it would be an easy sell to all the families at home if you told them that X% of their husbands and fathers had to die to end the war. Not that the public was informed about The Bomb or given a choice, but yeah.
Pragmatism ain't pretty.