Well, I will agree with you that atomic bombs require an enormous investment, but your reasoning fails on one account: in an atomic bomb, the potential energy is already there, provided by nature. Sure we had to refine it, but in the end it is supernovas that put all that energy in the Uranium (or what have you) for us. To create an antimatter bomb, we need to produce all that potential energy ourselves, in the form of antimatter. Not only do we need to put in the potential energy itself, but also excess energy to account for the inefficiency of the production process.
That is what makes antimatter too inefficient to be used as a weapon, let alone as a fuel source.
This changes if we could harvest it from space, as indeed it would be nature again who has stored that potential energy for us.
You scratch my tape, and I'll scratch yours.