Crossing bacterial genes with corn genes is not quite the same thing as mixing and matching corn genes from one variety with another. When we take a gene from bacteria and insert it into corn we are creating a quite unnatural thing and making a whole lot of assumptions about our understanding of genetic language in the process. Maybe it was safe to do, maybe it was not. With selective breeding and hybridization, you are at least starting with genetically compatible material. You also have millions of years of history demonstrating this to be reasonably safe. Nature prevents humans from impregnating hippos, but GMOs are effectively doing just that as well as things far more perverse.
For many of us, we feel an abundance of caution is merited. Given the players involved, it's really hard for us to simply accept their statements of "trust us." There's no independent verification of safety, just blind faith that Monsanto and co. aren't employing a calculus with our health and their profits.