Lumping lots of things together and calling them "GMOs" is like lumping things together and calling them all "chemicals". Sure, there are chemicals that we need to ingest. Does that mean all chemicals are safe to ingest?
We need to be talking about - and the manufacturers and/or government need to be certifying the safety of - each specific genetic modification. Some might be harmless to ingest, but have other side effects, like propagating the use of pesticides that are wiping out bee populations or other similar environmental damage. Show me that the changes you made have benefits that outweigh side effects that will affect consumers or dump on the commons, and I'd approve and/or eat them. I just don't see that happening, which is my concern.
And people lumping all possible genetic modifications together and saying "GMOs are safe you are idiots for doubting science!" scare me just a little more than hippies saying "All GMOs are evil!" Both groups are stupid, but while the latter might impact global food scarcity at some time in the future, the former give a blank check to manufacturers to do things that might hurt me or my family now.