Overall I think GM food could be done well and would be a benefit to society. However, they have been some terrible examples so far.
There is a lot of fear of GM foods, but when your body digests food from a GM crop, the DNA is broken down into mononucleotides. The original structure of the DNA is lost and resulting mononucleotides would be the same weather the original food was from a GM and Non-GMsource (because DNA is DNA). I have not heard of, or seen any studies showing any difference at this level. But this is where a lot of the fear-mongering over GM foods reside.
For the vast majority of people there should be no adverse impact fromGM foods. If you live in the US and have had corn chips without getting sick, congratulations you are (most likely) living proof that GM foods can be safe for humans.
So the DNA is not an issue, but the proteins created by the DNA must also be digested and metabolized and this could be an issue for people with food allergies. People can be deathly alergic to some proteins, e.g. peanuts.
This presents a significant problem for GM foods, especially with trans-genic modifications. Trans-genic modifications take a DNA sequence from one species and put in a completely unrelated species.
If you were allergic to fish or eggs, it may not be enough to stick to a vegetarian diet. It is possible that a GM crop could use a DNA sequence from a food you are allergic to and put it in a food you would never expect the allergen to occur. Here in the US there is NO labeling requirement and this makes it devilishly hard to trace back from an allergic reaction to the food that actually made you sick.
I think it is a good idea to require GM foods to be labled and without labels, I personally think it is reasonable to oppose them outright.
I really would love to see a process that allowed consumers to find out a bit more about the GM foods, specifically what species were used for the DNA in the GM food. This would allow people who do have food allergies to protect themselves. But I think this is unlikely in the world of I.P. and trade secrets.
The second issue I have with GM crops is the environmental impact caused by certain crops. For example BT corn contains A DNA sequence from the bacteria Bacillus thuringiensis which allows the plants to produce a protein lethal to common pest insects. In essence the plant makes it's own insecticide. Some early studies showed that the pollen contained significant quantities of the protein and since corn is wind pollinated the possibility of unintended consequences is substantial. More recent studies have dismissed most of the concerns. Still, I'm hesitant to declare victory because we are seeing a loss of Monarch butterflies and song birds in America. There are surely many contributing factors and the presence of GM crops may not be a direct cause of these losses. However it is quite possible that this is a contributing factor.
A similar case can be made for GM crops that make the plants resistant to herbicides.
The ecological concerns are harder to prove because the environment does not limit itself to a single cause when producing the effects. But sorting out multiple causes is very hard and so most studies try to reduce complex systems down to simple, repeatable, mechanisms.
The one thing that is clear, the law of unintended consequences does not appear to have any difficulty scaling to the level of human activity. As we do more and bigger thing, we get more and bigger consequences.
It this keeps up, it may some day be a good idea to choose precaution over profit, but I fear this too is unlikely.