The whole point of some of these changes is to make the food no longer attractive (or possibly even toxic) to pests. It seems reasonable that the changes required to do this may have some impact on people as well.
In this case, we know exactly how we are making it pest resistant. The Bt genes produce a protein that has no known affect on mammalians. It isn't 'possibly' toxic to the pests it targets, it kills them. It is, to them, toxic, but just like grapes and chocolate are toxic to dogs, that does not mean it is also toxic to humans. The Bt proteins have a very specific and well understood mode of action, and they simply have no impact on humans.
direct genetic modification is a lot less likely to cause problems than the radiation-based mutation where they just blast it and see what they end up with
You don't even need to go that far. People all over are breeding for pest and pathogen resistant crops. What is being increased in those? All plants produce toxic chemicals, like solanine in tomatoes, falcarinol in carrots, or psoralens in celery. It is how they evolved to defend themselves from herbivory since they obviously can't fight back. Maybe the conventional breeding of a new variety of a crop results in increased defensed by increasing one of those compounds. Maybe it results in an increase of pathogenesis related proteins, known allergens. Thing is though, this is a much more nuanced view of risk assessment of improved varieties, transgenic or conventionally bred or anything in between, and the various anti-GMO groups are not in the business of educating people.