Let's trust Wikipedia on bird strikes and assume that small objects (under about 10kg) rarely cause a catastrophic collision, mostly it looks like bird strikes and similar are survivable for planes, they just cost lots of money. Looks like most aircraft aren't going to fall out of the sky even faced with a drone operator who successfully crashes into a plane. However the photos show it can make a pretty mess of expensive jet engines.
So I suspect that commercial interest might also be at play, it would be in the airlines' interest to claim a terrorism threat to stop idiots going to the supermarket in the morning then flying a drone near commercial airspace in the afternoon. Going to cost a lot to replace one of those jet engines from the look of the wikipedia photos showing what happens when a bird hits them.
Seems like if you want to commit an act of terror then a 5kg lump of plastic isn't likely to knock an airliner out of the sky, but it will probably cost the airlines a lot of money so I can imagine they'd quite like some regulations in place to stop idiots flying them near their planes.