Skeet/trap clays take 7 or so pellets hitting it to break nicely. Competitive trap shooters with great handicaps stand 27 yards back from the bunker, clay isn't visible above the bunker until it has traveled another 5 or 6 yards... And it is moving at about 60mph out of the gate. Then ya gotta find it, get a good sight picture, and then shoot it. Granted, the target is flying mostly away from you, not complete left to right (or right to left) like the center stations when shooting skeet.
Using a larger shot size 50 to 60 yards isn't unreasonable for a practiced shooter to get enough hits to damage a drone. Damage enough to bring it down? Dunno... but multiple shots would almost guarantee it.