Maybe not in video, but I worked at an image processing place where we deliberately created a file called "bastard.tif". The purpose of this file was to exploit every aspect of TIFF we could find, and for those familiar with the standard you'll know that's a lot. We used non-standard pixel ratios, we switched encoding mechanisms multiple times through the file...we did everything we could to make a standards-valid TIFF that would crash everything.
Wasn't malicious, we were a commercial data processing shop and image creation/conversion was our thing. We could crash Photoshop (non-square pixels - this is early-to-mid-nineties, no idea if it still crashes it), we could bring down things like Kofax Libraries which at the time were fairly advanced pro image coding libraries. We could crash most Unix utilities for working with images...you get the idea. We definitely were thinking about it, and we were actually doing it. The idea was to know what we could and couldn't do whilst coding our image processing software - we didn't want to create a final image that, whilst technically valid, couldn't actually be used anywhere.