Red matter isn't that big a stretch from today's science. Strange Matter has fairly strong theory behind it and red is just another type of quark.

Besides I vaguely remember all the star trek episodes and definitely vaguely remember a series finale where Captain Bristow has to stop a big red matter ball from turning a whole city into zombies.

Actually it (thanks to 2s complement arithmetic) gives the right answer. "A" decrements down to MIN_INT then wraps round to MAX_INT then decrements all the way down to 0. It effectively calculates the unsigned equivalent of "a" (being 2**32 -a) times "b". The result of that is b*2**32 - b*a. The first part doesn't fit in your int, so you're left with -b*a. Of course, if you have 64-bit ints you may have wait a few decades.