That same, "um, not me," is all that's necessary with my bank's debit cards, as well. I suspect that it's because the card is also a Visa card. They called me three weeks ago with a suspicious transaction. I said, "um, not me," and that was it. The $817.43 charge was gone, the card number revoked.
The worst part was that I had to wait about ten days for a new card/number.