in fact two American destroyers were torpedoed (one sunk) by U-Boats in October 1941.
USS Reuben James.
We were escorting convoys of war material to the UK with US Navy ships, a clear violation of neutrality laws (both international and US).
Note, by the by, that the oil embargo against Japan was one of the few examples of actually following the neutrality laws at that time. Selling things to a belligerent was required to be "cash and carry" (they pay US dollars or gold, in advance, they shoip it home in their own bottoms). Japan wasn't actually capable of doing that even if they'd wanted to (not enough merchant hulls of their own, much less the hard currency reserves to pay cash).
Yes, we ignored the Neutrality Laws with respect to the UK and Germany (we liked the one, disliked the other), and we could have done the same for Japan (and probably would have if we'd liked Japan).
But we were under no obligation to break our own laws just to keep Japan happy.