I agree that the mathematicians in the direct employ of NSA should take a long hard, look at their own ethical code, but the fact of the matter is, the NSA provides lots of funding for university mathematics departments. For research that is open to public scrutiny. From TFA, $4 million goes to a grant program administrated by the AMS and things like undergraduate research programs and number theory conferences. The NSA is just throwing money at mathematicians on the off chance that they discover something useful to national security.
If the AMS were to sever ties with the NSA, there goes $4 million of funding for public mathematical research in a puff of impotent outrage.
I'm all with you when it comes to not working for fascists, but we're talking about public research here, for the enrichment of all humanity. Not shady spying stuff.