You do get to play the CYA card, but only a finite number of times. Eventually, people get tired of it. The threats won't end before you run out of cards, so eventually you've got to figure out how to take a risk. Having demonstrated an abundance of caution will not save you from criticism should one of those attacks finally materialize. Nor, of course, will it save you from the attack.
Which is what counts in the end. I'm not sure that the CYA really saves his job, one way or the other. George W. Bush didn't lose his job despite a whole bunch of claims that he was warned. I, personally, don't consider those claims sufficiently specific to blame Bush for failing to prevent it (and I assure you, I am no fan of his). I think that a superintendent who handles things well in the aftermath (provides appropriate levels of counseling, makes sufficiently brave and consoling statements to the press, puts some kind of action plan into place but avoids accusations of security theater) would be lauded as a hero.
This is all a little vague since I don't know what the "credible" threat is. I am a bit hard pressed to imagine a threat that is specific and detailed enough to be credible but so broad that you have to shut down every school to counter it. I'm willing to extend some benefit of the doubt, at least for the moment, as part of my larger point that sooner or later one of these threats will be genuine and still not cost his job if he mis-reads it.