I also find it hard to believe that the spammers have not figured this out. It's not like they are stupid. They try very hard to deliver their payloads. It would be trivial to update their software to retry messages that receive those codes.
Actually, some have. I started greylisting about a year ago, initially with a 1200 second interval. It cut the amount of spam actually delivered to the filters by 90%. Experimentally, I cut the delay period to 60 seconds and the numbers stayed steady, implying that none of the bots were retrying.
Last week, I saw a big run that obviously implemented retry. The logs said they retried at 15 minutes. I went to a 20-minute window and saw multiple retries. Then I changed the retry message to remove the "greylist" term (keeping the "Try again in x seconds") and the throughput is back down to a few percent of attempts, even as I cranked the delay back down to 120 seconds.
I did report the waves to the ISC handlers lists, and one of them confirmed that at least two botnets are confirmed greylist-aware.
That being said, I have had problems with a couple of legitimate vendors whose mail servers apparently don't understand a 451 status. I had to move them to using my Gmail address.