"Sorry, your honor, but I have forgotten the decryption password." or, actually, that would be stupid since it would imply they are his. It should be "Sorry, your honor, I never had the encryption password". Unless the FBI has concrete proof he actually decrypted them in the past, they're screwed. Go back to cracking away. I also want to know the encryption that they supposedly cracked. Unless it is junk, it is more than likely that they brute forced the encryption passphrase for that hard drive.
It's disgusting if he has CP, but something something plausible deniability.
It's like the IT worker with no morals who quits in an organization where he held all the passwords and then promptly claims to have forgot them all when the company demands them right after they quit. You cannot prove if they were forgotten or not.
Actually refusing to give them out is grounds for legal action in both instances.