Not if the key is the public key of the current user account using resources on the server. Again, I'm out of my field of expertise, but it sounds like something feasible.
The terminal would be the only unencrypted part of the process, a terminal that could be anything ranging from another computer to a screen. All you have to do is to plug your hardware stored keychain to a screen with no data connection (expected encrypted video signal) to the server to get the unencrypted output.
That's going far, but that looks like a nice objective that could start with a processor that would allow a part of it.