SSH security leaves a lot to be desired. Do your users all use ssh-agent? If not, they're probably using ssh keys with no passphrase, which can be stolen by anyone who gets read access to their local filesystem. At that point, the attacker can gain access to your system. If they do use ssh-agent, then the attacker needs to gain debug privilege on their local machine, but that's also not too hard. ssh-agent has no protection against a compromised host OS, for example, unless you set up PAM on your systems to require a second factor such as a U2F key (there's no SGX version of ssh-agent, for example).
If their private key is compromised, ssh doesn't have a global revocation mechanism, so you need to go and find all of the places where an authorized_keys file contains their public key. What is your revocation policy? Do you have a simple way for people to submit a compromised public key and automatically revoke it across your entire system?
By default (though, thankfully, now not the only option) the known_hosts file contains a good list of all systems that an attacker should look at next. Do you require that your users turn on the feature that stores hashes of the machines, or does any compromise of one of your users' systems lead immediately to the attacker knowing that they have compromised a key that gains access to your system.