Sure lax anything is a problem. If you are placing authorized_keys files that are wide open, to a wide open SSH that just sits around for years, ya I see a problem. If done right there are restrictions that can be added in an authorized_keys file
from="IP.address" - set a key to only be able to be accessed by a certain ip
command="some command" - only allow a certain command to be run.
I also feel that ssh should not be wide open if possible. IP restricted by either a firewall, tcp wrappers or AllowUsers in sshd_config.