SSH is a must use tool for system administrators. However, residing access security on a human entered password is not very wise.
Script kiddies may break into your system due to a lazy user with a weak password. And it is beyond the system administrator power to make users choose good passwords.
The good news is that there is a way to leave remote access open and have not to worry about passwords. The method consists on authentication via asymmetric cryptography. The user’s private key is the one that grants the authentication. You can even lock user’s account to disallow completely password authentication.
Another advantage of this method, is that one does not need different passwords to log on different servers. One can authenticate via the personal private key on all servers, needing not to remember several passwords.
It is also possible to make logins with no password asked with this method.