I have set up libpam-google-authenticator according to https://www.digitalocean.com/community/tutorials/how-to-set-up-multi-factor-authentication-for-ssh-on-ubuntu-16-04
It works, but the problem is that I can't log in with a password, it need to be with a key. If there is no key, I get permission denied public key without even being asked for a password. How can I get it to work with a public key + two factor OR password + two factor?