There are different ways to solve this: you can configure either sshd (server-side) or ssh (client-side) not to use password authentication. Disabling password authentication on the server makes your server more secure, but you will be in trouble if you loose your key.
To make ssh (client-side) using pubkey authentication, add some options to the ssh command:
ssh -o PubkeyAuthentication=yes -o PasswordAuthentication=no -X git@server
If this works, you can set the PasswordAuthentication=no option permanently in the ssh client config file /etc/ssh/ssh_config system-wide or ~/.ssh/config user-specific (on details, see man ssh_config).