Just solved this issue for myself.
Generate a public key pair on the machine virt-manager is used. Note that this is placed into the default key file. If you already have one it will ask you to override. If you do you will have to distribute this key everywhere you used it previously.
# ssh-keygen -t rsa
Remote login without a password, or with a passphrase, requires an SSH key to be distributed to the systems being managed. Use the ssh-copy-id to do this. Change root to be the user you want to log into the daemon as, logging in as root is risky. I use my local user on the host machine. Replace id_rsa.pub with the public part of the key you generated above. id_rsa.pub is the default. You will be asked to log into the remote machines via ssh for the copy to take place. This is your users (in my case root) ssh password not the pass-phrase of the key you just created.
# ssh-copy-id -i ~/.ssh/id_rsa.pub firstname.lastname@example.org
Now try logging into the host machine, with the ssh. Check that .ssh/authorized_keys file to make sure unexpected keys have not been added. A simple cat of the file to see if its the same should do.
Do this step for all other systems you wish to log into via virt-manager from your current system.
Now to make things automatic in the GUI you need to add the key to the ssh-agent so that it can log in for you.
# ssh-add ~/.ssh/id_rsa.pub
Now you should be able to shh into the machine through a terminal and not be asked for any passphrase or password.
Now try virt-manager.
If your getting issues with the agent failing to authenticate you, check out the last link in the sources with regards to a bug that hit me at this stage.
Managing Virtual Hosts:
Seems that ssh-agent can crash at some points and not accept keys from calling ssh-add for some reason. Not clear on this however the suggest setting of the SSH_AUTH_SOCK=0 helped get past this for me.
Once the key and its linked into the agent you should automatically be logged in via virt-manager.