ssh-add is the client part of
ssh-agent, it has nothing to do with granting access to hosts (which is done by adding the public key (
~/.ssh/authorized_hosts on the server side).
ssh-agent is a program which you use on the client side, the server side only needs
ssh-add -d, you have cleared a cached key from
ssh-agent such that you have to re-enter your SSH keyfile passphrase next time you ssh to a system. You won't lock yourself out by clearing this cache. Simply run
ssh-add ~/.ssh/id_rsa for a specific keyfile) to remember the keyfile passphrase again for this session.
-i option of
ssh is only necessary if your private key is located on a non-standard location.
ssh -i ~/.ssh/id_rsa user@host is nearly equivalent to
ssh user@host since
~/.ssh/id_rsa is one of the standard places where SSH looks for a keyfile.
The reason why
ssh-add ~/.ssh/authorized_keys prompts for a password is because it did not recognize an unprotected key, and thus assumed a password-protected key. In fact, the file has an invalid format so no password will work here.
See also the manual page