The problem is that ssh-agent running on your local machine is hanging, and preventing you from attempting to make another connection. I believe
ssh-agent should exit when your ssh connection is terminated and the process ends, but for some reason the OS X implementation hangs. This instance of ssh-agent isn't available to other instances of your user, for security reasons.
The solution, which I believe would apply if this was happening in Ubuntu as well, is simply to kill the ssh-agent process when you're done with your ssh session, before starting another one.
ssh-agent hanging turns out to be problem, you can automatically end it following your session with:
~$ ssh username@hostname && kill $(pgrep ssh-agent)
That command is if the client you are connecting from has only one user using ssh, if you are making connections from a machine with multiple, concurrent ssh connections from different users. You'll want to use:
~$ ssh username@hostname && kill $(ps -U your_local_username | grep ssh-agent | grep -oP "^\d+")