This sounds like a job for a reverse SSH tunnel.
I'm assuming from your question that Server B is on a network behind a firewall you don't have control over- in this case port-forwarding on the router wouldn't be possible. (If you do, you might want to look at going that route).
In a reverse tunnel, you tunnel the connection to B via the publicly accessible A. To do this, you SSH from B -> A with a port-forward specified on the command line.
ssh -f user@server-a -L 2222:server-a:22 -N
Then SSH from Local -> A with your forwarded port specified:
ssh -p 2222 user@server-a
... which would connect you to server B.
You might also want to use
autossh if you want to automatically create and keep open your connection.