0

I have a server, and I have a desktop that is behind an awkward network. In order to access my desktop remotely, I have it maintain a reverse SSH connection to my server, and then I can SSH to my server and SSH to the desktop via whatever reverse-SSH port it is using.

Specifically, my desktop is running a wee Bash script like this:

#!/bin/bash
while true; do
    ssh -X -R 19123:localhost:22 http://www.example.pro/
    sleep 1000
done

Then, on the server, the desktop is accessible using a command like the following:

ssh localhost -p 19123

This seems to work fine. The connection is maintained and re-established reliably, and I can run commands and the like on the server from the desktop, verifying that the connection is indeed alive.

However, after a while, like a few days, the desktop becomes inaccessible from the server. On the desktop, the connection is alive, but on the server the following error message gets returned:

ssh: connect to host localhost port 19123: Connection refused

Can anyone guess at what might be happening? And, more importantly, can anyone suggest how this can be prevented from happening, given that the server cannot be restarted regularly and the like?

9
  • In my answer, you don't have that set on both systems do you? I realized that I have it set on my server that is offering the connection to my remote with the reverse. – Terrance Jul 12 '18 at 20:33
  • Also, it looks as though your script might be killing and not reopening your ssh connection to your server. When it dies, do you actually check on your desktop to make sure that the ssh session to port 22 on your server is connected again? – Terrance Jul 12 '18 at 20:46
  • @Terrance I'm not quite sure what you mean by having it set on both systems. The desktop is running the script that attempts to re-establish the reverse-SSH connection if the connection goes down. The script doesn't have any code that kills the SSH connection that I can see. The idea is that it launches its loop, makes the reverse-SSH connection and then, if that connection dies for whatever reason, it pauses and then loops back to re-establish the connection. I have confirmed that the SSH connection is alive when it restarts like this, in that I can run commands on the server etc. – BlandCorporation Jul 13 '18 at 12:34
  • I have some questions here, why are you using a reverse connection then? What is your reason for it? You have established that ssh works. Do you really need a constant communication channel between two systems? Or could you simple set up a key exchange between hosts so that you don't need to type in a username and password then run CRON jobs when things are ready? – Terrance Jul 13 '18 at 13:18
  • The reason behind the ServerAliveInterval is so that there are keepalive packets being sent during idle times to prevent a connection from timing out. Since you establish one ssh connection one way, then have the other return it, both should need that setting to keep those connections from timing out. With my job there is only one thing I keep open constantly and that is a tunnel that I need through a bastion type server to bypass a firewall so that I can administer to servers on the opposite side. This is stuff that I use every single day. – Terrance Jul 13 '18 at 13:19
3

As @Terrance says, you should use a keepalive to keep the connection open. However, to increase stability, I would also recommend using autossh to monitor the ssh connections. As it's not installed by default, you will need to install it with sudo apt install autossh.

With it installed, you can then execute it like so:

/usr/bin/autossh -f -o "ServerAliveInterval 900" -TN -R30582:localhost:5724 tunnel@bobsrockets.io

This sets up an ssh tunnel to tunnel@bobsrockets.io. Note that any options that autossh doesn't understand get passed through to ssh itself.

It also doesn't allocate a pseudo-terminal, which saves on resources.

This is good, but we can go one better. To make this permanent and automatic, we can set up a systemd service. Create a file inside /etc/systemd/system with the suffix .service, and put inside it something like this:

[Unit]
Description=SSH tunnel

[Service]
Type=forking
Environment=AUTOSSH_PIDFILE=/var/run/ssh-tunnel/ssh-tunnel.pid
PIDFile=/var/run/ssh-tunnel/ssh-tunnel.pid
ExecStartPre=/bin/mkdir -p /var/run/ssh-tunnel
ExecStartPre=-/bin/chown username:username /var/run/sbrl-ssh-tunnel
ExecStart=/bin/sh -c 'until ping -c1 bobsrockets.io &>/dev/null && sleep 5; do :; done && /usr/bin/autossh -f -o "UserKnownHostsFile /home/username/.ssh/known_hosts" -o "IdentityFile /home/username/.ssh/ssh-tunnel_ed25519" -o "PubkeyAuthentication=yes" -o "PasswordAuthentication=no" -o "ServerAliveInterval 900" -TN -R30582:localhost:5724 -p 7261 ssh-tunnel@bobsrockets.io'

[Install]
WantedBy=network-online.target

A few things are of note here:

  • username should be replaced by the local user account that autossh will be running under
  • I disable password authentication, and enable public key authentication here, so that it can start without any outside intervention. Make sure you create a special account on the server (ssh-tunnel in my example) that does not have a shell for it to log in to, and ensure you set up the public key and test it before installing this service file.
  • -p 7261 is used to specify the target port on the remote server to connect to. Update it if needed, otherwise remove if not required.
  • I specify the location of the known_hosts file with the UserKnownHostsFile option, as when running as a systemd service autossh doesn't seem able to find it otherwise. Ensure you point it at a suitable known_hosts file so that the ssh connection can be opened automatically.
  • I use a bit of shell scripting (until ping -c1 bobsrockets.io &>/dev/null && sleep 5; do :; done) to ensure that it waits until we can reach the target host before starting autossh. I encountered issues otherwise.
  • -f is used here to get autossh to fork. This avoids a useless sh process from lying around.

Sources

  • I blogged about this here
  • I originally read this article, which explains some more of the different options of autossh in more detail, should you need them.
2
  • Thanks so much for all of that detail. I will investigate autossh based on what you've documented. I should be clear, however, that he issue is that the port is becoming inaccessible while the SSH session is fine and still alive. On the desktop commands can such can be run on the server just fine via the reverse-SSH connection, while on the server, a connection to the desktop cannot be established. The port is inaccessible. I'm trying to understand why this is happening and how to prevent it or how to get the systems to fix it automatically. – BlandCorporation Jul 12 '18 at 20:23
  • Ah, I see @BlandCorporation. It's most likely the issue that Terrance describes that you're running into. If not, it may be worth investigating the second source article I mention, as it includes some additional options for ensuring that a connection is kept open and healthy. – starbeamrainbowlabs Jul 13 '18 at 17:06
3

Have your ssh send a keepalive packet every so often to help.

In your ~/.ssh/config file add the following:

Host *
ServerAliveInterval 15

This should be set on both server and client.

You can change that number above to anything you want. This is set for every 15 seconds and my server connections seem to stay alive without any problems.

Hope this helps!

2
  • Thanks for that, but the issue is that the port is becoming inaccessible while the SSH session is fine and still alive. – BlandCorporation Jul 12 '18 at 20:11
  • @BlandCorporation It becomes inaccessible because the server that is running the ssh command disconnects from your client system and has to be reopened. Make sure you have this on both computers. – Terrance Jul 12 '18 at 20:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.