6

I have a PC behind a NAT which makes a reversed SSH connection to my Digitalocean VPC. I utilise this reversed SSH connection from home to login to my office PC (I am authorised to do so) and copy files and do other important things.

Although not often, I noticed that my office PC restarts (due to power failures etc) and breaks the reversed SSH connection it has made with my VPC. In these kind of cases, I am unable to connect from my home PC to my office PC.

I run the following script to make the reversed connection + dynamic proxy to anonymise my traffic ( As I am not required to share browsing information) generated at the office PC.

autossh -CD 8080 -i digitalOcean -R 8081:localhost:22 root@IPofDigitalOceanPC

There is no way I can run this script again on my office PC upon a restart as I am not physically there. In order to solve this problem I installed the following crontab.

Note: rev.sh file contains the above line. The certificate "digitalOcean" and rev.sh is located in Ubuntu home. Therefore, when I execute ./rev.sh in my Ubuntu terminal I obtain a dynamic proxy and also access to my DigitalOcean server. This method works 100%.

However when I install the crontab in the following method, My ubuntu PC never makes a Dynamic proxy. I can see this because when I check this proxy from Google Chrome, it says proxy is refusing connection.

Here are the cronjobs I tried as root's cronjobs. I also tried these as a normal user, still they didn't work.

@reboot bash /home/user/rev.sh 
@reboot /home/user/rev.sh 
@reboot cd /home/user && ./rev.sh

I then installed a crontab a several minutes before the current time and waited for it to execute.

24 12 * * * bash /home/user/rev.sh
24 12 * * * /home/user/rev.sh
24 12 * * * reboot 

These did not execute either.

I also tried 48 15 * * * bash /home/user/rev.sh >> test3 and */1 * * * * reboot -f >> test but the test3 and test have nothing. However the files has been CREATED!! by the crontab!

Please be kind enough to help me spot my mistake. There are many similar questions on this website on my issue. I have referred many answers hence but none of them seemed to help.

  • 1
    Could it be a problem with the PATH environment variable within the cron job? If you use the full path to autossh in your script does it work? You can find this path with which autossh. Also, does your script start with a shebang #!/bin/bash or similar? – Arronical Nov 17 '16 at 9:33
  • 1
    I confirm that full path doesn't work. I just tried it. Thanks for telling me about which. I didn't know that but wont forget it forever. I added the #!/bin/bash right now. But it seems it doesnt work either. How come the "34 14 * * * reboot" is not working either? that surprises me a lot! (I did it one minute behind current time). – Denis Nov 17 '16 at 10:00
  • 1
    I also tried "48 15 * * * bash /home/user/rev.sh >> test3 " and "*/1 * * * * reboot -f >> test "but the test3 anf test have nothing. However the files has been CREATED!! by the cronetab!! – Denis Nov 17 '16 at 10:18
  • 1
    Two suggestions (1) at the end of your commands in crontab, where you redirect the output >>test3etc can you add 2>&1? This will mean any error message also gets written to the file. Also (2), if you just want to start a programme when the PC boots, I suggest that you include the command in /etc/rc.local. Finally, could you edit the question and include the version of Ubuntu you are using? – Nick Sillito Nov 23 '16 at 21:35
  • Also, can you confirm if either of the files /etc/cron.allow or /etc/cron.deny exist (and if they do, what they contain)? – Nick Sillito Nov 23 '16 at 21:41
1

A better solution would be to make use of watchdog. watchdog is a daemon which will watch running processes and if they exit will automatically restart them.

Install watchdog on Ubuntu 16.04

sudo apt-get install watchdog

How watchdog works

The watchdog(8) daemon will execute scripts in /etc/watchdog.d with the argument test or repair. (see TEST DIRECTORY section of watchdog(8) man page). Your watchdog script handles those two arguments when it checks to see if a process is running and takes an action to repair it.

You can configure watchdog by modifying /etc/watchdog.conf (See watchdog.conf(5)).

Example watchdog.d script

Take for example /etc/watchdog.d/autossh_script (which has 755 permissions and is owned by root).

Note: you may need to customize $targetuser environment variable in the example script. sam is my username.

#!/bin/bash

targetuser=sam

runTest=false
runRepair=false

case $1 in
  test)
    runTest=true
  ;;
  repair)
    runRepair=true
    repairExitCode=$2
  ;;
  *)
    echo 'Error: script needs to be run by watchdog' 1>&2
    exit 1
  ;;
esac

if ${runTest}; then
  #run a test here which will tell the status of your process
  #the exit code of this script will be the repairExitCode if it is non-zero
  if ! pgrep autossh &> /dev/null; then
    #autossh not running; notify watchdog to repair
    exit 1
  else
    #autossh running; no action necessary
    exit 0
  fi
fi

if ${runRepair}; then
  #take an action to repair the affected item
  #use a case statement on $repairExitCode to handle different failure cases
  su - ${targetuser} -c 'nohup autossh -f -- -NCD 8080 -i digitalOcean -R 8081:localhost:22 root@IPofDigitalOceanPC'
  exit 0
fi
  • I added -N to your ssh command in the example script so it only starts the tunnel but does not attempt to create a login shell.
  • I added -f to autossh so that it runs in the background.
  • The example script uses a simple pgrep pattern. However, you can further narrow the test to a specific user or even use a pattern for the process. See pgrep(1) for how to further customize your test using pgrep.

Configuring watchdog

/etc/watchdog.conf and /etc/defaults/watchdog are places to configure watchdog. See watchdog.conf(5).

One thing to note is user scripts are executed once every second by default. I recommend increasing this to at least 30 seconds unless you have a need for more realtime checking. Adjust the interval setting in watchdog.conf.

Troubleshooting

You may need to create the /etc/watchdog.d directory before your script.

/var/log/watchdog/* contains watchdog related logs and errors. If your script outputs to stdout or stderr then it will be written there. On my system I notice my script executes test or repair roughly once every second. If you use echo in your script it should only be temporary for debugging purposes only. Otherwise, discarding output is recommended except in the case of errors.

If your script is not running at all then check the permissions: ls -l /etc/watchdog.d.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.