How to debug sshd not running at startup on server

Question

I have a (virtual) server running an Ubuntu 10.04 minimal system. I seem to have misconfigured sshd so that it does not start up on reboot. Since this is a server, I now do not have shell access to my server anymore. Any ssh request results in this:

ssh: connect to host xxx.xxx.xxx.xxx port 22: Connection refused

And /var/run/sshd.pid does not exist.

I do have access to the file system through a Virtuozzo Panel the server provider has installed.

I have not edited the /etc/init.d/ssh start script but I did copy it to /etc/rc2.d/ssh in hopes that this would start ssh at reboot.

I also tried adding a startup call to /etc/crontab:

7 * * * * root /usr/bin/touch /root/cron_is_running

8 * * * * root /etc/init.d/ssh start

Neither do I get the file /root/cron_is_running nor is sshd starting up

My logfiles do not provide much in the way of error messages for ssh, f.e. /var/log/syslog only shows a lot of named stuff at startup and some mysql upgrade stuff, that is surely not related.

I appreciate any help, I already configured a bunch of things on that server and by installing a new image I would loose all data on the server.

Update: Changing the script in /etc/rc2.d/ssh to /etc/rc2.d/S90ssh did not have any effect. /var/log/auth.log does not contain any lines that hint at ssh trying to start up, it is full of these lines however:

Update 2: As suggested I also tried adding /usr/sbin/sshd to /etc/rc.local. But a test with touch /tmp/test seems to suggest this file is not executed despite +x flag being set. I also tried adding sleep 10 at the beginning of the rc.local script because it was suggested elsewhere that this script had concurrency issues.

Final Update Thanks to the below answers again. They were huge help in fixing the missing sshd startup in the rcX.d/ directories. The final fix was to start a "Repair" from the Virtuozzo Panel, this mounts the original system as part of the filesystem of a recovery system with the original network settings. I could log into that with ssh. I copied the working ssh configs from the recovery system over the ssh configs of the broken original system, finished the recovery and now I can log in again.

Answer 1

Your attempt at copying to /etc/rc2.d/ssh was almost right.

In order for a script in /etc/rcX.d to be run at startup, it has to be called SxxZZZ where S is a literal S (for Start; you can also use K for Kill at shutdown time), xx is a number representing where in the order of startup it runs (01 first, 99 last), and ZZZ is your script name.

So, you'd want something like S90ssh. The number isn't critically important, 90 should be ok, the network will be up by then etc.

Note it will also have to be set executable - I don't know if the system you're using will allow you to do that? Hopefully copying the existing ssh script and simply renaming will keep the executable bits set.

As an alternative, /etc/rc.local can be used as a generic all-purpose startup script. Simply drop /usr/sbin/sshd in there before the exit line to get sshd started as a last resort.

Give that a go and update your question if you're still having trouble :)

You can check in /var/log/auth.log (/var/log/syslog does not contain sshd messages by default) for a line like:

sshd[18838]: Server listening on 0.0.0.0 port 22.

to see if it's starting.

Answer 2

If you go to /etc/init.d/ type sudo service ssh start and it comes clean (no faults) you only need to run sudo update-rc.d ssh defaults to make the service come up on the right levels and be killed when your system is going to shutdown.

You can also do that manually copying the script and renaming it to either indicate that it should be stopped or started on that level. Here are the files that are created with the update-rc.d command:

   /etc/rc0.d/K20ssh -> ../init.d/ssh
   /etc/rc1.d/K20ssh -> ../init.d/ssh
   /etc/rc6.d/K20ssh -> ../init.d/ssh
   /etc/rc2.d/S20ssh -> ../init.d/ssh
   /etc/rc3.d/S20ssh -> ../init.d/ssh
   /etc/rc4.d/S20ssh -> ../init.d/ssh
   /etc/rc5.d/S20ssh -> ../init.d/ssh

So a simple cp /etc/init.d/ssh /etc/rc0(1,6).d/K20ssh and cp /etc/init.d/ssh /etc/rc2(3,4,5).d/K20ssh should do the trick. (note that you have to run cp for each rc.d folder, the command is just an example of what name to give to the copied files).

K20ssh is run when the machine is going to shutdown and S20ssh is the name of the script for the levels where it should run.

S and K indicate start or kill and the number after it indicates the order on which the scripts inside the folder will run, a lower number makes run earlier and a higher number makes it run later, in case of ssh it should really not mather but 20 its the default assigned by update-rc.d.