32

My VPS wasn't rebooted for about 3 months. It is hosted on a server with OpenVZ virtualization type and the operating system is Ubuntu 16.04. For some reason, I rebooted the VPS and after that, I couldn't connect to the server through ssh, the message that I received is:

ssh: connect to host srvname.com port 22: Connection refused

So I opened a Serial Console on the VPS and start investigating... I've purged and reinstalled the openssh-server with no success. I spent two hours reading articles, question, and answers about similar issues on Internet.

Finally I managed to understand that the directory /var/run/sshd is not created during the system startup. And once I create it manually I can start the SSH service without any problem, but on the next reboot the issue remains. So my questions are:

  • What could be the cause of this issue? Why /var/run/sshd is not created during the system startup?

  • How can I solve the issue in a proper way? I found a temporal solution that is mentioned at the end of this post.

  • Does the issue could be related to the OpenVZ host of the VPS? Should I ask the hosting provider to solve it?


The output of systemctl status ssh.service, sshd -Ddp 22 and journalctl -xe is:

# systemctl status ssh.service
● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
   Active: failed (Result: start-limit-hit) since вт 2019-01-15 12:58:08 EET; 22s ago
  Process: 407 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=255)

яну 15 12:58:07 srvname systemd[1]: Failed to start OpenBSD Secure Shell server.
яну 15 12:58:07 srvname systemd[1]: ssh.service: Unit entered failed state.
яну 15 12:58:07 srvname systemd[1]: ssh.service: Failed with result 'exit-code'.
яну 15 12:58:08 srvname systemd[1]: ssh.service: Service hold-off time over, scheduling restart.
яну 15 12:58:08 srvname systemd[1]: Stopped OpenBSD Secure Shell server.
яну 15 12:58:08 srvname systemd[1]: ssh.service: Start request repeated too quickly.
яну 15 12:58:08 srvname systemd[1]: Failed to start OpenBSD Secure Shell server.
яну 15 12:58:08 srvname systemd[1]: ssh.service: Unit entered failed state.
яну 15 12:58:08 srvname systemd[1]: ssh.service: Failed with result 'start-limit-hit'.


# $(which sshd) -Ddp 22
debug1: sshd version OpenSSH_7.2, OpenSSL 1.0.2g  1 Mar 2016
debug1: private host key #0: ssh-rsa SHA256:...
debug1: private host key #1: ssh-dss SHA256:...
debug1: private host key #2: ecdsa-sha2-nistp256 SHA256:...
debug1: private host key #3: ssh-ed25519 SHA256:...
Missing privilege separation directory: /var/run/sshd


# journalctl -xe
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit ssh.service has begun starting up.
яну 15 13:21:21 srvname sshd[1688]: Missing privilege separation directory: /var/run/sshd
яну 15 13:21:21 srvname systemd[1]: ssh.service: Control process exited, code=exited status=255
яну 15 13:21:21 srvname systemd[1]: Failed to start OpenBSD Secure Shell server.
-- Subject: Unit ssh.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit ssh.service has failed.
-- 
-- The result is failed.
яну 15 13:21:21 srvname systemd[1]: ssh.service: Unit entered failed state.
яну 15 13:21:21 srvname systemd[1]: ssh.service: Failed with result 'exit-code'.
яну 15 13:21:22 srvname systemd[1]: ssh.service: Service hold-off time over, scheduling restart.
яну 15 13:21:22 srvname systemd[1]: Stopped OpenBSD Secure Shell server.
-- Subject: Unit ssh.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit ssh.service has finished shutting down.
яну 15 13:21:22 srvname systemd[1]: Starting OpenBSD Secure Shell server...
-- Subject: Unit ssh.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit ssh.service has begun starting up.
яну 15 13:21:22 srvname sshd[1691]: Missing privilege separation directory: /var/run/sshd
яну 15 13:21:22 srvname systemd[1]: ssh.service: Control process exited, code=exited status=255
яну 15 13:21:22 srvname systemd[1]: Failed to start OpenBSD Secure Shell server.
-- Subject: Unit ssh.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit ssh.service has failed.
-- 
-- The result is failed.
яну 15 13:21:22 srvname systemd[1]: ssh.service: Unit entered failed state.
яну 15 13:21:22 srvname systemd[1]: ssh.service: Failed with result 'exit-code'.
яну 15 13:21:22 srvname systemd[1]: ssh.service: Service hold-off time over, scheduling restart.
яну 15 13:21:22 srvname systemd[1]: Stopped OpenBSD Secure Shell server.
-- Subject: Unit ssh.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit ssh.service has finished shutting down.
яну 15 13:21:22 srvname systemd[1]: ssh.service: Start request repeated too quickly.
яну 15 13:21:22 srvname systemd[1]: Failed to start OpenBSD Secure Shell server.
-- Subject: Unit ssh.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit ssh.service has failed.
-- 
-- The result is failed.
яну 15 13:21:22 srvname systemd[1]: ssh.service: Unit entered failed state.
яну 15 13:21:22 srvname systemd[1]: ssh.service: Failed with result 'start-limit-hit'.

The content of /usr/lib/tmpfiles.d/sshd.conf and /etc/init/ssh.conf is:

# cat /usr/lib/tmpfiles.d/sshd.conf 
d /var/run/sshd 0755 root root

# cat /etc/init/ssh.conf | sed '/^#/ d'

description "OpenSSH server"

start on runlevel [2345]
stop on runlevel [!2345]

respawn
respawn limit 10 5
umask 022

env SSH_SIGSTOP=1
expect stop

console none

pre-start script
    test -x /usr/sbin/sshd || { stop; exit 0; }
    test -e /etc/ssh/sshd_not_to_be_run && { stop; exit 0; }

    mkdir -p -m0755 /var/run/sshd
end script

exec /usr/sbin/sshd -D

Additional information about the system:

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.5 LTS
Release:    16.04
Codename:   xenial

# uname -a
Linux srvname 2.6.32-042stab127.2 #1 SMP Thu Jan 4 16:41:44 MSK 2018 x86_64 x86_64 x86_64 GNU/Linux

# apt show openssh-server | grep 'Version'
Version: 1:7.2p2-4ubuntu2.6

The temporal solution: I found that /var/run is a symbolic link to /run, I do not know why this is needed, but when I modified the content of the file /usr/lib/tmpfiles.d/sshd.conf from:

d /var/run/sshd 0755 root root

to:

d /run/sshd 0755 root root

everything goes well on system startup, the SSH service is started normally and I'm able to log-in via SSH.

1
  • This problem may suddenly appear after a reboot because of a version upgrade that was done right before that reboot, as described in this linked question. The lesson: don't upgrade unless you're sure your kernel can support it.
    – Snow
    Sep 21, 2019 at 0:04

6 Answers 6

45

I found this is a bug with the current version of systemd and old kernels that are used by some VPS privdes as it is in my case. This bug appears time to time, as we can see on Launchpad: Bug #45234, Bug #1811580; or on ServerFault: Why am I missing /var/run/sshd after every boot?

There are few workarounds of this issue, they all come together to alternative way to create /var/run/sshd before running the SSH server. Here are three possible solutions.


Workaround 1: Modify /usr/lib/tmpfiles.d/sshd.conf in the following way:

d /run/sshd 0755 root root

As it is mentioned in the question, /var/run is a symbolic link to /run, the final result is identical: /var/run/sshd is created. I do not know why, but this works.


Workaround 2: Use Cron job that will create /var/run/sshd and restart the SSH server, you can use the root's crontab for this purpose - execute sudo crontab -e and add the following entry:

@reboot mkdir -p -m0755 /var/run/sshd && systemctl restart ssh.service

Currently I'm using this solution, so it is also tested.


Workaround 3: Use /etc/rc.local to do the same as the above, as it is shown in this comment on bug report #45234.

8
  • 3
    Thanks, that fixes ssh but not the broader problems of systemd being broken. Try running systemd-tmpfiles --create and see all the errors
    – paulzag
    Jan 22, 2019 at 11:32
  • 1
    You are right, @paulzag, but in my case I"m sure the general problem is the old kernel. I decided to ignore these errors that systemd-tmpfiles --create shows, because at the moment there are not any sensible malfunctions on the server. In general, the current question is about how to get the SSH service operational after reboot while the issue is engaged. If you wish you can upvote the solution :)
    – pa4080
    Jan 22, 2019 at 11:51
  • The "Workaround 1" worked for me... Thanks... Up Voted... Mar 15, 2019 at 16:47
  • 3
    It would be more proper to instead override /usr/lib/tmpfiles.d/sshd.conf rather than modifying it directly, as that file is managed by the package manager. To do so, simply make the change in /etc/tmpfiles.d/sshd.conf; this will take precedence over the sshd.conf in /usr/lib. See this section in tmpfiles.d(5). Great answer regardless, being on an OpenVZ VPS is exactly the situation I've encountered this in.
    – ZeroKnight
    May 27, 2019 at 14:03
  • 1
    As to why Workaround 1 works; you're avoiding using the /var/run symlink, which is what systemd-tmpfiles is having an issue with, and why the PrivSep dir is not being created. The 4th-last message of this thread sheds some light on this. Granted, it's regarding systemd-tmpfiles-clean, but I have a feeling the same thing applies here.
    – ZeroKnight
    May 27, 2019 at 14:13
5

Could you check whether your / (root filesystem) permissions are not changed? Have to be root:root like the two lines below:

drwxr-xr-x  25 root root      4096 дек 21 06:45 ..
drwxr-xr-x  25 root root      4096 дек 21 06:45 .

If the owner is another user (and not root) this will prevent creating all temporary files by systemd during system startup. You may check also with the command:

systemd-tmpfiles --create

If the root folder (/) has different permission, please change it with the following command:

chown root: /
0
4

For everybody that the fixes of @pa4080 do not work, I solved the error by disabling the ssh socket and enabling the ssh service directly. I was experiencing that the directory /run/sshd was deleted after a couple of hours even though I tried both solutions with cron but also with systemd-tmpfiles on a debian 11 container on proxmox.

The solution to my setup was to run

systemctl disable ssh.socket --now
systemctl enable ssh.service --now
2
  • Same problem for me with Ubuntu 20.04 LTS in LXC container on Proxmox. Perhaps triggered by an apt update? See also forum.proxmox.com/threads/…
    – NorbertM
    Jan 16, 2023 at 12:49
  • Yes that would match my experience too. It mostly happens when I create containers via ansible and start configuring them. This also includes apt update.
    – gxor
    Jan 17, 2023 at 19:50
2

Thanks everyone for helpful information. The problem with ssh-server on my Xenial Lubuntu was indeed related to ownership of '/' as suggested by Melebius & Stefan.
Manually creating /var/run/sshd and restarting ssh.service temporarily ssh-server temporarily. Editing the sshd.conf did not help in this system. Then following the last suggestion, I checked the root folder ownership with:

'ls -alF /' and sure enough, it had been accidentally changed to a local user/group. Issuing from the terminal: 'sudo chown root:root /' fixed my system, regardless of the edit to sshd.conf. So I restored that to its original state, i.e. d /var/run/sshd 0755 root root.

1

I am having this problem on my machine when I am running multiple instances of sshd on a single machine (18.04.02 LTS, OpenSSH 7.6p1).

The problem is that there are no switches in sshd (i.e. command line or the sshd_config file) provisioned for changing the location of the "privilege separation directory". The directory should be in the /var/empty, according to the OpenSSH 7.6p1 source code.

The Ubuntu package has remapped this to /run/sshd.

There is a "thread safety" issue in the init.d scripts at boot when both service script attempt to make the directory. I have asked both Ubuntu and OpenSSH to address the issue of hard-coded "privilege separation directory" path names in sshd. If I could upload files, I have the fixed based upon the 8.0p1 OpenSSH source code.

1
  • thanks - creating /var/empty was the only way that worked for me to start ssh -d ... to debug some ssh related issue (ubuntu 20.04).
    – harald
    Nov 4, 2021 at 19:43
0

If you still have internet connection, you can simply reinstall the openssh-server package by using this command:

sudo apt install openssh-server --reinstall

It will recreate missing files and directories.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .