0

I know almost nothing about this server. It was setup before I joined my company and I never used until now. My colleague told me that every month the ssh stop working. The server is still up because services in it keeps running but ssh. We reboot it from the main control panel, after the reboot ssh is back.

Ubuntu Server 12.04.1 LTS.

If you'd start to investigate on it, where would you start from?

EDIT: Everything related to ssh get stuck. Ssh connection using putty or SecureShell, Sftp [...] simply no answer. Unfortunately I'm new to this server and I don't know exactly what other services depends from ssh. What I know is that it just stops working once a month (and I only saw this once, around 22 of december, then my colleague told me of this issue happening once a month).

OpenSSH_5.9p1 Debian-5ubuntu1, OpenSSL 1.0.1 14 Mar 2012 (Yes I know..update it! I simply better not touch it now, production environment).. I only need advices on what to check first and last

Port 22
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
UsePrivilegeSeparation yes
KeyRegenerationInterval 3600
ServerKeyBits 768
SyslogFacility AUTH
LogLevel INFO
LoginGraceTime 120
PermitRootLogin yes
StrictModes yes
RSAAuthentication yes
PubkeyAuthentication yes
IgnoreRhosts yes
RhostsRSAAuthentication no
HostbasedAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
X11Forwarding yes
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
AcceptEnv LANG LC_*
Subsystem sftp /usr/lib/openssh/sftp-server -f LOCAL5 -l INFO
UsePAM yes

Rootlogin, yes I know too.. But..

EDIT 2 (not ssh this time):

It happened again but not as I expected! This time the server just froze and it is not related to ssh. The management console saw the server not responding to Ping. The last log logged on syslog say nothing to my eyes. I copied it in the main question.. Probably you (guys) can see something I do not see.. Or can give me advices on improve the logging system to catch every (every) [every] {every} log! I too am only an experienced user but I know Windows OSs better :/

Jan 10 06:05:01 ns237221 kernel: mdadm: sending ioctl 1261 to a partition!
Jan 10 06:06:01 ns237221 kernel: last message repeated 3 times
Jan 10 06:06:01 ns237221 CRON[11762]: (root) CMD (/usr/local/rtm/bin/rtm 27 > /dev/null 2> /dev/null)
Jan 10 06:06:24 ns237221 kernel: [UFW BLOCK] IN=eth0 OUT= MAC=4c:72:b9:24:c6:b3:e8:ba:70:42:e4:80:08:00 SRC=222.186.34.180 DST=37.59.5.180 LEN=40 TOS=0x00 PREC=0x00 TTL=112 ID=256 PROTO=TCP SPT=21404 DPT=2345 WINDOW=16384 RES=0x00 SYN URGP=0
Jan 10 06:06:36 ns237221 kernel: [UFW BLOCK] IN=eth0 OUT= MAC=4c:72:b9:24:c6:b3:1c:e6:c7:52:af:80:08:00 SRC=81.93.11.86 DST=37.59.5.180 LEN=52 TOS=0x00 PREC=0x00 TTL=52 ID=9496 DF PROTO=TCP SPT=4935 DPT=3389 WINDOW=65535 RES=0x00 SYN URGP=0
Jan 10 06:07:01 ns237221 CRON[11806]: (root) CMD (/usr/local/rtm/bin/rtm 27 > /dev/null 2> /dev/null)
Jan 10 06:08:01 ns237221 CRON[11850]: (root) CMD (/usr/local/rtm/bin/rtm 27 > /dev/null 2> /dev/null)
Jan 10 06:09:01 ns237221 CRON[11897]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete)
Jan 10 06:09:01 ns237221 CRON[11898]: (root) CMD (/usr/local/rtm/bin/rtm 27 > /dev/null 2> /dev/null)
Jan 10 06:09:39 ns237221 kernel: [UFW BLOCK] IN=eth0 OUT= MAC=4c:72:b9:24:c6:b3:e8:ba:70:42:e4:80:08:00 SRC=107.150.98.130 DST=37.59.5.180 LEN=40 TOS=0x00 PREC=0x00 TTL=243 ID=54321 PROTO=TCP SPT=42075 DPT=84 WINDOW=65535 RES=0x00 SYN URGP=0
Jan 10 07:22:25 ns237221 kernel: imklog 5.8.6, log source = /proc/kmsg started.
Jan 10 07:22:25 ns237221 rsyslogd: [origin software="rsyslogd" swVersion="5.8.6" x-pid="3475" x-info="http://www.rsyslog.com"] start
  • How do you know it is "stuck"? Doesn't it answer on the ssh port? Doesn't it answer at all? Does the authentication fail? What exactly happens? Also, please add the content of /etc/ssh/sshd_config as an edit to your question. – s3lph Jan 8 '16 at 13:28
  • What if you created a cronjob that restarts the ssh server regulary? Already opened connections somehow aren't affected by a ssh server restart, I just tested. – s3lph Jan 8 '16 at 14:39
  • well that would be just a patch, I don't really mind of rebooting the server because it would clean cache/ram and I'm ok with that. The fact no one cared to fix this also means it is not so necessary, would be just a plus to fix without using external tools (we thought of cron but that's why we said to not use it. thanks anyway! ) – Nihvel Jan 8 '16 at 14:45
1

What I would start with is :

  • before rebooting, if you still have access to the server (physical console, com port, etc.) : check the service status (service sshd status) and see if some process is still listening to your ssh port (netstat -anp)
  • check auth.log for any unusal ssh activity just before the service crashed
  • if the problem occurs on the same date every month, you could check the cron tables for anything planned at that time.

Please, note that I'm not an experienced sysadmin, only a somewhat experienced user.

  • Good hints, I need to wait for the next time it happen.. some day of some next month and if luck this january! beside auth.log are there other logs I could check? those from kernel panic perhaps..? Thanks man (: – Nihvel Jan 8 '16 at 14:47
  • It happened again but not as I expected! This time the server just froze and it is not related to ssh. The management console saw the server not responding to Ping. The last log logged on syslog say nothing to my eyes. I copied it in the main question.. Probably you (guys) can see something I do not see.. Or can give me advices on improve the logging system to catch every (every) [every] {every} log! I too am only an experienced user but I know Windows OSs better :/ – Nihvel Jan 11 '16 at 13:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.