1

In my application I have two servers (private and external) and there is a need for the private server to run a script on the external server.

Using ssh, I am trying to test the limits of how many connections it will hold:

while (1)
  foreach k (1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 )
    ssh -i /path/to/key user@host "ls;sleep 0.3" &
    sleep 0.06  # <=16.6 times/sec
  end
end

The external server (2 core CPU, 4 GB ram), manage to maintain the incoming connections when they are at a rate of about 16/sec. Anything faster than that, it has connection issues, and cpu load gets of the roof (total tasks & zombie processes amount start to accumulate. However memory usage seems low).

The error message that we see on the private server after a few seconds of high frequency connections is

 ssh_exchange_identification: Connection closed by remote host

running the same test script on the external server, we don't see an issue with that rate (cpu load is about 7%, no zombies).

Assuming it is a ssh issue, so we added these in the ssh config file and restarted the ssh service but it didn't help:

MaxStartups 20:50:100
MaxSessions 100

UPDATE

As suggested, I tried to use ControlMaster. Here is what I did:

Added this to the ~/.ssh/config in the private server:

Host <external_host_ip>
       ControlMaster auto
       ControlPath ~/.ssh/cm_socket/%r@%h:%p
       ControlPersist 600

Created the first connection:

 ssh -M -S /home/user/.ssh/cm_socket/user@host_ip:22 -i ~/key.pem  user@host_ip

I checked, and something happened in the 'cm_socket' folder:

$ ll ~/.ssh/cm_socket/
srw------- 1 user user    0 Oct 18 18:10 user@host_ip:22=

but when I tried to do a subsequent ssh without the key, it didn't work:

$ ssh -M -S /home/user/.ssh/cm_socket/user@host_ip:22 user@host_ip
Permission denied (publickey).

Is there a way to increase the throughput that can be handled by this setup?

  • Use ControlMaster functionality of client OpenSSH. – Jakuje Oct 18 '16 at 15:41
  • Thank you for the comment. I tried, for some reason it didn't work. I updated the question with the result, I would be glad if you can take a look – mimusheli Oct 18 '16 at 18:20
  • Using -M prevents the second connection to connect to the existing socket. – Jakuje Oct 18 '16 at 18:25
0

Use ControlMaster:

Host <external_host_ip>
   ControlMaster auto
   ControlPath ~/.ssh/cm_socket/%r@%h:%p
   ControlPersist 600
   User user
   IdentityFile ~/key.pem

And then connect just using ssh host_ip. The first session initiates the connection, the others (up to 100 consecutive according to your setup) will reuse the first one. It should get a lot faster.

| improve this answer | |
  • excellent. It worked! Since I don't want the connection to close, I am using ControlPersist=yes. This ssh tunnel must always be kept open. I have read about 'autossh'. Just to be sure, is there a need for 'autossh' here, or is the ControlMaster solution good enough? – mimusheli Oct 19 '16 at 7:25
  • autossh re-establishes the connection in case of failure (for tunnels). For normal connection, you don't need that. Also some ServerAliveTimeout and ServerAliveInterval will make the connection last in the times of inactivity. – Jakuje Oct 19 '16 at 7:27
  • Thanks for the answer. Now I have another related issue link – mimusheli Oct 19 '16 at 9:04
  • link to the issue => here (was not very clear in previous comment) – mimusheli Oct 19 '16 at 9:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.