Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

From a remote machine, I ran 100 concurrent requests against my Ubuntu VPS

ab -n 100 -c 100 http://...

And, as I suspected, the server "crashed". I can still ping it, and it responds very quickly (~50ms). But I can't ssh into it or reach the web-server. I ran 1000 requests with 10 concurrent before, and it responded quickly and reliably, running at about 80 requests per second.

Fast forward five minutes and I can log in, the web server works great, everything is perfect again.

  • Uptime tells me the load_average is 0.06, 0.04, 0.05, and I have a quarter gigabyte of memory left free (out of 512MB). Running netstat -n gives me lots and lots of lines like these:

    tcp        0      0 127.0.0.1:9311          127.0.0.1:35030         TIME_WAIT  
    tcp        0      0 127.0.0.1:5984          127.0.0.1:54384         TIME_WAIT  
    tcp        0      0 127.0.0.1:9311          127.0.0.1:35024         TIME_WAIT
    
  • The server is running nginx as a reverse proxy, with a handful of cherrypy servers behind it. These servers run on ports between 8000 and 9000, and they listen only to 127.0.0.1.

100 concurrent requests doesn't seem much to me, even though this is the absolute minimum spec server offered by my hosting company. How do I go about investigating what happend when the server crashed?


The server didn't reboot after the crash. There were no messages written to my kern.log, and there's no firewall in front of the server.

share|improve this question
    
Did the server rebooted? You can use uptime to check for that. Does your /var/log/kern.log show weird messages? Is there a firewall between your and the server which throttles simultaneous connections? Does the "crash" also happen when you've already an open SSH connection? –  Lekensteyn Oct 3 '11 at 17:06
    
I addressed some of this. I'll test it again at night, with an open ssh connection. –  Stefano Palazzo Oct 3 '11 at 17:12

1 Answer 1

up vote 3 down vote accepted

You need to install something that records performance data for your system and run it while you do these benchmarks. collectd is pretty popular for this, but will require some heavy learning up front. You can install 'sysstat' to get the 'sar' command, but the granularity is only 10 minutes so it may not catch all the problems. Also you can login and run something like 'vmstat 5' which will print out stats about IO/load/memory every 5 seconds.

100 is a lot for a small server if you are using Apache pre-fork (the default) with PHP.. since that will require 100 concurrent processes to satisfy the requests. If you have MaxClients set lower than 100, then the requests will backup in a queue, and go very slow. Thats probably preferrable to a total system crash though.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.