Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

I'm running nginx wordpress server in KVM using 12.04 server x86. It was running very well about 4 month until 2 hours ago. I found that my website is down and no ping response. Virt-manager logged high cpu load(plz see the picture below) before unexpected shut down. I want to know what process caused unexpected shutdown. The following log files make me think my server is attacked. Any suggestions and help would be appreciated.

enter image description here

kern.log and syslog showed me same output.

Nov 11 03:54:11 www kernel: [1344541.156239] [UFW BLOCK] IN=eth0 OUT= MAC= SRC=0.0.0.0 DST=224.0. 0.1 LEN=32 TOS=0x00 PREC=0xC0 TTL=1 ID=0 DF PROTO=2

Nov 11 03:54:11 www kernel: [1344541.156315] [UFW BLOCK] IN=eth0 OUT= MAC= SRC=0101:080a:2334:c90 0:0100:0000:0000:0000 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=72 TC=0 HOPLIMIT=1 FLOWLBL=0 PROTO=ICMPv6 TYPE=130 CODE=0

/nginx/access.log showed me

119.235.237.17 - - [11/Nov/2012:03:45:29 +0900] "GET /blog HTTP/1.1" 200 30493 "-" "Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)" my-server-ip - - [11/Nov/2012:11:05:30 +0900] "POST /wp-cron.php?doing_wp_cron=13 HTTP/1.0" 499 0 "-" "WordPress/3.4.2; http://mywebsite.com"

Server turned on in here. 119.235.237.16 - - [11/Nov/2012:11:05:30 +0900] "GET /blog HTTP/1.1" 200 32935 "-" "Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)"

share|improve this question
    
"Yeti is the crawler for Naver, a popular Korean search engine" (useragentstring.com/pages/Crawlerlist). Perhaps this "attack" was an unintentional DOS when Yeti began crawling your site? Have you done any previous load-testing of your site? –  Lars Rohrbach Nov 11 '12 at 5:10
    
Thanks for heads up @LarsRohrbach Oh yeah, just figured out about yeti stuff. I've never tried load-testing before. I'm doing that right now in here loadimpact.com. Will you tell me how I can use the result from load-testing? –  linuxk Nov 11 '12 at 5:27
    
Load-testing could help confirm whether your site responds well to demand, or reproduces a high-cpu-load issue, for example. –  Lars Rohrbach Nov 11 '12 at 5:41
    
Alright Thanks for your info. I found interesting results. style.css loads really slow when it comes in load-testing. 81.16ms to load. javascript is taking more than that,but I think js never been fast. –  linuxk Nov 11 '12 at 5:51
    
Decided block ip range of 119.235.237.0/24 in nginx –  linuxk Nov 12 '12 at 8:30
show 2 more comments

1 Answer

up vote 1 down vote accepted

While the web access may be coincidental, it's certainly possible that the Korean web crawler Naver created an unintentional Denial Of Service (DOS) against your site by aggressively crawling it. This can potentially crash nginx or your OS, or it can just make them too slow to respond effectively to new requests.

Using a website load-tester can help to determine just how much load your site can handle, and perhaps reproduce whether a heavy load could indeed cause a problem to your webserver. Depending on the detail provided by the load-tester, you may be able to determine whether there are any improvements you can make to your site code or configuration to improve response.

You can also choose just to block the offending web client, or create/modify your local robots.txt file to block a specific crawler (e.g. Yeti) or specify what part of your site they should not crawl. Sorry to hear in your comments, but apparently Yeti isn't honoring your robots.txt file, so blocking their IP range seems reasonable.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.