2

One of our servers ended up using about 20 GB of data yesterday because it was repeatedly requesting the packages.bz2 file from a mirror.

The requests began about 22:00 on Sunday night, and stopped about 21:00 on the Monday.

There is nothing in the apt history, unattended-upgrades, or the system log to indicate why.

All I can see is a bunch of blocked traffic in the UFW log from the mirror IP address - there are no rules to block this IP in the UFW setup, communication to the mirror is successful outside this period of unusual activity, so it looks like flood protection (though I've gone through the UFW configuration and couldn't find this explicitly set).

/var/log/apt/term.log and /var/log/apt/term.log have no entries after 6:45 last Friday (2 whole days before the event started).

/var/log/aptitude is empty since its creation over a year ago.

auth.log shows normal cron activity for an Ubuntu server - and not a lot else besides my logins and use of sudo when trying to diagnose the problems on there.

There is nothing in cron.d calling apt or aptitude - and cron.daily has the same collection of scripts as other servers that are not misbehaving.

I've spoken with the sysadmin for the mirror - he was very helpful. This is how I found out that my server was requesting the packages.bz2 file repeatedly. He says that each request was fulfilled and closed correctly.

So - what could have happened to start this process? Why would it go on for so long, and why did it just stop?


And one week later, it started again - still nothing in the logs to indicate why.

8
  • 2
    Are you using Lucid? What is the contents of /var/log/apt/term.log /var/log/aptitude
    – Braiam
    Dec 17, 2013 at 1:23
  • @Braiam - I've added that information to my questions
    – HorusKol
    Dec 17, 2013 at 5:33
  • "All I can see is a bunch of blocked traffic in the UFW log from the mirror IP address." <-- What do you actually see? Is that incoming or outgoing traffic (from the firewall's perspective)? And if it's blocked, it should not use any bandwidth.
    – gertvdijk
    Dec 24, 2013 at 18:36
  • 1
    While it's happening, make sure to monitor the network usage on your machine. Use tools like iftop, netstat, atop, tcpdump to see more about it. As your question currently stands, this is a needle in a haystack. Remember we can't see your screen and we don't have access to your machine, so we rely on the information in your question. I suggest to get familiar with basic troubleshooting with the tools I mentioned. This site isn't really made for trial&error finding the needle in a haystack, I'm afraid.
    – gertvdijk
    Dec 24, 2013 at 18:38
  • 1
    the logs doesn't register every call to apt-get update. Maybe the auth.log and the cron scripts may be more relevant.
    – Braiam
    Dec 25, 2013 at 2:32

2 Answers 2

2

apt-get update isn't logged in the APT logs.

To me, it appears that the server didn't get the file, or it was invalid. Another possibility is a strange cron script.

Now the reason is a bit more interesting. Either

  1. There is maybe a cron to run apt-get update
  2. Someone (another admin and/or hacker) was spamming apt-get update

Why it would run 23 hours... Maybe a bad cron entry?

Also, not really an expert on UFW/blocking.

Personally, I would check cron, and auth.log as @Braiam suggested.

0

To me this sounds like a scripts loop.

Does the system happen to be set to update it's software at sundays at around 22.00?

Do you have some sort of script running for updating it without attendance?

1
  • I believe this MAY be a dupe answer.... =]
    – Kaz Wolfe
    Dec 29, 2013 at 17:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .