I've seen that as well.
I don't have a solution, but I have a workaround (echo 3 | sudo tee /proc/sys/vm/drop_caches
) and potentially more information so someone can bring the investigation any further.
It's not a network issue because at "Reading package list...", it is just reading files in /var/lib/apt/lists/
. A:
strace -tt -T -fo strace.log apt-get update
gives:
16394 14:43:03.921130 open("/var/lib/apt/lists/gb.archive.ubuntu.com_ubuntu_dists_precise_main_binary-i386_Packages", O_RDONLY|O_LARGEFILE) = 7 <0.000012>
[...]
16394 14:43:03.995238 read(6, "-3.1ubuntu2)\nConflicts: linux86\n"..., 32444) = 32444 <0.000111>
16394 14:43:05.787187 read(6, "c (<< 1:14.b.4-dfsg), erlang-exa"..., 32239) = 32239 <0.000069>
16394 14:43:05.788025 read(6, ".deb\nSize: 42130\nMD5sum: c7de671"..., 31695) = 31695 <0.000068>
16394 14:43:05.870734 read(6, "5: 29c4b395a92bdc12932f151c3643a"..., 31607) = 31607 <0.000071>
16394 14:43:05.890862 read(6, "e-pack-af-base\nFilename: pool/ma"..., 32538) = 32538 <0.000070>
16394 14:43:05.891425 read(6, "buntu-usb-live, ubuntu-dvd-live,"..., 32090) = 32090 <0.000066>
16394 14:43:05.891960 read(6, "cd9755b03ac2c9b8251125c7b6618\nDe"..., 32195) = 32195 <0.000034>
16394 14:43:06.043001 read(6, "rg>\nArchitecture: all\nVersion: 2"..., 32535) = 32535 <0.000072>
See how those 8 read
system calls took over 2 seconds even though each individual call takes less than 1 ms.
Running time apt-get update
or looking at top
, that process is not busy between those two calls. So why the delay?
Then I did:
echo t > /proc/sysrq-trigger
a few times and looked at the outcome in kern.log
:
apt-get D 00000000 0 16790 12706 0x00000000
e8695d30 00000086 f7bd5e6c 00000000 f7bd5e44 f74a6580 c1990e00 c1990e00
efe46efe 000042cb f7b9de00 e71a7230 f74a6580 c107e116 00000000 00000000
044aa200 00000000 00000000 00000000 00000000 e8695d0c e8695d0c c1038de8
Call Trace:
[<c107e116>] ? enqueue_entity+0x186/0x220
[<c1038de8>] ? default_spin_lock_flags+0x8/0x10
[<c15e13bd>] ? _raw_spin_lock_irqsave+0x2d/0x40
[<c15e0533>] schedule+0x23/0x60
[<c15deecf>] schedule_timeout+0x12f/0x290
[<c1075c38>] ? ttwu_do_activate.constprop.86+0x58/0x70
[<c1055190>] ? usleep_range+0x40/0x40
[<c15e0846>] io_schedule_timeout+0x86/0xd0
[<c15cef7d>] balance_dirty_pages.isra.17+0x3f5/0x4b4
[<c15e118d>] ? _raw_spin_lock+0xd/0x10
[<c1180781>] ? __set_page_dirty_buffers+0x81/0xb0
[<c110deb5>] ? set_page_dirty+0x55/0x60
[<c11812c9>] ? __block_page_mkwrite+0xe9/0x170
[<c110f3ae>] balance_dirty_pages_ratelimited_nr+0xde/0x100
[<c1126f53>] do_wp_page+0x503/0x830
[<c1128ef7>] handle_pte_fault+0x267/0x2c0
[<c1129c62>] handle_mm_fault+0x1e2/0x280
[<c15e4988>] do_page_fault+0x158/0x4c0
[<c104e4dc>] ? irq_exit+0x5c/0xa0
[<c15e22d0>] ? do_debug+0x180/0x180
[<c15e4830>] ? vmalloc_fault+0x195/0x195
[<c15e1c53>] error_code+0x67/0x6c
So, not sure what that means but that looks about the handling of page faults, so points at a potential memory management issue.
I then tried a:
echo 3 >/proc/sys/vm/drop_caches
And that did make the problem go away.
Now, it very much looks like a kernel issue. So, I've updated to the latest kernel (3.8 backport from raring
) and that's where I'm at. Will update if the problem persists with the newer kernel.
Edit
The problem does persist with the new kernel, though not as bad. And same thing,
echo 3 | sudo tee /proc/sys/vm/drop_caches
clears the problem for a while. I've only seen that happen on MSI laptops (Product Name: CR61 2M/CX61 2OC/CX61 2OD).
Edit December 2015
As confirmed by btrace
aptitude
/apt-get
does appear to do some disk I/O at the time. It's got a temporary file (/var/cache/apt/pkgcache.bin.<random-chars>
) mmapped in memory which is why it doesn't show in the strace
output.
Still can't explain why it only happens on some machines only, why dropping caches helps, why switching to 64bit helps.
If somebody can reproduce it, an interesting test could be to see if that also happens when running under eatmydata
or if moving /var/cache/apt
onto tmpfs
or a ramdisk helps.
sudo apt-get update
, correct?Software Sources
, see if selecting another server, instead of your current one, helps.