1

I have a fresh install of Ubuntu 18.04 Server. I am trying to build it up to replace a server that has died.

I am having a lot of trouble with OOM Killer killing my logins, my rsync's, and lots of other stuff whenever I try to do something.

I have 64G of memory and a 64G swapfile. I have two 1.2TB disks.

I've tried using 'free -h' and 'vmstat' to look for memory leaking away but haven't seen any useful signs.

Here is a snippet from a kern.log starting at the end of a fresh boot through the first "invoked oom-killer" event. Eventually my process, perhaps the rsync or an ssh login will get killed. I have never successfully be able to complete an rsync.

Are the "lowmem_reserve[]: 0 0 0 0" messages indicative of what is happening?

Feb 20 10:28:08 forest3 kernel: [   10.516328] EXT4-fs (sdb1): mounted filesystem with ordered data mode. Opts: (null)
Feb 20 10:28:08 forest3 kernel: [   11.359022] EXT4-fs (sda3): mounted filesystem with ordered data mode. Opts: (null)
Feb 20 10:28:08 forest3 kernel: [   11.375396] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null)
Feb 20 10:28:08 forest3 kernel: [   11.870338] IPv6: ADDRCONF(NETDEV_UP): eno4: link is not ready
Feb 20 10:28:08 forest3 kernel: [   12.414585] audit: type=1400 audit(1613838486.469:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=921 comm="apparmor_parser"
Feb 20 10:28:08 forest3 kernel: [   12.436645] audit: type=1400 audit(1613838486.493:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default" pid=919 comm="apparmor_parser"
Feb 20 10:28:08 forest3 kernel: [   12.436648] audit: type=1400 audit(1613838486.493:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-cgns" pid=919 comm="apparmor_parser"
Feb 20 10:28:08 forest3 kernel: [   12.436651] audit: type=1400 audit(1613838486.493:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-mounting" pid=919 comm="apparmor_parser"
Feb 20 10:28:08 forest3 kernel: [   12.436653] audit: type=1400 audit(1613838486.493:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-nesting" pid=919 comm="apparmor_parser"
Feb 20 10:28:08 forest3 kernel: [   12.484707] audit: type=1400 audit(1613838486.541:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/snapd/snap-confine" pid=923 comm="apparmor_parser"
Feb 20 10:28:08 forest3 kernel: [   12.484710] audit: type=1400 audit(1613838486.541:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=923 comm="apparmor_parser"
Feb 20 10:28:08 forest3 kernel: [   12.528993] audit: type=1400 audit(1613838486.585:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/sbin/tcpdump" pid=925 comm="apparmor_parser"
Feb 20 10:28:08 forest3 kernel: [   12.530646] audit: type=1400 audit(1613838486.585:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/sbin/dhclient" pid=920 comm="apparmor_parser"
Feb 20 10:28:08 forest3 kernel: [   12.530650] audit: type=1400 audit(1613838486.585:11): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=920 comm="apparmor_parser"
Feb 20 10:28:08 forest3 kernel: [   13.466381] Adding 67108860k swap on /users2/swapfile.  Priority:-2 extents:43 across:74950140k FS
Feb 20 10:28:09 forest3 kernel: [   15.395795] new mount options do not match the existing superblock, will be ignored
Feb 20 10:28:14 forest3 kernel: [   20.328485] igb 0000:07:00.1 eno4: igb: eno4 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Feb 20 10:28:14 forest3 kernel: [   20.328697] IPv6: ADDRCONF(NETDEV_CHANGE): eno4: link becomes ready
Feb 20 10:30:35 forest3 kernel: [  162.417980] EXT4-fs (sdc): mounted filesystem with ordered data mode. Opts: (null)
Feb 20 10:32:04 forest3 kernel: [  250.474806] rsync invoked oom-killer: gfp_mask=0x15000c0(GFP_KERNEL_ACCOUNT), nodemask=(null), order=0, oom_score_adj=0
Feb 20 10:32:04 forest3 kernel: [  250.474808] rsync cpuset=/ mems_allowed=0
Feb 20 10:32:04 forest3 kernel: [  250.474813] CPU: 4 PID: 1986 Comm: rsync Tainted: G        W        4.15.0-135-generic #139-Ubuntu
Feb 20 10:32:04 forest3 kernel: [  250.474814] Hardware name: Dell Inc. PowerEdge R620/0PXXHP, BIOS 2.1.2 09/19/2013
Feb 20 10:32:04 forest3 kernel: [  250.474815] Call Trace:
Feb 20 10:32:04 forest3 kernel: [  250.474823]  dump_stack+0x60/0x7e
Feb 20 10:32:04 forest3 kernel: [  250.474827]  dump_header+0x5a/0x229
Feb 20 10:32:04 forest3 kernel: [  250.474830]  ? ___ratelimit+0x79/0xf0
Feb 20 10:32:04 forest3 kernel: [  250.474832]  oom_kill_process+0x20a/0x3e0
Feb 20 10:32:04 forest3 kernel: [  250.474834]  out_of_memory+0xe9/0x2a0
Feb 20 10:32:04 forest3 kernel: [  250.474836]  __alloc_pages_slowpath+0xb05/0xbb0
Feb 20 10:32:04 forest3 kernel: [  250.474839]  __alloc_pages_nodemask+0x269/0x290
Feb 20 10:32:04 forest3 kernel: [  250.474842]  alloc_skb_with_frags+0xce/0x190
Feb 20 10:32:04 forest3 kernel: [  250.474845]  sock_alloc_send_pskb+0x1c3/0x1f0
Feb 20 10:32:04 forest3 kernel: [  250.474848]  ? _cond_resched+0x17/0x40
Feb 20 10:32:04 forest3 kernel: [  250.474851]  unix_stream_sendmsg+0x199/0x330
Feb 20 10:32:04 forest3 kernel: [  250.474853]  ? unix_getname+0xb0/0xb0
Feb 20 10:32:04 forest3 kernel: [  250.474855]  sock_sendmsg+0x32/0x40
Feb 20 10:32:04 forest3 kernel: [  250.474856]  sock_write_iter+0x8b/0xe0
Feb 20 10:32:04 forest3 kernel: [  250.474859]  new_sync_write+0xd0/0x130
Feb 20 10:32:04 forest3 kernel: [  250.474861]  ? sock_sendmsg+0x40/0x40
Feb 20 10:32:04 forest3 kernel: [  250.474863]  __vfs_write+0x37/0x50
Feb 20 10:32:04 forest3 kernel: [  250.474865]  vfs_write+0x94/0x1a0
Feb 20 10:32:04 forest3 kernel: [  250.474867]  SyS_write+0x4f/0xd0
Feb 20 10:32:04 forest3 kernel: [  250.474870]  do_fast_syscall_32+0x7f/0x200
Feb 20 10:32:04 forest3 kernel: [  250.474873]  entry_SYSENTER_32+0x68/0xbb
Feb 20 10:32:04 forest3 kernel: [  250.474874] EIP: 0xb7f33d09
Feb 20 10:32:04 forest3 kernel: [  250.474875] EFLAGS: 00000246 CPU: 4
Feb 20 10:32:04 forest3 kernel: [  250.474876] EAX: ffffffda EBX: 00000004 ECX: 01362e50 EDX: 00008008
Feb 20 10:32:04 forest3 kernel: [  250.474877] ESI: 00002268 EDI: 004d4f30 EBP: 004d2cb8 ESP: bfcaa2d0
Feb 20 10:32:04 forest3 kernel: [  250.474878]  DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b
Feb 20 10:32:04 forest3 kernel: [  250.474880]  ? nmi+0x8b/0x198
Feb 20 10:32:04 forest3 kernel: [  250.474880] Mem-Info:
Feb 20 10:32:04 forest3 kernel: [  250.474884] active_anon:4986 inactive_anon:4699 isolated_anon:0
Feb 20 10:32:04 forest3 kernel: [  250.474884]  active_file:7234 inactive_file:446544 isolated_file:0
Feb 20 10:32:04 forest3 kernel: [  250.474884]  unevictable:0 dirty:4 writeback:0 unstable:0
Feb 20 10:32:04 forest3 kernel: [  250.474884]  slab_reclaimable:11170 slab_unreclaimable:8935
Feb 20 10:32:04 forest3 kernel: [  250.474884]  mapped:7622 shmem:321 pagetables:375 bounce:0
Feb 20 10:32:04 forest3 kernel: [  250.474884]  free:15807387 free_pcp:355 free_cma:0
Feb 20 10:32:04 forest3 kernel: [  250.474887] Node 0 active_anon:19944kB inactive_anon:18796kB active_file:28936kB inactive_file:1786176kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:30488kB dirty:16kB writeback:0kB shmem:1284kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
Feb 20 10:32:04 forest3 kernel: [  250.474889] DMA free:1180kB min:780kB low:972kB high:1164kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:0kB unevictable:0kB writepending:0kB present:15980kB managed:15904kB mlocked:0kB kernel_stack:112kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Feb 20 10:32:04 forest3 kernel: [  250.474890] lowmem_reserve[]: 0 106 63687 63687
Feb 20 10:32:04 forest3 kernel: [  250.474894] Normal free:5316kB min:5340kB low:6672kB high:8004kB active_anon:0kB inactive_anon:0kB active_file:3048kB inactive_file:2956kB unevictable:0kB writepending:0kB present:890872kB managed:163460kB mlocked:0kB kernel_stack:1560kB pagetables:0kB bounce:0kB free_pcp:976kB local_pcp:24kB free_cma:0kB
Feb 20 10:32:04 forest3 kernel: [  250.474894] lowmem_reserve[]: 0 0 508647 508647
Feb 20 10:32:04 forest3 kernel: [  250.474898] HighMem free:63223052kB min:512kB low:800216kB high:1599920kB active_anon:19944kB inactive_anon:18796kB active_file:25840kB inactive_file:1783140kB unevictable:0kB writepending:16kB present:65106888kB managed:65106888kB mlocked:0kB kernel_stack:0kB pagetables:1500kB bounce:0kB free_pcp:444kB local_pcp:32kB free_cma:0kB
Feb 20 10:32:04 forest3 kernel: [  250.474898] lowmem_reserve[]: 0 0 0 0
Feb 20 10:32:04 forest3 kernel: [  250.474900] DMA: 39*4kB (UME) 28*8kB (UE) 28*16kB (UME) 7*32kB (UM) 2*64kB (ME) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1180kB
Feb 20 10:32:04 forest3 kernel: [  250.474905] Normal: 287*4kB (UME) 175*8kB (UME) 69*16kB (U) 34*32kB (UME) 9*64kB (UME) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5316kB
Feb 20 10:32:04 forest3 kernel: [  250.474910] HighMem: 75*4kB (UM) 58*8kB (U) 43*16kB (UM) 19*32kB (UM) 4*64kB (UM) 2*128kB (UM) 1*256kB (M) 3*512kB (UM) 3*1024kB (UM) 25*2048kB (UM) 15421*4096kB (UM) = 63223052kB
Feb 20 10:32:04 forest3 kernel: [  250.474917] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb 20 10:32:04 forest3 kernel: [  250.474918] 454111 total pagecache pages
Feb 20 10:32:04 forest3 kernel: [  250.474928] 10 pages in swap cache
Feb 20 10:32:04 forest3 kernel: [  250.474929] Swap cache stats: add 133, delete 123, find 0/0
Feb 20 10:32:04 forest3 kernel: [  250.474930] Free swap  = 67108084kB
Feb 20 10:32:04 forest3 kernel: [  250.474930] Total swap = 67108860kB
Feb 20 10:32:04 forest3 kernel: [  250.474931] 16503435 pages RAM
Feb 20 10:32:04 forest3 kernel: [  250.474932] 16276722 pages HighMem/MovableOnly
Feb 20 10:32:04 forest3 kernel: [  250.474932] 181872 pages reserved
Feb 20 10:32:04 forest3 kernel: [  250.474933] 0 pages cma reserved
Feb 20 10:32:04 forest3 kernel: [  250.474933] [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Feb 20 10:32:04 forest3 kernel: [  250.474938] [  489]     0   489     8631     2185    73728        0             0 systemd-journal
Feb 20 10:32:04 forest3 kernel: [  250.474940] [  508]     0   508     3517      452    49152        0             0 lvmetad
Feb 20 10:32:04 forest3 kernel: [  250.474941] [  516]     0   516     4432     1140    49152        0         -1000 systemd-udevd
Feb 20 10:32:04 forest3 kernel: [  250.474943] [  787] 62583   787     4799      730    61440        0             0 systemd-timesyn
Feb 20 10:32:04 forest3 kernel: [  250.474944] [  812]   100   812     2952      999    53248        0             0 systemd-network
Feb 20 10:32:04 forest3 kernel: [  250.474945] [  864]   101   864     2607      996    53248        0             0 systemd-resolve
Feb 20 10:32:04 forest3 kernel: [  250.474947] [ 1001]     0  1001     8256     3310    86016       34             0 networkd-dispat
Feb 20 10:32:04 forest3 kernel: [  250.474949] [ 1009]   102  1009     6467      985    61440        0             0 rsyslogd
Feb 20 10:32:04 forest3 kernel: [  250.474950] [ 1014]     0  1014    10054     1591    77824        0             0 accounts-daemon
Feb 20 10:32:04 forest3 kernel: [  250.474951] [ 1018]   103  1018     1702      908    49152        0          -900 dbus-daemon
Feb 20 10:32:04 forest3 kernel: [  250.474953] [ 1085]     0  1085      934      498    40960        0             0 atd
Feb 20 10:32:04 forest3 kernel: [  250.474954] [ 1106]     0  1106     1458      693    45056        0             0 cron
Feb 20 10:32:04 forest3 kernel: [  250.474955] [ 1108]     0  1108     5139      356    49152        0             0 lxcfs
Feb 20 10:32:04 forest3 kernel: [  250.474956] [ 1114]     0  1114     2591     1264    57344        0             0 systemd-logind
Feb 20 10:32:04 forest3 kernel: [  250.474958] [ 1118]     0  1118     4155      761    53248        0             0 irqbalance
Feb 20 10:32:04 forest3 kernel: [  250.474959] [ 1132]     0  1132     9559     1500    73728        0             0 polkitd
Feb 20 10:32:04 forest3 kernel: [  250.474960] [ 1162]     0  1162     9389     3967    94208       48             0 unattended-upgr
Feb 20 10:32:04 forest3 kernel: [  250.474961] [ 1421]     0  1421     2639     1197    53248       10         -1000 sshd
Feb 20 10:32:04 forest3 kernel: [  250.474963] [ 1468]     0  1468     1296      858    45056        3             0 login
Feb 20 10:32:04 forest3 kernel: [  250.474964] [ 1776]  1000  1776     3098     1613    57344        0             0 systemd
Feb 20 10:32:04 forest3 kernel: [  250.474966] [ 1787]  1000  1787     3596      416    57344        4             0 (sd-pam)
Feb 20 10:32:04 forest3 kernel: [  250.474967] [ 1809]  1000  1809     1730     1018    49152        0             0 bash
Feb 20 10:32:04 forest3 kernel: [  250.474968] [ 1819]     0  1819     1862      971    53248        0             0 sudo
Feb 20 10:32:04 forest3 kernel: [  250.474969] [ 1821]     0  1821     1725      830    49152        0             0 su
Feb 20 10:32:04 forest3 kernel: [  250.474971] [ 1822]     0  1822     1730     1032    45056        0             0 bash
Feb 20 10:32:04 forest3 kernel: [  250.474972] [ 1837]     0  1837     2891     1534    57344        0             0 sshd
Feb 20 10:32:04 forest3 kernel: [  250.474973] [ 1929]  1000  1929     2891      847    53248        0             0 sshd
Feb 20 10:32:04 forest3 kernel: [  250.474974] [ 1930]  1000  1930     1729      974    45056        0             0 bash
Feb 20 10:32:04 forest3 kernel: [  250.474975] [ 1942]     0  1942     1862      968    49152        0             0 sudo
Feb 20 10:32:04 forest3 kernel: [  250.474977] [ 1943]     0  1943     1725      868    49152        0             0 su
Feb 20 10:32:04 forest3 kernel: [  250.474978] [ 1944]     0  1944     1762     1070    45056        0             0 bash
Feb 20 10:32:04 forest3 kernel: [  250.474979] [ 1982]     0  1982     1122      150    45056        0             0 tail
Feb 20 10:32:04 forest3 kernel: [  250.474980] [ 1986]     0  1986     9403     1102   114688        0             0 rsync
Feb 20 10:32:04 forest3 kernel: [  250.474982] [ 1987]     0  1987     9303      747   106496        0             0 rsync
Feb 20 10:32:04 forest3 kernel: [  250.474983] [ 1988]     0  1988     9238      587   110592        0             0 rsync
Feb 20 10:32:04 forest3 kernel: [  250.474984] [ 1992]     0  1992     2891     1561    57344        0             0 sshd
Feb 20 10:32:04 forest3 kernel: [  250.474986] [ 2079]  1000  2079     2891      739    57344        0             0 sshd
Feb 20 10:32:04 forest3 kernel: [  250.474987] [ 2080]  1000  2080     1729     1057    49152        0             0 bash
Feb 20 10:32:04 forest3 kernel: [  250.474988] Out of memory: Kill process 1162 (unattended-upgr) score 0 or sacrifice child
Feb 20 10:32:04 forest3 kernel: [  250.475042] Killed process 1162 (unattended-upgr) total-vm:37556kB, anon-rss:5016kB, file-rss:10852kB, shmem-rss:0kB
Feb 20 10:32:04 forest3 kernel: [  250.492223] oom_reaper: reaped process 1162 (unattended-upgr), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

Here is what free shows, while an rsync is running. Also including swapon -s output.

root@forest3:~# free -h
              total        used        free      shared  buff/cache   available
Mem:            62G        178M         60G        588K        1.7G         59G
Swap:           63G        4.3M         63G
root@forest3:~#
root@forest3:~# swapon -s
Filename                Type        Size    Used    Priority
/users2/swapfile                        file        67108860    4664    -2
root@forest3:~#

I added the /users2/swapfile with a series of 'dd' commands appending to it. I did it in a series of commands in the hope of not getting that process killed while creating a new swapfile. Old original swapfile was the default 2G file. Once the new one was created, I turned swap off, ran mkswap on the new one and then turned swap back on pointing to the new file. I also updated /etc/fstab to use the new file so it carried over reboots.

As a test of where the "new mount options..." message comes from I tried unmounting /backups and remounting it. And then turning swap off and back on. Here is the added kern.log messages from those operations:

Feb 20 12:54:32 forest3 kernel: [ 4289.790003] EXT4-fs (sdc): mounted filesystem with ordered data mode. Opts: (null)
Feb 20 12:54:45 forest3 kernel: [ 4302.387143] Adding 67108860k swap on /users2/swapfile.  Priority:-2 extents:43 across:74950140k FS

I also have been reading some posts about vm.overcommit and tried changing the defaults and things seem to be a bit better but still getting a lot of OOM killer events. here is my /etc/sysctl.d/10-no-overcommit.conf


## default
#vm.overcommit_memory = 0
#vm.overcommit_ratio = 50

# Try:
vm.overcommit_memory = 2
vm.overcommit_ratio = 100

My current state is that I have 4 machines I am experimenting with. Each is an identical chassis. Each has same CPU and Memory. forest2 has 4 1.2TB disks. The others have 2 1.2TB disks. All are running kernel 4.15.0-135-generic. The machines with NEW BIOS were all retested with my rsync test after BIOS upgrade.

name Ubuntu BIOS memtest rsynctest swap swap size swappiness install notes
forest2 18.04 Desktop OLD Pass partition 64G 60 14.04->16.04->18.04
forest3 18.04 Server NEW Fail file: /swapfile 4G 10 18.04 mini
forest3' 18.04 Server NEW Pass file: /swapfile 8G 60 18.04.5-live-server-amd64.iso
forest4 18.04 Server NEW PASSED Fail file: /swapfile 4G 60 18.04 mini
forest4' 18.04 Server NEW PASSED PASS partition 8G 60 14.04->16.04->18.04
forest5 18.04 Server NEW Fail partition 8G 60 18.04 mini

The difference between forest3 and forest3' is that forest3' was built directly from 18.04.5-live-server-amd64.iso instead of the 18.04 server mini ISO. This new machine works fine!

The difference between forest4 and forest4' is that forest4' was built starting from a Ubuntu 14.04 Server Install CD and then do-release-upgrade to 16.04 and then do-release-upgrade to 18.04.

forest2 was built from a 14.04 Desktop Install DVD and then upgraded to 16.04 and then 18.04.

All machines have 64G of RAM. Original question was with regard to forest3 when it had 64G of swap. Since then it has been changed to have 4G of swap.

Improve this question
0

2 Answers 2

Reset to default
0

BIOS

Dell Inc. PowerEdge R620/0PXXHP

You have BIOS version 2.1.2 from 09/19/2013. There's a much newer BIOS available, version 2.9.0 from Feb 21, 2020, that can be downloaded here.

Note: Confirm that I have the correct web page for your model #.

Note: Have good backups before updating the BIOS.

/swapfile

64G swap is crazy.

Note: Incorrect use of the dd command can cause data loss. Suggest copy/paste.

In the terminal...

sudo swapoff -a           # turn off swap
sudo rm -i /users2/swapfile      # remove old /swapfile

sudo dd if=/dev/zero of=/swapfile bs=1M count=4096

sudo chmod 600 /swapfile  # set proper file protections
sudo mkswap /swapfile     # init /swapfile
sudo swapon /swapfile     # turn on swap
free -h                   # confirm 64G RAM and 4G swap

Confirm this /swapfile line at the end of /etc/fstab... and confirm no other “swap” lines...

To edit, use sudo -H gedit /etc/fstab or sudo pico /etc/fstab

/swapfile  none  swap  sw  0  0

reboot                    # reboot and verify operation

Note: Set these back to default...

## default
#vm.overcommit_memory = 0
#vm.overcommit_ratio = 50

Note: Optional... set vm.swappiness=10

Update #1:

Go to https://www.memtest86.com/ and download/run their free memtest to test your memory. Get at least one complete pass of all the 4/4 tests to confirm good memory. This may take many hours to complete.

Update #2:

All effected servers had 18.04 mini installed. By installing 18.04.5-live-server-amd64.iso, it all seems to be working now.

Improve this answer
0

From the kern.log,

Feb 20 10:32:04 forest3 kernel: [  250.474988] Out of memory: Kill process 1162 (unattended-upgr) score 0 or sacrifice child
Feb 20 10:32:04 forest3 kernel: [  250.475042] Killed process 1162 (unattended-upgr) total-vm:37556kB, anon-rss:5016kB, file-rss:10852kB, shmem-rss:0kB
Feb 20 10:32:04 forest3 kernel: [  250.492223] oom_reaper: reaped process 1162 (unattended-upgr), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

I think this has been answered previously.

http://unix.stackexchange.com/questions/374748/ddg#399531

The solution is to stop the unattended upgrade and the rsync until you can get your system stable. Both of those could have potentially high I/O and high CPU costs as well as exhausting memory.

Improve this answer
4
  • 1
    I'm not sure how this applies to my situation. I don't have pkg updates that are stuck and I think my installed system is stable. But I tried turning off unattended upgrades. Then I did, 'apt udpate; apt -y upgrade; apt -f install ; apt full-upgrade'. There were no lingering pkgs to handle. And after all that the rsync still failed. Then I turned unattended upgrades back on and tried rsync again and it still failed.
    – John DeHart
    Feb 22, 2021 at 14:34
  • Is your machine still dying due to the OOM condition? Please post a new kern.log if so.
    – mondotofu
    Feb 23, 2021 at 3:37
  • Would you like to re-edit your question and show the rsync configuration so that we can further troubleshoot the issue?
    – mondotofu
    Feb 23, 2021 at 3:38
  • No, I am no longer having the problem. I reinstalled from 18.04.5-live-server-amd64.iso instead of the 18.04 Sever mini iso I had been using and the problem went away. I had tried 3 different machines with the mini version and all exhibited the problem. But I have been beating on the reinstalled machine all day without any problems.
    – John DeHart
    Feb 23, 2021 at 22:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.