4
top - 15:54:32 up 9 days, 49 min,  1 user,  load average: 38.23, 30.61, 25.64
Tasks: 1777 total,   8 running, 1769 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.1 us, 75.2 sy,  0.0 ni, 22.7 id,  1.9 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 32648604 total,   870392 free,  2642164 used, 29136048 buff/cache
KiB Swap: 16760828 total, 16552632 free,   208196 used. 18500956 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 1713 root      20   0       0      0      0 R  79.5  0.0   1439:34 txg_quiesce
 1714 root      20   0       0      0      0 D  62.0  0.0   1310:11 txg_sync

My server gets stuck like this for minutes at a time and no processes can do I/O with the ZFS filesystem. The load average goes from under 1.0 when this starts to over 80.0 by the time that it completes and returns to normal.

What can I do to stop this?


Updates:

root@ca0:~# zpool iostat -v 1   
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
lxc          780G  1.00T  6.38K      0  8.51M      0
  mirror     780G  1.00T  6.38K      0  8.51M      0
    sda4        -      -  3.25K      0  4.40M      0
    sdb4        -      -  3.13K      0  4.11M      0
----------  -----  -----  -----  -----  -----  -----

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
lxc          780G  1.00T  4.69K      0  7.33M      0
  mirror     780G  1.00T  4.69K      0  7.33M      0
    sda4        -      -  2.27K      0  3.42M      0
    sdb4        -      -  2.42K      0  3.91M      0
----------  -----  -----  -----  -----  -----  -----

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
lxc          780G  1.00T  7.46K      0  10.3M      0
  mirror     780G  1.00T  7.46K      0  10.3M      0
    sda4        -      -  3.72K      0  5.13M      0
    sdb4        -      -  3.74K      0  5.13M      0
----------  -----  -----  -----  -----  -----  -----

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
lxc          780G  1.00T  7.37K      0  10.1M      0
  mirror     780G  1.00T  7.37K      0  10.1M      0
    sda4        -      -  3.67K      0  5.02M      0
    sdb4        -      -  3.70K      0  5.03M      0
----------  -----  -----  -----  -----  -----  -----

^C
root@ca0:~# zpool status
  pool: lxc
 state: ONLINE
  scan: scrub repaired 0 in 6h58m with 0 errors on Sun May 14 07:22:22 2017
config:

    NAME        STATE     READ WRITE CKSUM
    lxc         ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        sda4    ONLINE       0     0     0
        sdb4    ONLINE       0     0     0

errors: No known data errors
  • 1
    Are there anything logged in /var/log/messages just before, during or just after this happens ? And/or does dmesgproduce interesting output when run in a terminal just after the problem ? – Soren A May 16 '17 at 9:43
  • 2
    What do zpool iostat -v 1 and zpool status show you? Did you enable dedup (you shouldn't)? – bahamat May 20 '17 at 17:54
  • @bahamat updated. – fadedbee May 22 '17 at 16:16
  • @SorenA no, nothing grepping zfs or txg. – fadedbee May 22 '17 at 16:17
  • 2
    @chrisdew There is a lot of data being read. Any idea what that might be? Ordinarily, I'd say dtrace it... And what about dedup? – bahamat May 23 '17 at 5:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.