8

I am running Ubuntu with ZFS on root, and my bpool seems to be almost full.

Output of zpool list bpool:

NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
bpool  1.88G  1.69G   191M        -         -    60%    90%  1.00x    ONLINE  -

Output of df -h /boot:

Filesystem                                        Size  Used Avail Use% Mounted on
bpool/BOOT/ubuntu_s2c09x                          367M  303M   64M  83% /boot

This is causing problems with completing the installation of kernel updates, e.g. running sudo apt upgrade:

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
2 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] 
Requesting to save current system state
ERROR couldn't save system state: Minimum free space to take a snapshot and preserve ZFS performance is 20%.
Free space on pool "bpool" is 10%.
Please remove some states manually to free up space. 
Setting up linux-firmware (1.197.5) ...
update-initramfs: Generating /boot/initrd.img-5.11.0-41-generic
I: The initramfs will attempt to resume from /dev/nvme0n1p2
I: (UUID=abdb40da-2d0b-4a45-b024-1c7245ea27ea)
I: Set the RESUME variable to override this.
Error 24 : Write error : cannot write compressed block 
E: mkinitramfs failure cpio 141 lz4 -9 -l 24
update-initramfs: failed for /boot/initrd.img-5.11.0-41-generic with 1.
dpkg: error processing package linux-firmware (--configure):
 installed linux-firmware package post-installation script subprocess returned error exit status 1
Setting up linux-image-5.11.0-34-generic (5.11.0-34.36) ...
Processing triggers for linux-image-5.11.0-34-generic (5.11.0-34.36) ...
/etc/kernel/postinst.d/dkms:
 * dkms: running auto installation service for kernel 5.11.0-34-generic
Error! Your kernel headers for kernel 5.11.0-34-generic cannot be found.
Please install the linux-headers-5.11.0-34-generic package,
or use the --kernelsourcedir option to tell DKMS where it's located
   ...done.
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-5.11.0-34-generic
I: The initramfs will attempt to resume from /dev/nvme0n1p2
I: (UUID=abdb40da-2d0b-4a45-b024-1c7245ea27ea)
I: Set the RESUME variable to override this.
Error 24 : Write error : cannot write compressed block 
E: mkinitramfs failure cpio 141 lz4 -9 -l 24
update-initramfs: failed for /boot/initrd.img-5.11.0-34-generic with 1.
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
dpkg: error processing package linux-image-5.11.0-34-generic (--configure):
 installed linux-image-5.11.0-34-generic package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 linux-firmware
 linux-image-5.11.0-34-generic
ZSys is adding automatic system snapshot to GRUB menu

I've tried destroying some bpool snapshots, but zfs list -t snapshot | grep bpool doesn't list any over 100K. I have also tried removing some kernels, see output of dpkg -l linux-{image,headers}-"[0-9]*":

un  linux-headers-3.0               <none>         <none>       (no description available)
un  linux-headers-5.10.0-1045-oem   <none>         <none>       (no description available)
un  linux-headers-5.11.0-34-generic <none>         <none>       (no description available)
un  linux-headers-5.11.0-35-generic <none>         <none>       (no description available)
un  linux-headers-5.11.0-36-generic <none>         <none>       (no description available)
un  linux-headers-5.11.0-37-generic <none>         <none>       (no description available)
un  linux-headers-5.11.0-38-generic <none>         <none>       (no description available)
ii  linux-headers-5.11.0-40         5.11.0-40.44   all          Header files related to Linux kernel version 5.11.0
ii  linux-headers-5.11.0-40-generic 5.11.0-40.44   amd64        Linux kernel headers for version 5.11.0 on 64 bit x86 SMP
ii  linux-headers-5.11.0-41         5.11.0-41.45   all          Header files related to Linux kernel version 5.11.0
ii  linux-headers-5.11.0-41-generic 5.11.0-41.45   amd64        Linux kernel headers for version 5.11.0 on 64 bit x86 SMP
un  linux-headers-686-pae           <none>         <none>       (no description available)
ic  linux-image-5.10.0-1045-oem     5.10.0-1045.47 amd64        Signed kernel image oem
un  linux-image-5.11.0-27-generic   <none>         <none>       (no description available)
iF  linux-image-5.11.0-34-generic   5.11.0-34.36   amd64        Signed kernel image generic
rc  linux-image-5.11.0-35-generic   5.11.0-35.37   amd64        Signed kernel image generic
rc  linux-image-5.11.0-36-generic   5.11.0-36.40   amd64        Signed kernel image generic
rc  linux-image-5.11.0-37-generic   5.11.0-37.41   amd64        Signed kernel image generic
rc  linux-image-5.11.0-38-generic   5.11.0-38.42   amd64        Signed kernel image generic
ii  linux-image-5.11.0-40-generic   5.11.0-40.44   amd64        Signed kernel image generic
ii  linux-image-5.11.0-41-generic   5.11.0-41.45   amd64        Signed kernel image generic

The oldest currently installed one seems to be linux-image-5.11.0-34-generic, however sudo apt purge linux-image-5.11.0-34-generic suggests that it wants to also install linux-image-unsigned-5.11.0-34-generic, taking up more space.

What else can I try? Is it conceivably an option to resize/grow my bpool? I am pretty new to using ZFS.

Edit: I did some more investigating. zfs list bpool/BOOT/ubuntu_s2c09x -o space -r reports

NAME                                                             AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
bpool/BOOT/ubuntu_s2c09x                                          229M  1.51G     1.22G    303M             0B         0B
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_weekly-2021-11-19-1344        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-21-1339         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-22-1336         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-23-1340         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-24-1335         -    56K         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-25-1339         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-26-1338         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_weekly-2021-11-26-1343        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-27-1338         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-28-1336         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-29-1338         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-30-1340         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-01-1340         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-02-1340         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-03-1339         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_weekly-2021-12-03-1344        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-04-1337         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-05-1338         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-06-1338         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-07-1339         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-08-1337         -    80K         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-09-0645         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@autozsys_a2u0ly                             -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-10-1336         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_weekly-2021-12-10-1341        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-11-1339         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-12-1335         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@autozsys_omna1o                             -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@autozsys_5q7106                             -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-13-1336         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@autozsys_fwalez                             -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-14-1336         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@autozsys_dsm4th                             -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-15-1335         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@autozsys_4t769s                             -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-16-1336         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@autozsys_z8aa09                             -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-17-1339         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_weekly-2021-12-17-1344        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@autozsys_dqsnib                             -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@autozsys_7w0l6s                             -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@autozsys_323q08                             -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@autozsys_gqbsa5                             -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-18-1338         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-19-1339         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0217        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0317        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0417        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0517        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0617        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0717        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0817        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0917        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1017        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1117        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1217        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1317        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-21-1337         -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1417        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1517        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1617        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1717        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1817        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1917        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-2017        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-2117        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-2217        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-2317        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-22-0017        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_frequent-2021-12-22-0100      -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_frequent-2021-12-22-0115      -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-22-0117        -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_frequent-2021-12-22-0130      -     0B         -       -              -          -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_frequent-2021-12-22-0145      -     0B         -       -              -          -

I don't understand how it can claim that 1.22 GB are used by snapshots, but none of the listed bpool snapshots are using any space?

1

3 Answers 3

15

Uninstalling old and unused kernels and associated files is the correct action to take; to avoid the suggestion to install an unsigned kernel, remove packages that depend on linux-image-5.11.0-34-generic via:

sudo apt purge linux*5.11.0-34*

The second step required to free up space is to remove old snapshots referencing the files that were removed; list the snapshots via:

zfs list -r -t snapshot -o name,used,referenced,creation bpool/BOOT

You can see in the REFER column how much data is referenced. Identical numbers often coincide with minimal USED values; the snapshots are effectively copies of previous snapshots.

Manually remove the five oldest snapshots via:

zfs list -H -r -t snapshot -o name,used,referenced,creation bpool/BOOT | head -n 5 | cut -c 35-40 | xargs -n 1 sudo zsysctl state remove --system

Keep removing snapshots until you see space being freed up.

(Source of the last one-liner and more info: https://github.com/ubuntu/zsys/issues/155)

Notes:

  1. Growing a ZFS pool is possible if there is unallocated space, but Ubuntu allocates the remaining space to rpool. Shrinking a ZFS pool in-place is apparently not possible.

  2. I see auto-snapshotting is turned on for bpool, but unless you expect any files to change outside of APT, ZSys snapshotting should be sufficient and would reduce the number of snapshots you have to consider.

  3. To avoid this problem in the future, stay on top of kernel upgrades and remove old ones regularly. Also consider reducing the ZSys garbage cleaning defaults in /etc/zsys.conf. For an example, see: https://github.com/ubuntu/zsys/issues/155#issuecomment-758902487

13
  • 2
    auto-snapshotting on bpool does seem to be a part of the problem. Deleting older snapshots did indeed free up space, despite their minimal USED values. FYI, the one-liner to remove the 5 oldest snapshots breaks when auto-snapshots are present, but I could delete them all leaving only the ZSys ones via zfs list -r -t snapshot -o name bpool/BOOT | grep auto-snap | xargs -n 1 sudo zfs destroy.
    – l k
    Jan 3, 2022 at 19:59
  • 2
    USED only shows the size of data unique to that snapshot. Via -o referenced you can see REFER values, which capture all the data referenced by a snapshot, providing a total snapshot size (that often overlaps with that of other snapshots, most obviously when the REFER value is greater than the USED value). The REFER values should show that auto-snapshots reference a lot of old files, preventing disk space from being freed up until the last snapshot referencing those old files is destroyed. Thank you for sharing how you got rid of the auto-snapshots!
    – Alex
    Jan 7, 2022 at 14:55
  • How remove only large snapshots?
    – Fakhamatia
    Apr 5, 2022 at 15:44
  • @Fakhamatia, I manually check whether the value in the USED column is large, but generally assume that any snapshot after it will reference much of the same data. Therefore, only removing snapshots that appear large may not free much space, since they build on each others' data.
    – Alex
    Apr 11, 2022 at 15:55
  • 1
    @CiroSantilliOurBigBook.com I have edited my answer to include sudo so that it won't keep asking for a password five times.
    – Alex
    Jan 15, 2023 at 17:47
0

This will remove the oldest 5 kernels from zfs. It will not remove the most recent 5 if there are 5 or fewer kernels installed.

# Run "zfs list" to list all the installed kernels.
# Tail removes the header line
# head -n -5 preserves the last 5 in the list, listing all others
# head -5 removes only the first 5 (if at least that many exist)
zfs list -r -t snapshot -o name,used,referenced,creation bpool/BOOT | tail -n +2 | head -n -5 | head -5 | cut -c 35-40 | xargs -n 1 /usr/bin/sudo zsysctl state remove --system
# List the amount of space remaining on /boot
df -h /boot
1
  • That command seems broken to me.
    – lindhe
    Jul 11, 2023 at 18:52
0

I have been working with ZFS since its incarnation when i worked for Sun Microsystems. Super early adopter.

Anyway, snapshots are just that.... REMOVE THEM ALL ON THE BPOOL

for i in `zfs list -t snapshot | grep bpool | awk '{print $1}'`; do echo $i ; zfs destroy $i; done

you can then

zfs list | grep bpool
You should see a lot of space opened up.

Now you can try to update your server.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .