0

Short summary:

When creating a single zfs disk pool consisting of only one disk with a capacity of 1TB (= 931GiB) the file system only showed 899 GiB free space (df -h or zfs list; zpool list actually showed the partition size (931 GiB) minus some overhead (resulting in 928 GiB space).

Longer version:

I was trying to setup a zfs disk pool consisting of only one disk with a capacity of 1TB (= 931,53 GiB):

# fdisk -l /dev/sdb
Disk /dev/sdb: 931.53 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: xxxx
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: xxx

Device     Start        End    Sectors   Size Type
/dev/sdb1   2048 1953516976 1953514929 931.5G Linux filesystem

When setting up a zfs pool

# zpool create -f -o ashift=12 tank /dev/sdb1;

there are 32,5 GiB missing:

# zfs list
NAME   USED  AVAIL     REFER  MOUNTPOINT
tank   360K   899G       96K  /tank

What is causing a 32,5 GiB overhead on a single disk setup?

zpool list is reporting:

# zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
tank   928G   444K   928G        -         -     0%     0%  1.00x    ONLINE  -

but this isn't the actually useable space since also df -h is reporting:

# df -h
Filesystem                  Size  Used Avail Use% Mounted on
tank                        899G  128K  899G   1% /tank

There is no quota or reservation set:

# zfs get quota
NAME  PROPERTY  VALUE  SOURCE
tank  quota     none   default

# zfs get reservation
NAME  PROPERTY     VALUE   SOURCE
tank  reservation  none    default

# zfs get refquota
NAME  PROPERTY  VALUE   SOURCE
tank  refquota  none    default

# zfs get refreservation
NAME  PROPERTY        VALUE      SOURCE
tank  refreservation  none       default

# zfs get usedbyrefreservation
NAME  PROPERTY              VALUE          SOURCE
tank  usedbyrefreservation  0B             -

Creating the zpool with ashift=9 doesn't make any difference.

I can live with an actual overhead of 3,5 GiB (= partition size vs. zpool reported size), but not with an overhead of 32,5 GiB or 29 GiB (= zfs reported size - zpool reported size) on such a small disk.

When trying the same with btrfs I get more free space:

# btrfs filesystem show
Label: 'tank'  uuid: xxx
        Total devices 1 FS bytes used 128.00KiB
        devid    1 size 931.51GiB used 2.02GiB path /dev/sdb1

# df -h
Filesystem                  Size  Used Avail Use% Mounted on
/dev/sdb1                   932G  3.8M  930G   1% /mnt

more details

# zfs --version
zfs-0.8.3-1ubuntu12.5
zfs-kmod-0.8.3-1ubuntu12.5

# uname -a
Linux xxx 5.4.0-58-generic #64-Ubuntu SMP Wed Dec 9 08:16:25 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/os-release | grep VERSION
VERSION="20.04.1 LTS (Focal Fossa)"
VERSION_ID="20.04"
VERSION_CODENAME=focal

# zfs list -o space
NAME  AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
tank   899G  88.5K        0B     24K             0B      64.5K

Update: re-created with the command zpool create -oashift=12 tank /dev/sdb1.

No difference:

# zpool list
NAME             SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
tank             928G   432K   928G        -         -     0%     0%  1.00x    ONLINE  -

# zfs list
NAME                  USED  AVAIL     REFER  MOUNTPOINT
tank                  336K   899G       96K  /tank

Output of zdb tank | grep metaslab | tail -n 3:

# zdb tank | grep metaslab | tail -n 3
loading concrete vdev 0, metaslab 115 of 116 ...
        metaslab    114   offset   e400000000   spacemap      0   free       8G
        metaslab    115   offset   e600000000   spacemap      0   free       8G
        vdev          0         metaslabs  116          fragmentation  0%

Output of zdb | grep metaslab_shift:

# zdb | grep metaslab_shift
            metaslab_shift: 33
1
  • 1
    as I know there is a percentage of the filesystem that is reserved for snapshots
    – Lews
    Jan 1, 2021 at 23:58

2 Answers 2

1

I cannot reproduce your problem.

root@banshee:/tmp# truncate -s 931G disk.bin
root@banshee:/tmp# zpool create -oashift=12 test /tmp/disk.bin
root@banshee:/tmp# zpool list test
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
test   928G   480K   928G        -         -     0%     0%  1.00x    ONLINE  -

In the above example, I begin with a 931GiB (1TB, roughly) block device, and create a pool on it using 4KiB sectors. The available capacity is 928GiB, as expected when you account for metaslab remainder.

root@banshee:/tmp# zdb test | grep metaslab | tail -n 3
loading concrete vdev 0, metaslab 115 of 116 ...
    metaslab    114   offset   e400000000   spacemap      0   free       8G
    metaslab    115   offset   e600000000   spacemap      0   free       8G
    vdev          0     metaslabs  116      fragmentation  0%

My 931GiB "disk" was divided into 116 8GiB metaslabs; this left a 0.44125 metaslab remainder.

0.44125 metaslabs * 8GiB/metaslab == 3.53GiB

931GiB disk - 3.53Gib metaslab remainder == ~~928GiB usable

... and Bob's your uncle. Why you are seeing roughly ten times that amount of overhead, I have no idea; I'm also on Focal, with the same ZFS version as you're reporting.

root@banshee:/tmp# apt policy zfsutils-linux
zfsutils-linux:
  Installed: 0.8.3-1ubuntu12.5
  Candidate: 0.8.3-1ubuntu12.5
  Version table:
 *** 0.8.3-1ubuntu12.5 500
        500 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     0.8.3-1ubuntu12 500
        500 http://us.archive.ubuntu.com/ubuntu focal/main amd64 Packages

It might help to see the actual creation of your pool itself command by command from scratch and/or the output of zdb | grep metaslab_shift on that pool.

1
  • can you post the output of zfs list? zpool list also printed the correct free space for me.
    – cre
    Jan 2, 2021 at 19:33
0

As pointed out by the user grenkins in the subreddit /r/zfs (https://www.reddit.com/r/zfs/comments/kp2nnk/zfs_single_disk_setup_1tb_9315_gib_325_gib/ - all credit goes to him) zfs is reserving 3,2% space by default. This reservation is done by the kernel module on the system and has nothing to do with the quota/reservation/refquota/refreservation/usedbyrefreservation zpool settings. See https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Module%20Parameters.html#spa-slop-shift

The reservation is done by the spa_slop_shift setting of the zfs kernel module.

Summary:

Normally, the last 3.2% (1/(2^spa_slop_shift)) of pool space is reserved to ensure the pool doesn’t run completely out of space [...]
For large pools, when 3.2% may be too conservative and more usable space is desired, consider increasing spa_slop_shift

I increased the spa_slop_shift setting to 15 by creating the following file:

# printf "options zfs spa_slop_shift=15" > /etc/modprobe.d/zfs.conf

After a reboot the are 928 GiB available:

# zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
tank   928G   576K   928G        -         -     0%     0%  1.00x    ONLINE  -

# zfs list
NAME   USED  AVAIL     REFER  MOUNTPOINT
tank   408K   928G       96K  /tank

which is actually the free space zpool list was reporting (931,5 partition size - some small overhead).

So this is fixed for me.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .