TAR `--listed-incremental` is always a FULL backup after ZFS (and Ubuntu) upgrade

Question

TL;DR

After upgrading my ZFS pool from 0.7.x to 0.8.3 (and Ubuntu from 18.04.x to 20.04.1) my Nextcloud data backup is (almost) always a full backup. Before upgrade everything was fine plus my other, system's rpool, behaves as intended.

The true story

I have two TAR backups configured. System backup, which is and was all fine, and Nextcloud data backup, which also was fine but is not anymore. It had been working fine for over a year on ZFS 0.7.x and Ubuntu 18.04.x. Some time ago I migrated to Ubuntu 20.04 and then 20.04.1 and since first upgrade Nextcloud backup has been (almost) always a full backup. It happens, like 1 out of 10, that it makes an incremental backup as intended but unfortunately it is more like a glitch than a rule.

Juice

My backup isn't anything special:

tar -cpz \
    --listed-incremental="$backupIncrementalMetadataFullFileName" \
    --exclude="$backupLocation" \
    --exclude="*RychuSrv*Backup*.*" \
    "/srv/nextcloud/${nextcloudFolderName}" \
        | tee "$tarBackupFullFileName" \
        | gpg [censored]

ZFS happens?

I'm pointing the attention at ZFS because... what else? :) But I can't figure out what's causing this to happen. I tried comparing my behaving well rpool with Nextcloud's one and, except obvious differences like dates or guids, I didn't find anything meaningful. Properties that have different value are:

• devices
• createtxg
• autotrim (rpool is SSD)
• canmount (backup's has on and rpool's has noauto)

Other features/properties that I know can have an impact on the issue are atime and realtime and both on both pools are on.

How do the files look like

So what is backed up are mostly image and video files in folders that most of them haven't changed for a very long time. For example:

# ls -1l . | tail -n 500 | head -n 10
-rw-r--r-- 1 www-data www-data    2113359 Jan  5  2020 IMG_20200105_172639.jpg
-rw-r--r-- 1 www-data www-data    2029782 Jan  5  2020 IMG_20200105_172641.jpg
-rw-r--r-- 1 www-data www-data    2374428 Jan  5  2020 IMG_20200105_172652.jpg
-rw-r--r-- 1 www-data www-data    2523738 Jan  5  2020 IMG_20200105_172654.jpg
-rw-r--r-- 1 www-data www-data    3405077 Jan  6  2020 IMG_20200106_083530.jpg
-rw-r--r-- 1 www-data www-data    1989491 Jan  6  2020 IMG_20200106_183744.jpg
-rw-r--r-- 1 www-data www-data    2220897 Jan 11  2020 IMG_20200111_131056.jpg
-rw-r--r-- 1 www-data www-data    2850718 Jan 11  2020 IMG_20200111_132928.jpg
-rw-r--r-- 1 www-data www-data    2095188 Jan 11  2020 IMG_20200111_132956.jpg
-rw-r--r-- 1 www-data www-data    2312352 Jan 11  2020 IMG_20200111_133414.jpg

# stat IMG_20200111_131056.jpg
  File: IMG_20200111_131056.jpg
  Size: 2220897         Blocks: 4369       IO Block: 131072 regular file
Device: 43h/67d Inode: 328087      Links: 1
Access: (0644/-rw-r--r--)  Uid: (   33/www-data)   Gid: (   33/www-data)
Access: 2020-08-05 00:16:30.136312800 +0200
Modify: 2020-01-11 13:10:57.000000000 +0100
Change: 2020-01-13 14:36:14.531413322 +0100
 Birth: -

You can see the file was accessed right after midnight that means the backup script pulled it in into backup.

Why? The file was changed over 6 months ago!

PS. I have just noticed that Access time has a different timezone. Isn't that strange? 🤔

Answer 1

I have just learned that modification timestamp is not the only thing that determines whether the file was modified or not (in TAR terms). Snapshot file also contains information on which drive the file is located. Which device was used last time can be checked inside a snapshot file with the tar-snapshot-edit script. Unfortunately looks like that device ids like to change often. Following this link, where more details can be found:

Various situations can cause device numbers to change: upgrading your kernel version, reconfiguring your hardware, loading kernel modules in a different order, using virtual volumes that are assembled dynamically (such as with LVM or RAID), hot-plugging drives (e.g. external USB or Firewire drives), etc. In the majority of cases, this change is unnoticed by the users. However, it influences tar incremental backups: the device number is stored in tar snapshot files (see section Format of the Incremental Snapshot Files) and is used to determine whether the file has changed since the last backup. If the device numbers change for some reason, by default the next backup you run will be a full backup.

In my case, the easiest fix is to simply add --no-check-device option which does the job.