TL;DR
After upgrading my ZFS pool from 0.7.x to 0.8.3 (and Ubuntu from 18.04.x to 20.04.1) my Nextcloud data backup is (almost) always a full backup. Before upgrade everything was fine plus my other, system's rpool, behaves as intended.
The true story
I have two TAR backups configured. System backup, which is and was all fine, and Nextcloud data backup, which also was fine but is not anymore. It had been working fine for over a year on ZFS 0.7.x and Ubuntu 18.04.x. Some time ago I migrated to Ubuntu 20.04 and then 20.04.1 and since first upgrade Nextcloud backup has been (almost) always a full backup. It happens, like 1 out of 10, that it makes an incremental backup as intended but unfortunately it is more like a glitch than a rule.
Juice
My backup isn't anything special:
tar -cpz \
--listed-incremental="$backupIncrementalMetadataFullFileName" \
--exclude="$backupLocation" \
--exclude="*RychuSrv*Backup*.*" \
"/srv/nextcloud/${nextcloudFolderName}" \
| tee "$tarBackupFullFileName" \
| gpg [censored]
ZFS happens?
I'm pointing the attention at ZFS because... what else? :) But I can't figure out what's causing this to happen. I tried comparing my behaving well rpool
with Nextcloud's one and, except obvious differences like dates or guids, I didn't find anything meaningful. Properties that have different value are:
- devices
- createtxg
- autotrim (
rpool
is SSD) - canmount (backup's has
on
andrpool
's hasnoauto
)
Other features/properties that I know can have an impact on the issue are atime
and realtime
and both on both pools are on
.
How do the files look like
So what is backed up are mostly image and video files in folders that most of them haven't changed for a very long time. For example:
# ls -1l . | tail -n 500 | head -n 10
-rw-r--r-- 1 www-data www-data 2113359 Jan 5 2020 IMG_20200105_172639.jpg
-rw-r--r-- 1 www-data www-data 2029782 Jan 5 2020 IMG_20200105_172641.jpg
-rw-r--r-- 1 www-data www-data 2374428 Jan 5 2020 IMG_20200105_172652.jpg
-rw-r--r-- 1 www-data www-data 2523738 Jan 5 2020 IMG_20200105_172654.jpg
-rw-r--r-- 1 www-data www-data 3405077 Jan 6 2020 IMG_20200106_083530.jpg
-rw-r--r-- 1 www-data www-data 1989491 Jan 6 2020 IMG_20200106_183744.jpg
-rw-r--r-- 1 www-data www-data 2220897 Jan 11 2020 IMG_20200111_131056.jpg
-rw-r--r-- 1 www-data www-data 2850718 Jan 11 2020 IMG_20200111_132928.jpg
-rw-r--r-- 1 www-data www-data 2095188 Jan 11 2020 IMG_20200111_132956.jpg
-rw-r--r-- 1 www-data www-data 2312352 Jan 11 2020 IMG_20200111_133414.jpg
# stat IMG_20200111_131056.jpg
File: IMG_20200111_131056.jpg
Size: 2220897 Blocks: 4369 IO Block: 131072 regular file
Device: 43h/67d Inode: 328087 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
Access: 2020-08-05 00:16:30.136312800 +0200
Modify: 2020-01-11 13:10:57.000000000 +0100
Change: 2020-01-13 14:36:14.531413322 +0100
Birth: -
You can see the file was accessed right after midnight that means the backup script pulled it in into backup.
Why? The file was changed over 6 months ago!
PS. I have just noticed that Access time has a different timezone. Isn't that strange? 🤔