I have a 1.4TiB drive which has Nextcloud & normal Mint 18.2 distro stuff on it. Now using 25% of capacity... and would like a backup of this. Am using dd to copy (very slow) but my question is will it attempt to copy the entire 1.4TiB or will it stop at the point where data is not present... Can it even tell the difference ?
It's VERY slow... I'm using pv to monitor progress and it says 15 hours.
Hmmm
Dave
dd doesn't pay attention to "files", just disk blocks (if you specify if=/dev/sda), so it's going to try to copy all 1.4TiB to the 750GB partition. This will fail.
Use a file-oriented backup tool like rsync, burp, or duplicity to just backup files, and not free space.
gziped image. Also, if you later want to recover a few specific files, you'd have to gunzip the entire image before able to mount it and recover them. It still seems better to use file-oriented tools like rsync, as waltinator suggests.
$HOME). To me, the latter case is more useful, but of course YMMV.
There are alternative device oriented or partition oriented cloning or backup tools.
You have used dd which is a very basic tool, that clones, copies every byte.
Clonezilla is another device oriented or partition oriented cloning or backup tool. It can tell the difference between blocks, that contain file data and free blocks, and it clones/copies only blocks that contain file data plus the data that contain information about the partition table and metadata of the file system in each partition. This makes Clonezilla faster than dd, particularly when there is a lot of free space on the device.
dd is notoriuosly dangerous and deserves the nickname 'Data Destroyer' becauses it does what you tell it to do without questions. If you tell it to delete the family pictures ..., and it is a minor typing error away.
Clonezilla has checkpoints, where you can double-check, that you will be writing to the correct target device (and not to the device (the main drive or an external drive), where you store valuable data).
Clonezilla can either clone directly, or create a compressed image of a whole device. This image is a a directory with several files, where the big files are compressed. Clonezilla can restore from an image to a new device (drive) with the same size or bigger, so that you can get a working system with the same properties as the original system, a clone.
Clonezilla can also {clone/make a compressed image} of one or more partitions separately, but in this case there will be no complete image to restore a complete clone of the original system.
I think that a Clonezilla image file will be small enough in your case to fit in the target drive, at least if the original source drive is far from full with already compressed data. So you can try to back up your data by creating an image file of the whole source drive.
In order to rely on the backup, you should get a third drive, of at least the same size as the original source drive, and try to restore from the image to this third drive. It should work as a replacement of the original drive.
If this is 'too much' for you, if it is enough to backup your personal data and maybe some of the settings in your computer, a file oriented backup according to @waltinator's answer will be better.
The problem with dd is that it WILL copy everything -- even sectors that contain nothing but binary zeros and are not allocated to any file. AND it is easy to get the parameters wrong -- earning it the moniker "disk destroyer".
May I suggest CloneZilla -- it's a bootable front end that sets up your backup/restore request and coordinates their execution. It is particularly good at backing up/restoring just one partition on a physical disk.
CloneZilla has three options as to which utility to use to do the actual backup -- one of which is dd, but it has faster and better options, too.
Jennifer
Making an image of the filesystem data, instead of doing a dumb block-by-block copy of the underlying block device, is called a "dump".
The inverse is called "restore". Those man pages for dump utilities mentions the restore program.
For incremental backups, something at the file level is usually much better, but dumpfs tools can often read faster. And since their output is a single stream, you can easily pipe it through a light-weight compressor like lz4 or lzop, or just gzip. It's also fast to write because it's a single stream, so doesn't require any seeking or meta-data ops on the destination.
It's also potentially faster to read the source, because it's aware of the FS layout and will read the file/directory data in approximately the order it appears on disk, only skipping over holes. Descending through the directories might result in an access pattern that's a lot slower, seeking all over the place if files in a directory aren't near each other on disk.
TL:DR: dump/restore is a good way to copy whole filesystems around when messing around with multiple disks. It's often fast.
ddis pretty slow on most devices if you don't use a more appropriate block size than the default 512 byte. For modern drives 8, 16 or 32 MiB should be fine.