2

To avoid having to reset my operating systems on my Lenovo laptop (80Q0001NUS) each time they fail, I need to be able to fully backup Ubuntu and Windows. I was hoping to make tar.xz files to save to my external hard drive, but I am completely unsure what to do even after searching online. My goal with my backups is to be able to restore either Ubuntu or Windows back to exactly how they were when they were compressed and archived. I have these partitions:

Storage Drives

128 GB SSD

  • /dev/sda1: FAT32 /boot/efi partition
  • /dev/sda2: NTFS Windows 10 Primary partition
  • /dev/sda3: BTRFS Ubuntu Root partition
  • /dev/sda4: Swap Partition

1 TB HDD

  • /dev/sdb1: NTFS (Windows) Storage partition
  • /dev/sdb2: EXT4 (Shared) Storage partition
  • /dev/sdb3: BTRFS (Ubuntu) Storage partition

4 TB HDD (External Backup Drive)

  • /dev/sdc1: NTFS partition
  • /dev/sdc2: EXT4 partition
  • /dev/sdc3: BTRFS partition

I am hoping to backup /dev/sda1 and /dev/sdb1 (separately if it's the best way) to /dev/sdc1, /dev/sdb2 to /dev/sdc2, and /dev/sda3 and /dev/sdb3 to /dev/sdc3. I need each of these backups to be easily organizable (like single files that I can use to restore each partition just as they were from the time of the backup creation). I also need them to be highly compressed if applicable. I don't want to backup the full partitions if avoidable. I just would like to restore the written data that I can restore to a similar setup of partitions and storage drives. Finally, I need to be able to restore my system with them like a LiveCD or something similar which will prevent any loss of software, files, etc. I don't want to have to download files and setup my operating systems again and again each time they fail. I hope to retain all of my configurations, settings, files, and anything else once the restoration is done. Please let me know the best way to do this. Thank you.

  • Do you want to backup the data inside the partitions or the whole partition? – Raphael Aug 8 '16 at 6:25
  • Have you looked at this? askubuntu.com/a/7811/579074 (I would have commented, but need 50 reputation for that...) – usbpc102 Aug 8 '16 at 6:30
  • @Raphael Here, I will edit my question to explain this. – NAE Aug 11 '16 at 21:33
  • @usbpc102 I have looked at such and similar, but it doesn't really help me understand which is the best for this situation. I need something to be able to create a highly (lossless) compressed backup archive of as much of both my Windows and Ubuntu operating systems (and any other operating systems I have in the future) as possible to allow me to restore both operating systems without any errors. I have to be able to restore Windows' files without breaking the system. I also need to be able to restore Ubuntu the same way. – NAE Aug 11 '16 at 21:49
  • For NTFS partitions, look into "ntfsclone". Comes with the default installation. For FAT12/16/32, just use dd. They usually are just small partitions anyway. For ext2/3/4, use the "dump" command. I do not know for btrfs. There surely is something... If you want to be able to restore to identical disks, also get yourself the MBR sector of the disk, wit partition table using dd. Technically, you can dd everything... You just need, well let's say quite some time. Use a decent buffer (bs=1G, for example) to speed it up a bit. Still, you'll need a lot of time. – jawtheshark Aug 11 '16 at 22:02
2

I will be assuming you booted with a LiveUSB or LiveCD througout my explanation and you need root for basically anything I explain here.

First lest me explain what "dd" is. The abbreviation means "disk dump" and can make an image over basically everything that is a block device (that's disks to you) of files. Let's go over the basics:

dd if=source-device of=target-device bs=blocksize count=numberofblocks
  • if Input File, aka, where the data comes from

  • of Output File, aka, where the data goes to

  • bs Block Size, aka, how much data you want to "buffer" in memory. If you do not use this, it basically copies directly from one device to another and will be slooooooow. For most operations, do specify something. I usually use something like bs=1G. Of course, you must have 1GB of free RAM to actually do that

  • count How many "block" (quantity defined by bs) you want to move. In the general case this is not needed, because you want all blocks.

Now, imagine you have a SSD of 128GB, named /dev/sda, and you want to "image" it (that's what the operation is called) fully, including everything that's on it. Partition tables. You have a backup disk with plenty of space on /mnt/bigdisk. To image the whole disk, you issue the following command:

dd if=/dev/sda of=/mnt/bigdisk/backup-20160812.img bs=1G

This will take quite a while (but you haven't lived until you do this on a 4TB disk), and finally you will find a 128GB file named backup-20160812.img in /mnt/bigdisk. dd will not give any output during that time and you will notice a big degradation of system performance. Oh, and for the love of all what is good: make sure nothing, not a single partition, is mounted or using /dev/sda.

Once you have the big file, you can do whatever you want with the real disk (well, except destroying it physically). If you want to restore, you do:

dd if=/mnt/bigdisk/backup-20160812.img of=/dev/sda bs=1G

These are bit-exact copies which means the following:

  • The size of the big file is exactly the size of the disk
  • Obviously you cannot restore to a disk, smaller than that size
  • Even if you restore to a disk larger, it may not work. Usually, it winds up being the partitions you want followed by a huge free space block. You'll have a lot of resizing to do and if it's an SSD, check alignment.
  • Should the source disk be defective, dd will stop upon errors. If you do not want this, you need to specify extra parameters. The man pages are obviously your friend.
  • Again, everything is imaged even unused space. If you have a disk of 100GB, and it's only 10% used, your file will be 100GB large and it will have wasted 90% of it's time on copying unused blocks.

With this knowledge, logic says, you can use dd to backup a single partition. Look at the following command an you may see why:

dd if=/dev/sda1 of=/mnt/bigdisk/backup-sda1-20160812.img bs=1G

The only difference is using /dev/sda1 instead of /dev/sda (and I used another target filename). Why? Because /dev/sda represents the full disk, and /dev/sda1 represents the first partition on that disk. That's it... All the remarks about being a bitwise copy persist.

The file generated to dd can be gzipped and will result in smaller files. You can do this using pipes. The image command would look like:

dd if=/dev/sda bs=1G | gzip > /mnt/bigdisk/backup-20160812.img.gz

The restore command something like this:

dd if=/mnt/bigdisk/backup-20160812.img.gz bs=1G | gunzip > /dev/sda

Some general remarks for understanding: Omitting the of will send all output to stdout. This data is send to gzip (or gunzip) using the pipe |. Since gzip/gunzip have no specifed file, they use this data, and gzip it. The output is send to stdout, which we then send to a file using the > symbol.

Now off to ntfs partitions. ntfsclone There are other associated tools you might want to look into (ntfsresize, ntfsfix. Type ntfs on the command line and do tab completion) Instead of just copying all bits, ntfsclone will copy the filesystem structure and data (unless you tell it not to), and thus ignore unused space. This translates into the fact that the files are much smaller and aren't much bigger than the actual "used" filesize of the ntfs partition. Of too the command:

ntfsclone -s -o /mnt/bigdisk/backup-20160812-sda2.ntfsclone /dev/sda2

It will give all sorts of information while it's working, which is much more reassuring than dd. Anyway, what does it all mean?

  • -s Means you want to save an image (and not do a device to device clone) - -o Where do you send the data to. In this case a file. If the file exists, it will refuse to overwrite it.
  • The last option in the command is the source, in this case the partition two of sda.

Here for the restore:

ntfsclone -r -O /dev/sda2 /mnt/bigdisk/backup-20160812-sda2.ntfsclone

Again, soothing information and here is what the options mean:

  • -r means restore from image. It basically is -s pendant for restore.
  • -O is where to write the data, in this case it's an upper case -O because that causes it to overwrite the file. This is necessary, because /dev/sda2 exists already and you want to overwrite it.
  • The final option, again, is the source of the data. This is, obviously, the file we just made with the first command.

The size of these images are not changeable. You're not going to restore it to a smaller disk/partition, even if it would fit based on the filesize. The structures of the file system that have been backed up, are bound to the size of the disk. So, even though that image you made of a 100GB partition with 10GB data could fit on a smaller 50GB, it's not going to work. You can restore to a a larger partition, but again, the structures stayed the same, so, you'll have to use ntfsresize in order to actually be able to use that extra space.

Let's get to dump. As I mentioned in my comment, I haven't used this in ages. I just backup my data files, as I know that Linux re installations are basically painless, especially if you keep /home on a different partition. What is writen here is basically what I found out while I wrote it. My OpenBSD backup scripts that use dump are so old, I wouldn't dare to say that I still know how they work. To dump.

dump -0 -f /mnt/bigdisk/backup-20160812-sda3.dump /dev/sda3

Again, what does this mean?

  • -0 is the level of the dump. This is to do incremental backups and is very useful, but for now, you'll remember that a level 0 dump means the full filesystem.
  • -f The data of the dump is sent to the specified file, in this case: /mnt/bigdisk/backup-20160812-sda3-level0.dump

Restore? Well, you need to know that you can only restore to a clean filesystem and mounted filesystem, so you first have to make sure the target is formatted an clean and mount it:

mkfs.ext4 /dev/sda3
mount /dev/sda3 /mnt
cd /mnt
restore -r -f /mnt/bigdisk/backup-20160812-sda3-level0.dump
  • -r means restore
  • -f from file
  • The target is the current directory, hence the /mnt

It will probably give you an warning like restore: ./lost+found: File exists, because lost+found exists on any newly create filesystem.

Interestingly, given the way it works, you should be able to restore to a smaller disk, but I haven't tried.

Now that I addressed "mounting". Doing any of these operations, shouldn't be done on mounted disks, except for the restore part of dump.

Finally, backing up the MBR and GPT: The MBR is easy and to be honest, I prefer using it as long as my disks don't exceed 2TB. Anyway, the MBR is basically block 0 on your disk, of which the first part is the boot code and the second part is the partition table. The first 446 bytes of the first sector, the next 66 Bytes are the partition. So, exctracting only the boot code, looks like this:

dd if=/dev/sda of=/mnt/bigdisk/backup-20160812-bootcode.dd bs=446 count=1

Extracting the boot code with the partition table looks like this:

dd if=/dev/sda of=/mnt/bigdisk/backup-20160812-mbr.dd bs=512 count=1

That's it. Obviously, you can only restore these to exactly the same disk (or one with exactly the same characteristics).

As you see MBR is simple. GPT is not and it's a pain. GPT is variable length, and you better Google it for a better understanding. I have no GPT disks on the machine I'm testing this, so double check everything. From what I Google, the tool to use is gdisk. Alas it seems to be an interactive tool. That's fine, but we want a simple one-liner:

printf "b\\n/mnt/bigdisk/backup-20160812-gpt.gdisk\n" | gdisk /dev/sda

Basically, send the text:

b
/mnt/bigdisk/backup-20160812-gpt.gdisk

to the applicaton gdisk working on device /dev/sda. The file /mnt/bigdisk/backup-20160812-gpt.gdisk should now contain a backup of the GPT. The restore should be something like this, but I did not try, so use at your own risk.

printf "b\\nr\\nl\\n/mnt/bigdisk/backup-20160812-gpt.gdisk" | gdisk /dev/sda

This should the following text to to the interactive gdisk menu, to go onto recovery mode and restore from file.

If you start Googling GPT in conjunction with dd, you'll see many people who warn you not to do this because of the uuid used by GPT. That is most certainly true: you will get problems if you have two disks active in the same system with the same uuid. Except of course, that's not what you are doing here. At no time, you'll have two identical uuids in the system if you stick to copying to files (which is what all my examples do)

As you see, this is rather a big chuck of information to digest and I'm not going to even bother reading it again for spelling mistakes, typos, etc. If it's all too much, look into Clonezilla. It might be closer to what you actually need. I might have even saved so much writing, had I started off with telling you that.

  • sgdisk seems to be the better tool for treating GPT from the command line. – jawtheshark Aug 12 '16 at 10:36

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.