4

Question

Using only the command line, is it possible to tell if a rsync backup (on a NAS) and the original folder tree (on a web server) are exactly the same? I mean in terms of file sizes, attributes, and symbolic links?

I feat the most for symbolic links. In case of a hard-drive failure – let's say I can solve all the booting issues – will I be able to put back all the files with all the symbolic links done right and their permissions / attributes?

Context I am setting up an rsync backup system in which my NAS (arm processor) connects to my dedicated webserver hosted far far away and does a backup of the / partition.

In case of a disk crash, I plan on making the exact same partition setup on the new disk, on installing the same operating system (14.04) so I get the MBR / boot loader in place.

Then I will rsync back the backup from the NAS to the server, reboot and cross my fingers.

The issue is that I will only know if everything works the day I need the backup…. But I can take measures to know if it will work before. One of them is checking that every single file, folder, sym link… in the backup has the same permissions, attributes, whatever than the original. I also need to check that the sym links and similar are correctly done.

The options I use with rsync are :

rsync -rlptgoDXvxzH --numeric-ids --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"}

The option A is causing issues.

EDIT: ADDITIONAL INFO

What I am looking for is a working backup strategy that will allow me:

– to perform blazing fast backups. Because I do backup the / system partition, I take most services down on the server (apache, postfix, dovecot, mysql, etc.) so I can have a safe backup. Any solution such as rsync is good, because I can update a mirror directory on the NAS. Then, offline, I can take care of the backup strategy (incremental, etc…) from the updated mirror.

– to have fast restore (I don't want to transfer a 100 GB disk image). I have a really really fast connection between the NAS and the server. I mean really fast. Transferring a 10 GB disk image is a matter of minutes.

What I can't do: change the physical setup of the hosted server. I have one 500 GB disk. That's all.

What I can do: boot the server in a rescue mode in which the disk isn't mounted at all. Rescue mode even works with a dead disk. I use this mode to perform binary images of partitions.

Current setup: / 10 GB, 17 % full SWAP 512 Mo /var 80 GB /bkup the rest… to 500 GB.

  • to save your MBR you will have to make a backup using dd: dd if=/dev/sda of=mbr-backup.img bs=512 count=1 you can write it afer the crash back onot your drive using dd if=mbr-backup.img of=/dev/sda bs=512 if sda is the essential drive – LittleByBlue Dec 28 '14 at 14:29
  • Thank you for your comment. I have a copy of the MBR. My current issue is: Without attempting an actual restore operation (which would bring offline my only web server AND kill it in case of a backup failure), how can I ensure that my backup will work as expected? I ask this because I did a "tar" backup… which failed to be restored… and it took me like 20 hours to set up everything right again, secure the server, install ISPConfig, re-create all the websites, restore the databases… The only option I currently have is the dd image. Which is so slow... – Adeline Dec 28 '14 at 19:41
  • Seems like this is a good question and nobody knows the answer. Are you able to simulate this e.g. on two old PCs or Raspberry Pis? – LittleByBlue Dec 29 '14 at 13:01
  • 1
    I will make an experiment with my Rpi with Raspbian GNU/Linux (will be copied) and Arch GNU/Linux (represents your NAS). – LittleByBlue Dec 29 '14 at 13:16
  • This doesn't exactly answer your question, but might be useful. I use backintime CLI version for my server. This is based on rsync and backs up nightly, saving incremental copies (as many as you specify). This means you can restore from the version three days ago, if something went wrong two days ago. In addition, every week I make it do an rsync checksum backup to verify all the existing backups. I've used it multiple times to restore my server. – Sparhawk Dec 29 '14 at 14:53
2

My experiment

  1. Setup

At first I created an Ext4 Partition on my PC using gparted.

Then I booted into a small Arch GNU/Linux installation in the hope that this would bring a better performance. Then I attached my Raspberry Pi with a only a bit modified Raspbian GNU/Linux Installation via Ethernet switch to the PC.

Then I mounted the Ext4 partition.

  1. Copying

After this I copied the / partition of the Pi to my PC.

rsync -rlptgoDXvxzH --numeric-ids --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} root@raspberrypi:/ rsync_mount/

This took about 20 minutes. If you copy your server to your NAS you may need

  • a flatrate (or a lot of money;) )
  • a lot of time

    1. bricking the Raspberry Pi

Then I bricked my Raspberry Pi using

ssh root@raspberrypi

rm -r /run /var /etc /usr

Then I had to unplug the power cable because nothing was working anymore.

  1. repairing the Raspberry Pi

Finally I plugged the SD-Card of the Rpi into my computer, mounted it and repaired it using

cp -rpvxH  rsync_mount/* rpi_mount/

after unmounting the SD-Card and booting the Rpi everything worked fine again.

Fazit

The part with rsync worked fine, but I do not know if it will work with a complete new installation. Maybe then the kernel will not fit to what you copy back on the server.

I will make another try, where I install a newer kernel version after rsyncing.

  • Thank you for your reply. However, I don't think I really understand it. I do not have a raspberry pi. Sometimes, I use arduino boards, when I have the need. But we are discussing a real-world web serverhosted far far away by a big company. – Adeline Dec 29 '14 at 16:07
  • The Raspberry Pi is just a small arm computer. Raspbian is a Debian GNU/Linux distro for the Raspberry Pi. You could do the same with two PCs, your server or whatever you want. The essential part is: I fetched the / filesystem from a computer, and copied it back onto it and it works – LittleByBlue Dec 29 '14 at 16:28
  • Thank you. Can I use that to tell, only with the command line, if a rsync backup (on a NAS) and the original folder tree (on a web server) are exactly the same? – Adeline Dec 29 '14 at 16:32
  • This tells you, that they are the same. If you copy it and everything works fine (I tested nearly everything) it means, that they are identical. – LittleByBlue Dec 29 '14 at 16:37
  • Do you have multiple partitions on your raspberry pi? Is the kernel not part of the copied data? – Adeline Dec 29 '14 at 16:38
1

What you're really asking is: What is a good Backup Strategy for my server and that depends what you want to do when the lightning hits your server...

For me, speedy restore is what I want, so what I do is:

I have one additional small, old, cheap disk containing 2 additional partitions on the server: a 500GB ext4 partition containing the system back-ups and a 500MB FAT32 boot, diag partition (yep, MegaByte, and it's a bit too big!) containing CloneZilla.

Whenever I'm going to do some drastic changes to the server or whenever I feel I can afford a 5 minute down-time, I boot the server into the FAT32 CloneZilla partition, which then backs up my root partition excluding "/home" in 5 minutes! (using disk-to-image)

I've had to restore the image only once and it works like a charm: server was set back 1 month (as I wasn't sure when I introduced the problem), installed all its updates and then I had to redo my work from the last month which was basically installing a few utilities.

In addition to that, I rsync my /home and the system image backup partition weekly at Monday 03:00 am (which is when the server has its lowest usage).

If that's good enough for you, use the same strategy.

  • 1
    Thank you for your help. I have looked at many backup strategies. I even implemented one following a guide (TAR backup on ubuntu official website). It just failed to restore properly after a disk crash. The server had troubles booting properly and apache wasn't working right. I suspect it was a sym link / permission issue. Plus TAR is too slow to perform the backup for me. – Adeline Dec 29 '14 at 16:05
  • 1
    No. I would like to know if there is a tool capable of telling me if the original file/folder architecture is the same than that of the copy, expecially regarding sym links and file permission (special / extended attributes). That, I think, is the reason why my previous backup failed. Also, I can't add a second disk in my server… I have updated my question to reflect this fact. – Adeline Dec 29 '14 at 16:10
  • 1
    You do not need this diff tool, in my answer I prooved that symlinks etc. are working with rsync. – LittleByBlue Dec 29 '14 at 16:31
  • 1
    Thank you. I guess that's what I wanted to know, deep inside. Will it work. But given my (very negative) experience with a tar backup that everyone told me would work, I was a little skeptical and needed to see for myself. I just have one last question: my / and /var are on different physical partitions (sda1 and sda3). If there are sym links between the two (I think there are between /etc and /var), will they be correctly restored with the backup? – Adeline Dec 29 '14 at 16:36
  • 1
    edit: with the rsync option --copy-links you may need more disk space, but you also can prevent the mount problem. – LittleByBlue Dec 29 '14 at 16:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.