Script for an incremental file system backup

Question

I'd like to run the following incomplete script weekly in as a cron job to backup my home directory to an external drive mounted as /mnt/backups

#!/bin/bash
#

TIMEDATE=$(date +%b-%d-%Y-%k:%M)

LASTBACKUP=pathToDirWithLastBackup

rsync -avr --numeric-ids --link-dest=$LASTBACKUP /home/myfiles /mnt/backups/myfiles$TIMEDATE

My first question is how do I correctly set LASTBACKUP to the the the directory in /backs most recently created?

Secondly, I'm under the impression that using --link-desk will mean that files in previous backups will not will not copied in later backups if they still exist but will rather symbolically link back to the originally copied files? However, I don't want to retain old files forever. What would be the best way to remove all the backups before a certain date without losing files that may think linked in those backups by currents backups? Basically I'm looking to merge all the files before a certain date to a certain date if that makes more sense than the way I initially framed the question :). Can --link-dest create hard links, and if so, just deleting previous directories wouldn't actually remove linked file?

Finally I'd like to add a line to my script that compresses each newly created backup folder (/mnt/backups/myfiles$TIMEDATE). Based on reading this question, I was wondering if I could just use this line

gzip --rsyncable /backups/myfiles$TIMEDATE

after I run rsync so that sequential rsync --link-dest executions would find already copied and compressed files?

I know that's a lot, so many thanks in advance for your help!!

Answer 1

You may wish to use a tool that automates the whole process for you, such as rsnapshot, which seems to implement what you are trying to do.

Answer 2

One question at a time please, I'm only going to answer the first question and you can post the others to another question:

In order to know what was the last backup dir, you simply add:

echo /mnt/backups/myfiles$TIMEDATE > /mnt/backups/last.dir

To the very end of the script, and then change your LASTBACKUP= to:

LASTBACKUP=`cat /mnt/backups/last.dir`

That way you always know your going to be using the last complete backup dir and not any other files/folders created in that directory, or backup folders that failed (which would create a mass of duplication)

Answer 3

You could use rsync.

Listing one: make_snapshot.sh

#!/bin/bash
# ----------------------------------------------------------------------
# mikes handy rotating-filesystem-snapshot utility
# ----------------------------------------------------------------------
# this needs to be a lot more general, but the basic idea is it makes
# rotating backup-snapshots of /home whenever called
# ----------------------------------------------------------------------

unset PATH  # suggestion from H. Milz: avoid accidental use of $PATH

# ------------- system commands used by this script --------------------
ID=/usr/bin/id;
ECHO=/bin/echo;

MOUNT=/bin/mount;
RM=/bin/rm;
MV=/bin/mv;
CP=/bin/cp;
TOUCH=/bin/touch;

RSYNC=/usr/bin/rsync;


# ------------- file locations -----------------------------------------

MOUNT_DEVICE=/dev/hdb1;
SNAPSHOT_RW=/root/snapshot;
EXCLUDES=/usr/local/etc/backup_exclude;


# ------------- the script itself --------------------------------------

# make sure we're running as root
if (( `$ID -u` != 0 )); then { $ECHO "Sorry, must be root.  Exiting..."; exit; } fi

# attempt to remount the RW mount point as RW; else abort
$MOUNT -o remount,rw $MOUNT_DEVICE $SNAPSHOT_RW ;
if (( $? )); then
{
    $ECHO "snapshot: could not remount $SNAPSHOT_RW readwrite";
    exit;
}
fi;


# rotating snapshots of /home (fixme: this should be more general)

# step 1: delete the oldest snapshot, if it exists:
if [ -d $SNAPSHOT_RW/home/hourly.3 ] ; then         \
$RM -rf $SNAPSHOT_RW/home/hourly.3 ;                \
fi ;

# step 2: shift the middle snapshots(s) back by one, if they exist
if [ -d $SNAPSHOT_RW/home/hourly.2 ] ; then         \
$MV $SNAPSHOT_RW/home/hourly.2 $SNAPSHOT_RW/home/hourly.3 ; \
fi;
if [ -d $SNAPSHOT_RW/home/hourly.1 ] ; then         \
$MV $SNAPSHOT_RW/home/hourly.1 $SNAPSHOT_RW/home/hourly.2 ; \
fi;

# step 3: make a hard-link-only (except for dirs) copy of the latest snapshot,
# if that exists
if [ -d $SNAPSHOT_RW/home/hourly.0 ] ; then         \
$CP -al $SNAPSHOT_RW/home/hourly.0 $SNAPSHOT_RW/home/hourly.1 ; \
fi;

# step 4: rsync from the system into the latest snapshot (notice that
# rsync behaves like cp --remove-destination by default, so the destination
# is unlinked first.  If it were not so, this would copy over the other
# snapshot(s) too!
$RSYNC                              \
    -va --delete --delete-excluded              \
    --exclude-from="$EXCLUDES"              \
    /home/ $SNAPSHOT_RW/home/hourly.0 ;

# step 5: update the mtime of hourly.0 to reflect the snapshot time
$TOUCH $SNAPSHOT_RW/home/hourly.0 ;

# and thats it for home.

# now remount the RW snapshot mountpoint as readonly

$MOUNT -o remount,ro $MOUNT_DEVICE $SNAPSHOT_RW ;
if (( $? )); then
{
    $ECHO "snapshot: could not remount $SNAPSHOT_RW readonly";
    exit;
} fi;

and the second:

Listing two: daily_snapshot_rotate.sh

#!/bin/bash
# ----------------------------------------------------------------------
# mikes handy rotating-filesystem-snapshot utility: daily snapshots
# ----------------------------------------------------------------------
# intended to be run daily as a cron job when hourly.3 contains the
# midnight (or whenever you want) snapshot; say, 13:00 for 4-hour snapshots.
# ----------------------------------------------------------------------

unset PATH

# ------------- system commands used by this script --------------------
ID=/usr/bin/id;
ECHO=/bin/echo;

MOUNT=/bin/mount;
RM=/bin/rm;
MV=/bin/mv;
CP=/bin/cp;

# ------------- file locations -----------------------------------------

MOUNT_DEVICE=/dev/hdb1;
SNAPSHOT_RW=/root/snapshot;

# ------------- the script itself --------------------------------------

# make sure we're running as root
if (( `$ID -u` != 0 )); then { $ECHO "Sorry, must be root.  Exiting..."; exit; } fi

# attempt to remount the RW mount point as RW; else abort
$MOUNT -o remount,rw $MOUNT_DEVICE $SNAPSHOT_RW ;
if (( $? )); then
{
    $ECHO "snapshot: could not remount $SNAPSHOT_RW readwrite";
    exit;
}
fi;


# step 1: delete the oldest snapshot, if it exists:
if [ -d $SNAPSHOT_RW/home/daily.2 ] ; then          \
$RM -rf $SNAPSHOT_RW/home/daily.2 ;             \
fi ;

# step 2: shift the middle snapshots(s) back by one, if they exist
if [ -d $SNAPSHOT_RW/home/daily.1 ] ; then          \
$MV $SNAPSHOT_RW/home/daily.1 $SNAPSHOT_RW/home/daily.2 ;   \
fi;
if [ -d $SNAPSHOT_RW/home/daily.0 ] ; then          \
$MV $SNAPSHOT_RW/home/daily.0 $SNAPSHOT_RW/home/daily.1;    \
fi;

# step 3: make a hard-link-only (except for dirs) copy of
# hourly.3, assuming that exists, into daily.0
if [ -d $SNAPSHOT_RW/home/hourly.3 ] ; then         \
$CP -al $SNAPSHOT_RW/home/hourly.3 $SNAPSHOT_RW/home/daily.0 ;  \
fi;

# note: do *not* update the mtime of daily.0; it will reflect
# when hourly.3 was made, which should be correct.

# now remount the RW snapshot mountpoint as readonly

$MOUNT -o remount,ro $MOUNT_DEVICE $SNAPSHOT_RW ;
if (( $? )); then
{
    $ECHO "snapshot: could not remount $SNAPSHOT_RW readonly";
    exit;
} fi;

After creating the script to your needs add it to cron jobs.

crontab -e

add the following:

0 */4 * * * /usr/local/bin/make_snapshot.sh

0 13 * * * /usr/local/bin/daily_snapshot_rotate.sh

They cause make_snapshot.sh to be run every four hours on the hour and daily_snapshot_rotate.sh to be run every day at 13:00 (that is, 1:00 PM).

source: http://www.mikerubel.org/computers/rsync_snapshots/

* * * * * command to be executed
- - - - -
| | | | |
| | | | ----- Day of week (0 - 7) (Sunday=0 or 7)
| | | ------- Month (1 - 12)
| | --------- Day of month (1 - 31)
| ----------- Hour (0 - 23)
------------- Minute (0 - 59)

If you want it to run hourly you would add a cron job for each hour.

Another possible option is using rsnapshot

1. Install rsnapshot (available in software center)

2. Configure rsnapshot and Specify Backup Source Directory

Open the /etc/rsnapshot.conf and uncomment the following lines.

# nano /etc/rsnapshot.conf

cmd_cp          /bin/cp
cmd_ssh /usr/bin/ssh
cmd_du          /usr/bin/du
cmd_rsnapshot_diff      /usr/local/bin/rsnapshot-diff
logfile /var/log/rsnapshot

1. Define your destination backup directories in /etc/rsnapshot.conf as shown below. In this example,

   /home – source directory that should be backed-up localhost/ – destination directory where the backup will be stored. Please note that this directory will be created under /.snapshots/{internal.n}/ directory as shown in the last step.

   nano /etc/rsnapshot.conf

   backup /home/ localhost/

2. Test rsnapshot Configuration

Perform configuration test to make sure rsnapshot is setup properly and ready to perform linux rsync backup.

# rsnapshot configtest
Syntax OK

1. Verify rsnapshot Hourly Backup Configuration

You can backup linux directories or files at various intervals. By default, the hourly and daily backups are configured.

Verify the hourly backup configuration.

# rsnapshot -t hourly
echo 6490 > /var/run/rsnapshot.pid
mkdir -m 0700 -p /.snapshots/
mkdir -m 0755 -p /.snapshots/hourly.0/
/usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded /home \
/.snapshots/hourly.0/localhost/
mkdir -m 0755 -p /.snapshots/hourly.0/
/usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded /etc \
/.snapshots/hourly.0/localhost/
mkdir -m 0755 -p /.snapshots/hourly.0/
/usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded \
/usr/local /.snapshots/hourly.0/localhost/
touch /.snapshots/hourly.0/

1. Verify rsnapshot Daily Backup Configuration

Verify the daily rsnapshot cwrsync backup process is configured properly.

# rsnapshot -t daily
echo 6493 > /var/run/rsnapshot.pid
mkdir -m 0700 -p /.snapshots/
/.snapshots/hourly.5 not present (yet), nothing to copy

1. Add Crontab Entry for rsnapshot

Once you’ve verified that the rsync hourly and daily backup configurations are setup properly in the rsnapshot cwrsync utility, it is time to set this puppy up in the crontab as shown below.

# crontab -e
0 */4 * * * /usr/local/bin/rsnapshot hourly
30 23 * * * /usr/local/bin/rsnapshot daily

source: http://www.thegeekstuff.com/2009/08/tutorial-backup-linux-using-rsnapshot-rsync-utility/

Answer 4

So I will try to answer the second (and third) question: --link-dest creates additional hard links to existing files on the target filesystem. The number after the access permissions is the "link count", it tells you, how many links (think filenames) point to the file. For example:

$ touch x
$ ls -l
-rw-rw-r-- 1 frank frank 0 2011-01-07 22:18 x
$ ln x y
$ ls -l
-rw-rw-r-- 2 frank frank 0 2011-01-07 22:18 x

You can simply rm -rf old backups. rm means unlink. It decreases the link count of a file. A filesystem will only overwrite files with a link count of zero. If you create additional files with gzip, they will just use extra space and you loose the advantage of the hard links.