Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

I'd like to run the following incomplete script weekly in as a cron job to backup my home directory to an external drive mounted as /mnt/backups

#!/bin/bash
#

TIMEDATE=$(date +%b-%d-%Y-%k:%M)

LASTBACKUP=pathToDirWithLastBackup

rsync -avr --numeric-ids --link-dest=$LASTBACKUP /home/myfiles /mnt/backups/myfiles$TIMEDATE

My first question is how do I correctly set LASTBACKUP to the the the directory in /backs most recently created?

Secondly, I'm under the impression that using --link-desk will mean that files in previous backups will not will not copied in later backups if they still exist but will rather symbolically link back to the originally copied files? However, I don't want to retain old files forever. What would be the best way to remove all the backups before a certain date without losing files that may think linked in those backups by currents backups? Basically I'm looking to merge all the files before a certain date to a certain date if that makes more sense than the way I initially framed the question :). Can --link-dest create hard links, and if so, just deleting previous directories wouldn't actually remove linked file?

Finally I'd like to add a line to my script that compresses each newly created backup folder (/mnt/backups/myfiles$TIMEDATE). Based on reading this question, I was wondering if I could just use this line

gzip --rsyncable /backups/myfiles$TIMEDATE

after I run rsync so that sequential rsync --link-dest executions would find already copied and compressed files?

I know that's a lot, so many thanks in advance for your help!!

share|improve this question

4 Answers 4

You could use rsync.

Listing one: make_snapshot.sh

#!/bin/bash
# ----------------------------------------------------------------------
# mikes handy rotating-filesystem-snapshot utility
# ----------------------------------------------------------------------
# this needs to be a lot more general, but the basic idea is it makes
# rotating backup-snapshots of /home whenever called
# ----------------------------------------------------------------------

unset PATH  # suggestion from H. Milz: avoid accidental use of $PATH

# ------------- system commands used by this script --------------------
ID=/usr/bin/id;
ECHO=/bin/echo;

MOUNT=/bin/mount;
RM=/bin/rm;
MV=/bin/mv;
CP=/bin/cp;
TOUCH=/bin/touch;

RSYNC=/usr/bin/rsync;


# ------------- file locations -----------------------------------------

MOUNT_DEVICE=/dev/hdb1;
SNAPSHOT_RW=/root/snapshot;
EXCLUDES=/usr/local/etc/backup_exclude;


# ------------- the script itself --------------------------------------

# make sure we're running as root
if (( `$ID -u` != 0 )); then { $ECHO "Sorry, must be root.  Exiting..."; exit; } fi

# attempt to remount the RW mount point as RW; else abort
$MOUNT -o remount,rw $MOUNT_DEVICE $SNAPSHOT_RW ;
if (( $? )); then
{
    $ECHO "snapshot: could not remount $SNAPSHOT_RW readwrite";
    exit;
}
fi;


# rotating snapshots of /home (fixme: this should be more general)

# step 1: delete the oldest snapshot, if it exists:
if [ -d $SNAPSHOT_RW/home/hourly.3 ] ; then         \
$RM -rf $SNAPSHOT_RW/home/hourly.3 ;                \
fi ;

# step 2: shift the middle snapshots(s) back by one, if they exist
if [ -d $SNAPSHOT_RW/home/hourly.2 ] ; then         \
$MV $SNAPSHOT_RW/home/hourly.2 $SNAPSHOT_RW/home/hourly.3 ; \
fi;
if [ -d $SNAPSHOT_RW/home/hourly.1 ] ; then         \
$MV $SNAPSHOT_RW/home/hourly.1 $SNAPSHOT_RW/home/hourly.2 ; \
fi;

# step 3: make a hard-link-only (except for dirs) copy of the latest snapshot,
# if that exists
if [ -d $SNAPSHOT_RW/home/hourly.0 ] ; then         \
$CP -al $SNAPSHOT_RW/home/hourly.0 $SNAPSHOT_RW/home/hourly.1 ; \
fi;

# step 4: rsync from the system into the latest snapshot (notice that
# rsync behaves like cp --remove-destination by default, so the destination
# is unlinked first.  If it were not so, this would copy over the other
# snapshot(s) too!
$RSYNC                              \
    -va --delete --delete-excluded              \
    --exclude-from="$EXCLUDES"              \
    /home/ $SNAPSHOT_RW/home/hourly.0 ;

# step 5: update the mtime of hourly.0 to reflect the snapshot time
$TOUCH $SNAPSHOT_RW/home/hourly.0 ;

# and thats it for home.

# now remount the RW snapshot mountpoint as readonly

$MOUNT -o remount,ro $MOUNT_DEVICE $SNAPSHOT_RW ;
if (( $? )); then
{
    $ECHO "snapshot: could not remount $SNAPSHOT_RW readonly";
    exit;
} fi;

and the second:

Listing two: daily_snapshot_rotate.sh

#!/bin/bash
# ----------------------------------------------------------------------
# mikes handy rotating-filesystem-snapshot utility: daily snapshots
# ----------------------------------------------------------------------
# intended to be run daily as a cron job when hourly.3 contains the
# midnight (or whenever you want) snapshot; say, 13:00 for 4-hour snapshots.
# ----------------------------------------------------------------------

unset PATH

# ------------- system commands used by this script --------------------
ID=/usr/bin/id;
ECHO=/bin/echo;

MOUNT=/bin/mount;
RM=/bin/rm;
MV=/bin/mv;
CP=/bin/cp;

# ------------- file locations -----------------------------------------

MOUNT_DEVICE=/dev/hdb1;
SNAPSHOT_RW=/root/snapshot;

# ------------- the script itself --------------------------------------

# make sure we're running as root
if (( `$ID -u` != 0 )); then { $ECHO "Sorry, must be root.  Exiting..."; exit; } fi

# attempt to remount the RW mount point as RW; else abort
$MOUNT -o remount,rw $MOUNT_DEVICE $SNAPSHOT_RW ;
if (( $? )); then
{
    $ECHO "snapshot: could not remount $SNAPSHOT_RW readwrite";
    exit;
}
fi;


# step 1: delete the oldest snapshot, if it exists:
if [ -d $SNAPSHOT_RW/home/daily.2 ] ; then          \
$RM -rf $SNAPSHOT_RW/home/daily.2 ;             \
fi ;

# step 2: shift the middle snapshots(s) back by one, if they exist
if [ -d $SNAPSHOT_RW/home/daily.1 ] ; then          \
$MV $SNAPSHOT_RW/home/daily.1 $SNAPSHOT_RW/home/daily.2 ;   \
fi;
if [ -d $SNAPSHOT_RW/home/daily.0 ] ; then          \
$MV $SNAPSHOT_RW/home/daily.0 $SNAPSHOT_RW/home/daily.1;    \
fi;

# step 3: make a hard-link-only (except for dirs) copy of
# hourly.3, assuming that exists, into daily.0
if [ -d $SNAPSHOT_RW/home/hourly.3 ] ; then         \
$CP -al $SNAPSHOT_RW/home/hourly.3 $SNAPSHOT_RW/home/daily.0 ;  \
fi;

# note: do *not* update the mtime of daily.0; it will reflect
# when hourly.3 was made, which should be correct.

# now remount the RW snapshot mountpoint as readonly

$MOUNT -o remount,ro $MOUNT_DEVICE $SNAPSHOT_RW ;
if (( $? )); then
{
    $ECHO "snapshot: could not remount $SNAPSHOT_RW readonly";
    exit;
} fi;

After creating the script to your needs add it to cron jobs.

crontab -e

add the following:

0 */4 * * * /usr/local/bin/make_snapshot.sh

0 13 * * * /usr/local/bin/daily_snapshot_rotate.sh

They cause make_snapshot.sh to be run every four hours on the hour and daily_snapshot_rotate.sh to be run every day at 13:00 (that is, 1:00 PM).

source: http://www.mikerubel.org/computers/rsync_snapshots/

* * * * * command to be executed
- - - - -
| | | | |
| | | | ----- Day of week (0 - 7) (Sunday=0 or 7)
| | | ------- Month (1 - 12)
| | --------- Day of month (1 - 31)
| ----------- Hour (0 - 23)
------------- Minute (0 - 59)

If you want it to run hourly you would add a cron job for each hour.

Another possible option is using rsnapshot

  1. Install rsnapshot (available in software center)

  2. Configure rsnapshot and Specify Backup Source Directory

Open the /etc/rsnapshot.conf and uncomment the following lines.

# nano /etc/rsnapshot.conf

cmd_cp          /bin/cp
cmd_ssh /usr/bin/ssh
cmd_du          /usr/bin/du
cmd_rsnapshot_diff      /usr/local/bin/rsnapshot-diff
logfile /var/log/rsnapshot
  1. Define your destination backup directories in /etc/rsnapshot.conf as shown below. In this example,

    /home – source directory that should be backed-up localhost/ – destination directory where the backup will be stored. Please note that this directory will be created under /.snapshots/{internal.n}/ directory as shown in the last step.

    nano /etc/rsnapshot.conf

    backup /home/ localhost/

  2. Test rsnapshot Configuration

Perform configuration test to make sure rsnapshot is setup properly and ready to perform linux rsync backup.

# rsnapshot configtest
Syntax OK
  1. Verify rsnapshot Hourly Backup Configuration

You can backup linux directories or files at various intervals. By default, the hourly and daily backups are configured.

Verify the hourly backup configuration.

# rsnapshot -t hourly
echo 6490 > /var/run/rsnapshot.pid
mkdir -m 0700 -p /.snapshots/
mkdir -m 0755 -p /.snapshots/hourly.0/
/usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded /home \
/.snapshots/hourly.0/localhost/
mkdir -m 0755 -p /.snapshots/hourly.0/
/usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded /etc \
/.snapshots/hourly.0/localhost/
mkdir -m 0755 -p /.snapshots/hourly.0/
/usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded \
/usr/local /.snapshots/hourly.0/localhost/
touch /.snapshots/hourly.0/
  1. Verify rsnapshot Daily Backup Configuration

Verify the daily rsnapshot cwrsync backup process is configured properly.

# rsnapshot -t daily
echo 6493 > /var/run/rsnapshot.pid
mkdir -m 0700 -p /.snapshots/
/.snapshots/hourly.5 not present (yet), nothing to copy
  1. Add Crontab Entry for rsnapshot

Once you’ve verified that the rsync hourly and daily backup configurations are setup properly in the rsnapshot cwrsync utility, it is time to set this puppy up in the crontab as shown below.

# crontab -e
0 */4 * * * /usr/local/bin/rsnapshot hourly
30 23 * * * /usr/local/bin/rsnapshot daily

source: http://www.thegeekstuff.com/2009/08/tutorial-backup-linux-using-rsnapshot-rsync-utility/

share|improve this answer

So I will try to answer the second (and third) question: --link-dest creates additional hard links to existing files on the target filesystem. The number after the access permissions is the "link count", it tells you, how many links (think filenames) point to the file. For example:

$ touch x
$ ls -l
-rw-rw-r-- 1 frank frank 0 2011-01-07 22:18 x
$ ln x y
$ ls -l
-rw-rw-r-- 2 frank frank 0 2011-01-07 22:18 x

You can simply rm -rf old backups. rm means unlink. It decreases the link count of a file. A filesystem will only overwrite files with a link count of zero. If you create additional files with gzip, they will just use extra space and you loose the advantage of the hard links.

share|improve this answer
    
Thanks much, that clears it up for me! Now I just have to figure out how to add compression and I'm set :) –  Evan Jan 7 '11 at 22:15
    
A compressing filesystem might work, but I have no experience wth that. See ubuntuforums.org/showthread.php?t=245430 –  Frank Jan 7 '11 at 22:33

You may wish to use a tool that automates the whole process for you, such as rsnapshot, which seems to implement what you are trying to do.

share|improve this answer
1  
I'm looking into it right now, that does seem to easily do everything I was asking about in this questions, thanks!! –  Evan Jan 7 '11 at 20:22

One question at a time please, I'm only going to answer the first question and you can post the others to another question:

In order to know what was the last backup dir, you simply add:

echo /mnt/backups/myfiles$TIMEDATE > /mnt/backups/last.dir

To the very end of the script, and then change your LASTBACKUP= to:

LASTBACKUP=`cat /mnt/backups/last.dir`

That way you always know your going to be using the last complete backup dir and not any other files/folders created in that directory, or backup folders that failed (which would create a mass of duplication)

share|improve this answer
    
That is a very simple way to keep track of the last backup made, thanks! I don't really think the other questions warrant fully new questions though since they are very much based on the specific script I'm using. To me it's one question with three parts, but if it is better to edit it and then link follow up questions to this one I'd be happy to do it! –  Evan Jan 7 '11 at 20:10
    
If last.dir doesn't exist or if it is empty (first use) should I create it and run rysnc without the --link-desk or will it just ignore the attribute if it is empty? –  Evan Jan 7 '11 at 20:15
    
You should probably try and use hard links if you can, and since hard links can't cross file systems, the backup will be forced to copy them. –  Martin Owens -doctormo- Jan 7 '11 at 21:18
1  
An alternative approach is to simply create /mnt/backups/myfiles_last as a symbolic link to the last backup. You can then use /mnt/backups/myfiles_last for the --link-dest . –  João Pinto Jan 7 '11 at 22:57
    
That sounds overly complex Joao. –  Martin Owens -doctormo- Jan 7 '11 at 23:41

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.