Rsync

Notes in progress

For backups

Initiation (assuming use of the standard backup or replication scripts):

In the root of your mounted storage volume, create a directory for the backup target's data:

mkdir mytarget

Then make an initial copy run

# rsync --old-compress -rlptD --numeric-ids --hard-links --inplace --acls --xattrs --rsync-path "/usr/bin/sudo /usr/bin/rsync" --link-dest /mnt/replication/digital-photos/latest --old-compress replicationv2@10.0.10.101:/srv/rsync-target/digital-photos/current/ /mnt/replication/digital-photos/inprogress &

If your initial copy is while both target and backup storage are on the same network, you may get greater tranfer speeds without the "compress" statement, and if both are relatively current installs, you can just use --compress rather than --old-compress, which is only needed if the live server has an old version of rsync.

If you are moving an existing multi-day backup set, you can use

# rsync -aHP sourceserver:/path/* /pathto/mytarget

which aims to preserve time/date stamps, permissions and hardlinks.

Prune script

This is a script that deletes old entries of rsync backup.

#!/bin/bash

dir=/srv/rsync-target/

threemonthago=`date -d '-3 month' +"%y-%m-%d"`
year=`date -d $threemonthago +%y`
month=`date -d $threemonthago +%m`
find $dir -name ${year}-${month}-0* -delete

fivemonthago=`date -d '-5 month' +"%y-%m-%d"`
year=`date -d $fivemonthago +%y`
month=`date -d $fivemonthago +%m`
find $dir -name ${year}-${month}-2* -delete

oneyearago=`date -d '-1 year' +"%y-%m-%d"`
year=`date -d $oneyearago +%y`
month=`date -d $oneyearago +%m`
if [ $month -eq 02 ]; then
	find $dir -name ${year}-${month}-1* -not -path ${dir}*${year}-${month}-19* -delete
else
	find $dir -name ${year}-${month}-1* -delete
	find $dir -name ${year}-${month}-31* -delete
fi

This crontab command can be used for setting the script to run first sunday of every month:

00 01 * * 0 [ $(date +\%d) -le 07 ] && /path/to/script

Tidying up

Script for tidying up after an outage causes previous backup to fail (start but not complete, leaving the inprogress directory)

NB only use this script if the previous backup failed to complete, but mostly finished, else the next run will copy lots of new data. If the backup failed early in its normal runtime (which you can get an idea of from the log entries, or by using ls -l to see which directories have the right timestamp (as opposed to the timestamp of the failed backup run). If it failed fairly early, it's best to just delete the "inprogress" directory, and if relevant resolve whatever issue caused the backup to fail.

if [ $# -ne 1 ] ; then
       echo "Error: you need to provide the full path to a directory"
       exit 1
fi

#!/bin/bash
DIRDATE=`/bin/date --date="1 day ago" +"%F"`

FIRSTCHAR=${1:0:1}

echo $FIRSTCHAR

if [ ${FIRSTCHAR} = "/" ] ; then
        ROOTDIR=$1
else
        ROOTDIR=/$1
fi

if [ -d $ROOTDIR/inprogress ] ; then

DESTDIR=${ROOTDIR}/${DIRDATE}

if [ -d $DESTDIR ] ; then
        DIRDATE=`/bin/date +"%F"`
        DESTDIR=${ROOTDIR}/${DIRDATE}
fi

/bin/mv ${ROOTDIR}/inprogress ${DESTDIR}
/bin/rm ${ROOTDIR}/latest
/bin/ln -s ${DESTDIR} ${ROOTDIR}/latest


else
        echo No inprogress directory found at $ROOTDIR
fi

Deleting old Backups

The Following script deletes old backups in the form YYYY-MM-DD with the following rules:

• one week old, if not monday
• one month old, if not first monday of the month
• three months old

#!/bin/bash

if [$# -ne 1] ; then
     echo "Error: you need to provide the dir as a parameter."
     echo "Syntax: ./script.sh /path/to/dir/"
     exit 1
fi

DIR=$1

oneweekold=`date -d '-7 day' +"%F"`
if [[ $(date -d '-7 day' +%u) -ne 1 ]] ; then
    find $DIR -name $oneweekold -delete
    if [ $? -ne 0 ] ; then
        echo `date +"%F %T"` " : Failed to delete one week old backup: $DIR$oneweekold, " | mail -s "York - Tidy up script" supportedsitealerts@newro.co
    fi
fi

onemonthold=`date -d '-1 month' +"%F"`
day=`date -d $onemonthold +%d`
if ! [[ $(date -d '-1 month' +%u) -eq 1 && $day -le 7  ]] ; then
    find $DIR -name $onemonthold -delete
    if [ $? -ne 0 ] ; then
        echo `date +"%F %T"` " : Failed to delete one month old backup: $DIR$onemonthold, " | mail -s "York - Tidy up script" supportedsitealerts@newro.co
    fi
fi

threemonthsold=`date -d '-3 month' +"%F"`
find $DIR -name $threemonthsold -delete
if [ $? -ne 0 ] ; then
    echo `date +"%F %T"` " : Failed to delete three months old backup: $DIR$threemonthsold, " | mail -s "York - Tidy up script" supportedsitealerts@newro.co
fi

It also takes an argument with the directory in which to look and sends an email if delete failed