FreeNAS & online snapshot backups
Tonight I’ve spent some time playing a bit more with my supernas (FreeNAS). I have managed to learn how to make an automated snapshot style rotating backup system. The key is in the use of UNIX hard-links feature. Deriving from this info and this script (which was wrong by the way), I have built a script that automatically pulls data from my NAS onto my online storage (dreamhost) and rotating it every day (keeping 5 snapshots max).
#!/bin/sh
#
# Easy Automated Snapshot-Style Drive-snapshot using Rsync
# Copyright (C) 2008 Sylvain Rebaud (sylvain at rebaud dot com)
#
# Based on Dan Merschi & Mikes Handy
# Easy Automated Snapshot-Style Backups with Linux and Rsync
# http://www.mikerubel.org/computers/rsync_snapshots/
#
#-- Verify input arguments --
if [ $# -lt 2 ] ; then
echo >&2 "usage: $0 [source] [dest]"
exit 1
fi
#-- Path to backup directory --
BACKUP="$2/snapshot"
#-- Rotating-filesystem-snapshot --
rm -fR ${BACKUP}5
rm -f ${BACKUP}5.date
for i in 4 3 2 1
do
y=`expr $i + 1`
if [ -d ${BACKUP}${i} ]; then mv -f ${BACKUP}${i} ${BACKUP}${y} ; fi
if [ -d ${BACKUP}${i}.date ]; then mv -f ${BACKUP}${i}.date ${BACKUP}${y}.date ; fi
done
date > ${BACKUP}1.date
#-- Taking filesystem snapshot using rsync & hardlinks (backup1) --
#
rsync -aHvz --exclude=".*" --delete --rsh="ssh -c arcfour -o Compression=no -x" --link-dest=../snapshot2 "$1" ${BACKUP}1/
#
#-- End Script --
Then I set up freeNAS to execute a cron job using the following command:
/usr/bin/ssh soothe@plutinosoft.com "./backup/backup.sh root@sylvain.homeip.net:/mnt/bigboy/share/Photos/ backup/Photos"
This tells FreNAS to login to my site (plutinosoft.com) using a RSA key with no passphrase and execute the remote backup script which in turns connects back to my FreeNAS server (using Dynamic DNS host and a RSA key with no passphrase as well) and pull/backup/sync-up the data locally using a rotation of 5 folders.
Phew!
Finally, snapshot1 should be the latest full sync and snapshot2 disk space should be for the differences from snapshot1 (meaning only the files that were deleted later that are not present in snapshot1 anymore), and so on until snapshot5. So total amount of space used is really just a full snapshot plus the changes made and not 5x the full snapshot.
Neat, isnt’it?
Update: I have updated the script to not rely on cp -al which is not available on Freenas and instead use the rsync --link-dest feature recently implemented. This permits to use this script to backup data on a Freenas box as well. I also added a snapshot1.date file to keep track of when the last backup was actually made.
Jan 02, 2008
[…] you’re an advanced FreeNAS/FreeBSD user you may find these two post interesting: FreeNAS & online snapshot backups, with a script how to backup your data to an online hoster, e.g. Bluehost, or this post: Diy NAS […]
Feb 14, 2008
Cool! how about submitting that script as a patch to FreeNAS so that they can integrate it and add it to their webinterface for easy configuration :)
Feb 24, 2008
Nice script, but you can not run this script on FreeNAS just because FreeBSD ‘cp’ don’t support ‘-a’ option.
Anyway, I fix my script and now is running as expected.
Regards,
Dan
Feb 25, 2008
The script is not running on FreeNAS, it is running on the server which does the backup (pulling from FreeNAS) so it’s working fine.
Mar 26, 2008
Hi Dan, how did you get around not having the cp -al option?
Mar 26, 2008
There seems to be some confusion. The backup script which I showed here is called rsync_snapshots.sh and is on my online server (plutinosoft.com) which has cp -al option. It is executed remotely using ssh by freenas using a cron job. This script connects back to my home freenas (using dynip – sylvain.homeip.net) and rsync from it onto my online server so it’s pulling from my freenas but the cp -al is executed on my online server (where the backups are)…
Mar 27, 2008
I’m using rsync to backup from a development server (fedora), to the box with freenas installed, this then executes the rotation of each backup – needing to use cp -al. I haven’t yet worked out how to replicate this with the freebsd supplied with freenas. I found some reference to cpio, but this doesn’t work with freenas.
May 01, 2008
Hi Tim
If you take a look on my script I use rsync -link-dest option to hardlink dir1 on dir2, like that you don’t have to use ‘cp -al dir1 dir2’.
My answer is late, but sorry.
Regards,
Dan
Dec 09, 2008
Question: Why do you need to setup a cron on your Freenas to trigger the pull on your online server? Couldn’t that be achieved by setting up a cron on your online server to start pulling?
Dec 10, 2008
I don’t control my online server. It’s hosted (dreamhost).
Jan 26, 2010
fyi, you can SSH into your dreamhost server and setup a cron.