Tonight I’ve spent some time playing a bit more with my supernas (FreeNAS). I have managed to learn how to make an automated snapshot style rotating backup system. The key is in the use of UNIX hard-links feature. Deriving from this info and this script (which was wrong by the way), I have built a script that automatically pulls data from my NAS onto my online storage (dreamhost) and rotating it every day (keeping 5 snapshots max).
#!/bin/sh
#
# Easy Automated Snapshot-Style Drive-snapshot using Rsync
# Copyright (C) 2008 Sylvain Rebaud (sylvain at rebaud dot com)
#
# Based on Dan Merschi & Mikes Handy
# Easy Automated Snapshot-Style Backups with Linux and Rsync
# http://www.mikerubel.org/computers/rsync_snapshots/
#
#-- Verify input arguments --
if [ $# -lt 2 ] ; then
echo >&2 "usage: $0 [source] [dest]"
exit 1
fi
#-- Path to backup directory --
BACKUP="$2/snapshot"
#-- Rotating-filesystem-snapshot --
rm -fR ${BACKUP}5
rm -f ${BACKUP}5.date
for i in 4 3 2 1
do
y=`expr $i + 1`
if [ -d ${BACKUP}${i} ]; then mv -f ${BACKUP}${i} ${BACKUP}${y} ; fi
if [ -d ${BACKUP}${i}.date ]; then mv -f ${BACKUP}${i}.date ${BACKUP}${y}.date ; fi
done
date > ${BACKUP}1.date
#-- Taking filesystem snapshot using rsync & hardlinks (backup1) --
#
rsync -aHvz --exclude=".*" --delete --rsh="ssh -c arcfour -o Compression=no -x" --link-dest=../snapshot2 "$1" ${BACKUP}1/
#
#-- End Script --
Then I set up freeNAS to execute a cron job using the following command:
/usr/bin/ssh soothe@plutinosoft.com "./backup/backup.sh root@sylvain.homeip.net:/mnt/bigboy/share/Photos/ backup/Photos"
This tells FreNAS to login to my site (plutinosoft.com) using a RSA key with no passphrase and execute the remote backup script which in turns connects back to my FreeNAS server (using Dynamic DNS host and a RSA key with no passphrase as well) and pull/backup/sync-up the data locally using a rotation of 5 folders.
Phew!
Finally, snapshot1 should be the latest full sync and snapshot2 disk space should be for the differences from snapshot1 (meaning only the files that were deleted later that are not present in snapshot1 anymore), and so on until snapshot5. So total amount of space used is really just a full snapshot plus the changes made and not 5x the full snapshot.
Neat, isnt’it?
Update: I have updated the script to not rely on cp -al which is not available on Freenas and instead use the rsync --link-dest feature recently implemented. This permits to use this script to backup data on a Freenas box as well. I also added a snapshot1.date file to keep track of when the last backup was actually made.
Antec Sonata III super silent case on sale at Fry’s = $60
2 x 320 GB Maxtor HD on sale at Fry’s = $88
2 GB USB Key on sale at Fry’s = $5
Old PC left for dead = $0
FreeNAS software = $0
TwonkyMedia developer license = $0
A fully functional NAS streaming Music,Videos & Photos to my XBox360 = Priceless !
Yeah Yeah, I am a nerd so what else would I do on the last 3 days of my vacations while my geek-father is visiting from France? I had been thinking a lot about setting up a NAS for backup purposes and recently noticed that some NAS now have a built-in UPnP Media server (TwonkyVision most of the time). Since I knew Twonky could stream to XBox360, and that it was time to reinstall XP on my home machine, I figured I’d buy one of these nifty boxes with their RAID built-in…until I found out about their prices. Fogetaboutit!
After a few days of research, I found out about FreeNAS, a free software based solution with UPnP support (uShare). Since I had an old PC (my old 700 Mhz AMD Athlon linux server) in a corner of my home office, I’d figured I’d play with it. Of course, there was a reason it hadn’t been turned on for years. The power supply was as loud as my first car when pushed to its limits on highway 101 (which was about close to 55 mph) and the only HD in it had a whooping 4GB capacity. Not a good start for a silent backup NAS system…
The first step was to install the software on the box. One solution in order to save disk space was to make the NAS boot from a USB Key. After spending a day in vain trying to do that with a old USB Key I had, we gave up and instead simply booted up from a CD.
Then we tried to build a RAID system with old IDE hard drives I had kept around for years (meaning two 10GB Seagates, a 4GB Maxtor and a 40GB IBM). None but the 4GB worked in the end. We decided to go to Central Computer that day to buy a RAID controller and some SATA drives instead until we found out the cheapest SATA PCI RAID controller they had was $140. That was xMas eve. We then decided to drive to BestBuy in Colma to get a cheaper RAID controller and some HDs. By the time we got there, 10 min before they closed, we realized the entire store had been vandalized by angry shoppers looting for last minute gifts including internal SATA drives on sale! We ended up buying just a RAID controller.
Spent a bit of xmas day playing with the only functional HD (4GB) we had and managed to get FreeNAS working a bit with some promising results.
On Dec 26th, we drove to PA, the closest Fry’s to us. There we had 1h to get all we had planned to buy (including stuff for my greedy father feeling rich with his pockets full of Euros). That ended up being perfect since lots of stuff was on sale including PATA Maxtor drives. $44 for 320GB!! Can’t beat that. So I gave up on SATA drives and my idea of a RAID 1+0 system. As soon as we had paid, I realized I should have bought a new power supply as well but there was an even better deal going on, a super quiet case WITH power supply for just $60 after mail-in rebate, almost 1/2 the price of a power supply alone so I went for it and there I was at the counter paying for the second time.
Enough with shopping, came back home, installed everything in the new case, booted off the CD, configured it to save the config on the USB Key and there I was with 2 PATA drives of 320 GB each. Since I didn’t want to use Software RAID as it didn’t seem to be stable according to people on the freeNAS forums, I decided to use one drive as a backup using the “Local” RSYNCD feature. At 3am, a script kicks in and synchronizes (copies) all the data from my “bigboy” disk onto the “backup” disk so I have an exact copy. This is my poor man’s RAID 1 :-)
Additionnally, I have set up a cron job so that at 5am, it synchronizes the data with my online server hosted at dreamhost (where plutinosoft.com is). So everything is backed up on line as well. Now I can sleep at night knowing that all my dearest digital photos are safe.
But the other reason for all this was to be able to stream music, photos and videos to my HDTV. Since the fall update, the Xbox 360 supports AVI and DIVX playback via UPnP. This is killer. I started the UPnP sharing feature (uShare) to realize that indeed it worked with the xbox but all the music was not organized properly by artists/albums/tracks. Instead it listed the same thing (tracks) whichever selection you were in. After digging a bit, I find out that their UPnP server didn’t implement the UPnP feature (Search) that the Xbox wants.
Here comes TwonkyVision. I remembered during my UPnP work in XBMC that I had contacted the guys there to test the UPnP client with their server and they gave me a developer license. Also they announced their support for the Xbox recently so I decided to use it as the UPnP server. Unfortunately they don’t have a FreeNAS version so for now I have to use my XP Home machine to run it and points to Samba Shares (CIFS) I mounted off my NAS. Since Twonky needs quite a bit of CPU for transcoding, it sounds like a good solution anyway.
Well guess what. It works like a charm! I now have all my music well organized on the xbox 360 (except no album arts yet …), it plays avi/divx fine and I can see all my photos on my HD screen with a much higher resolution than XBMC could ever render.
Now if only I could make Rhapsody work on the xbox 360 … hmm I hear there’s a version of the Slimserver for freeNAS… hmmmmmmmmm
Saw this show a couple months ago and made me realize how saying someone is a “Pioneer” is a nice way of saying he’s not a “Visionary”. Heh.
Today my boss wanted me to unlock his virgin iPhone so I did. My iphone was starting to be a bit flaky so I decided to upgrade mine as well. Mine being already unlocked 1.0.2, it was a little trickier but I followed the instructions here and voila (check out the nice T-Mobile logo):
Then I had to get my own ringtones on it using this. I now have the must-have CTU ringtone.
Last week-end, we decided to go to Half-moon bay to the pumpkin patch so we drove down highway 1, stopped to get some food and there it was. My dream car the AC Cobra… One day, just one day, you wait…
I finally got to integrate mediabox with wordpress to play a movie. You’ll need quicktime to see her very elegant first walk. (That was 1 month ago already).
Of course now we have to run after her!
