Name: ~parafin/linux/backups
Summary: some scripts and suggestions for doing backups
Author: parafin
Created: Tue, 14 Dec 2010 19:37:45 GMT
Last update: Mon, 27 Dec 2010 17:03:17 GMT by parafin
Atom feed
Modify access level: moderator

My backup system at home

There are some basic rules about backups. First of all - don't keep all copies of your backups in the same building. I, personally, copy them to a server in Germany. Second - encrypt anything sensitive. I use sys-fs/encfs for this - it's quite handy tool. Third - make backups regularly. I usually update my system every week and make backups right after it.

Usual problem is how much space backups use. Some stuff is quite big. Big stuff can be divided into 2 categories - what needs to be backupped and what is not. Examples of what should be excluded from backup: swap file, suspend file, temporary directories and (something more Gentoo-specific) - distfiles and packages. In the other category we have for example some photos we took or videos we made. There isn't much point in having multiple backups of them from different dates, so they should be backupped separately from system files and home directories. Then there is a question of compression. Well, photos and videos don't get much smaller, so obviously don't compress them. System backups on the other hand can be made around 2 times smaller. This is quite useful for copying over Internet, but no so much in case of local backups. Moreover (you may not know this) recent versions of app-arch/tar finally learned how to use seek on archive files, which of course quite useful when you want to extract just part of it or list the content. But seek of course works only on uncompressed files. So there is a quite good reason not to use compression options with tar.

So we handled space question, now about speed. Remote servers are good for keeping your data safe and secure, but are not so good when it comes to speed. Even if you have excellent Internet connection at home, local data storage always will be faster. So I suggest keeping local copies. For that buy external hard drive or enclosure (case, mobile rack, whatever they call it) for usual one. ESATA is definitely the way to go, USB is quite slow really. If your laptop doesn't have eSata ports - not to worry, there are PCMCIA/ExpressCard eSATA controllers - buy one. For PC it's even simpler - eSATA drives work quite fine plugged into SATA ports, you'll just need special cable or adapter with bracket (my external enclosure came with one of these).

Dependencies

  • sys-fs/sshfs-fuse
  • sys-fs/encfs

You'll have to create needed directories and change some lines in following scripts, they are not ready for use as is. Scripts assume that there are 3 partitions: /, /home and /boot (which isn't mounted automatically). Also there is a mountpoint /mnt/backups for external drive, so you should add smth like that to /etc/fstab:

/etc/fstab
/dev/disk/by-uuid/ad97a4e5-10d6-42e3-984c-d4518da423a7 /mnt/backups ext3 noauto,users,noatime 0 0

Encfs will ask for password and settings on the first run for each encrypted directory. As for settings, I suggest default paranoia mode.

Code listings

make_backup
#!/bin/bash
ionice -c 3 -p $$
chrt -i -p 0 $$
if [[ -n $1 ]]
then
        if [[ $1 == all ]]
        then
                do_home=1
                do_root=1
        elif [[ $1 == root ]]
        then
                do_root=1
        elif [[ $1 == home ]]
        then
                do_home=1
        else
                echo "$0 [all|root|home]"
                exit
        fi
else
        do_home=1
        do_root=1
fi
date=$(date +%Y%m%d)
host=$(hostname)
mount /mnt/backups || exit 1
encfs /mnt/backups/.enc-backups /root/enc-backups || { umount /mnt/backups ; exit 1 ; }
[[ -n $do_root ]] && [ -f /root/enc-backups/$host-$date.tar ] && \
        { echo "File $host-$date.tar exists, delete it first!" ;\
        umount /root/enc-backups && sleep 1 && umount /mnt/backups ; exit 1 ; }
[[ -n $do_home ]] && [ -f /root/enc-backups/$host-home-$date.tar ] && \
        { echo "File $host-home-$date.tar exists, delete it first!" ;\
        umount /root/enc-backups && sleep 1 && umount /mnt/backups ; exit 1 ; }
[[ -n $do_root ]] && mount /boot && tar -cf /root/enc-backups/$host-$date.tar -X /root/excludes --one-file-system / /boot
[[ -n $do_root ]] && umount /boot
[[ -n $do_home ]] && tar -cf /root/enc-backups/$host-home-$date.tar -X /root/excludes_home --one-file-system /home
sleep 2
umount /root/enc-backups && sleep 1 && umount /mnt/backups
excludes
/swap_file
/suspend_file
/tmp
/usr/portage/packages
/usr/portage/distfiles
/var/tmp
excludes_home
/home/parafin/tmp
/home/parafin/junk/images/photos
rsync
#!/bin/bash
ionice -c 3 -p $$
chrt -i -p 0 $$
mount /mnt/backups || exit 1
rsync -av --delete /home/parafin/junk/images/photos/ /mnt/backups/photos/
sleep 1
umount /mnt/backups
rsync_remote
#!/bin/bash
ionice -c 3 -p $$
chrt -i -p 0 $$
sshfs -o idmap=user,workaround=rename,reconnect,ServerAliveInterval=30,ConnectTimeout=30 user@remote.server.name:/path/to/.photos-backup /home/parafin/tmp/remote-backups || exit 1
encfs /home/parafin/tmp/remote-backups /home/parafin/tmp/enc-backups || \
        { fusermount -u /home/parafin/tmp/remote-backups ; exit 2 ; }
rsync -rtv --delete /home/parafin/junk/images/photos/ /home/parafin/tmp/enc-backups/
sleep 1
fusermount -u /home/parafin/tmp/enc-photos
sleep 1
fusermount -u /home/parafin/tmp/remote-backups
cp_backups
#!/bin/bash
ionice -c 3 -p $$
chrt -i -p 0 $$
mount /mnt/backups || exit 1
sshfs -o reconnect,ServerAliveInterval=30,ConnectTimeout=30 user@remote.server.name:/path/to/.backups /home/parafin/tmp/remote-backups || \
        { umount /mnt/backups ; exit 2 ; }
encfs -o allow_root /home/parafin/tmp/remote-backups /home/parafin/tmp/enc-backups || \
        { umount /mnt/backups ; fusermount -u /home/parafin/tmp/remote-backups ; exit 3 ; }
su -c "
encfs /mnt/backups/.enc-backups /root/enc-backups || exit 1
find /root/enc-backups -mtime -4 -type f -exec /root/cp_compress \{} /home/parafin/tmp/enc-backups \;
umount /root/enc-backups
"
sleep 1
fusermount -u /home/parafin/tmp/enc-backups
sleep 1
fusermount -u /home/parafin/tmp/remote-backups
sleep 1
umount /mnt/backups
cp_compress
#!/bin/bash
bzip2 -cv9 "$1" > "$2"/"$(basename "$1")".bz2

Usage

Well, you run ./make_backup, ./rsync, ./rsync_remote and ./cp_backups. ./cp_backups have to go after ./make_backup, other scripts don't care about order in which they are run. If you have multiple machines, ./cp_backups should be run only on one of them obviously (assuming you use just one hard drive).