mythago - Using Subversion for Backup

Using Subversion for Backup

I use Subversion to backup my Linux boxes. With some minor creativity, it easily covers:

Daily snapshots and offsite backup.
Easy addition and removal of files and folders.
Detailed tracking of file versions.

It also allows for a few bonus features:

Regular log emails to keep track of filesystem activity via Subversion's event hooks.
Users may request a checkout of their home folders from any respository revision.
New or replacement servers can be setup with a few svn checkout commands.

And like everything, a few cons:

Depending on file types, disk usage can be heavy.
One must keep careful track of .svn folders in certain places.

Create the Repositories

All my repositories live in /svn/reposname which seems as good a place as any. As root, create a respository for each location you wish to back up. The following are proabably the most common locations, and a good place to start.

root@host:/svn$ svnadmin create etc
root@host:/svn$ svnadmin create home
root@host:/svn$ svnadmin create var

System Settings - /etc

First checkout the empty /svn/etc repository into /etc - Note the trailing . (period)!

root@host:/etc$ svn checkout file:///svn/etc .

Add the sub-folders and files you wish to backup. Including the entire /etc tree may be impractical, or an overkill, or exactly what you need - it is entirely up to you. If in doubt, start with the configuration folders for your services, and any files you have customized.

root@host:/etc$ svn add hosts.allow
root@host:/etc$ svn add hosts.deny
root@host:/etc$ svn add exim/
root@host:/etc$ svn add apache/
root@host:/etc$ svn add proftpd/
root@host:/etc$ svn add rcS.d/

Once satisfied with your selections, commit everything to the repository, replacing "initial commit" with a relevant log message.

root@host:/etc$ svn commit -m "initial commit"

Since we are not nessasarily backing up the entire /etc tree, this is as much as we can do. In the future new files and folders must be manually added to the respository on a case by case basis. This shouldn't be too dificult; for a desktop machine where regular application installs and updates may change /etc, review the contents of /etc once a month; for a server which is more likely to remain static in a production environment, you might review the contents of /etc less often.

User Data - /home

Exactly what gets backed up from users' home folders is a matter for the system administrator to decide. It doesn't really hurt keeping large files in Subversion, particularly if they are static. However, to save space and time on a home machine, it can be useful to supply each user with a sub-folder for backup purposes, allowing them to place relevant files there at their own discretion.

Another consideration is that the average user may not be familiar with Subversion, and may not know to treat hidden .svn folders with care. Or, it may not be safe to have hidden .svn folders in places where sensitive information could be leaked to the outside world, such as inside website folders like ~/public_html. In either case, it is best to checkout the repository elsewhere, and periodically import files from users' home folders with a crontab driven script, thus removing .svn folders from danger altogether.

root@host:/svn$ mkdir home-backup
root@host:/svn$ svn checkout file:///svn/home home-backup/

For the reasons listed above, the process for backing up home folders must be a little smarter than for /etc. Also, since home folders constantly change, you certainly don't want to be constantly adding and removing files to the repository. Build a script in your favourite shell language to construct and maintain a folder per user per service under /svn/home-backup.

As an example: Import each user's ~/public_html folder into their respective backup sub-folder, ie /svn/home-backup/websites/username, automatically handling adding and removing of new and deleted files to and from the repository.

root@host:/svn$ mkdir home-backup/websites
root@host:/svn$ svn add home-backup/websites

#!/usr/local/bin/raven 
 
# /svn/generate-home 
 
# list of users we want to backup 
[ 'user001' 'user002' ] list as $users 
 
# loop over each user 
$users each as $user 
 
	# user's backup directory 
	$user '/svn/home-backup/websites/%s' format as $bdir 
 
	# user's website directory 
	$user '/home/%s/public_html'         format as $udir 
 
	# auto-create backup dir if needed 
	$bdir exists not if $bdir mkdir 
 
	# import a current copy of the user's directory over top of the 
	# existing backup. 
	$bdir $udir 'cp -rf %s/*  %s/' format shell as $rs 
 
	# build a list of files in the user's backup directory, and check 
	# whether they still exist in the current directory. If a file no 
	# longer exists, remove it from the repository. Of course this 
	# does not remove old versions of the file from previous 
	# repository revisions, only future ones. 
	$bdir 'find %s | grep -v .svn' format shell trim LF split each as $file 
 
		# construct the file path relative to the backup dir 
		$file $bdir length over length extract as $relfile 
 
		# check for the file's existence relative to the user's dir 
		$relfile $udir '%s/%s' format exists not 
		if	$file 'svn del %s' format shell drop 
 
# build a list of imported files that have not been previously added 
# to the repository. These files will be newly created, so add them. 
"svn status /svn/home-backup | grep ? | awk '{print $2}'" 
shell trim LF split each as $file 
	$file 'svn add %s' format shell drop

System Data - /var

What you backup from /var will depend on what services your machine is running. Also backing up /var may have special considerations:

/var can change rapidly, and a raw snapshot of data may be useless for a recovery.
/var can be large, and have a lot of redundant data, ie rotated log files.
Services operating on files in /var may lock or restrict access to those files.

All this suggests /var needs an external snapshot setup similar to /home, yet with an even smarter process script that can connect to and export service data where required. I'll give an example of backing up MySQL.

root@host:/svn$ mkdir var-backup
root@host:/svn$ svn checkout file:///svn/var var-backup/
root@host:/svn$ mkdir var-backup/mysql
root@host:/svn$ svn add var-backup/mysql

Dump each MySQL database as a plain text .sql file.

#!/usr/local/bin/raven 
 
# /svn/generate-var 
 
# connect to mysql as "backup" user, which should be configured with only 
# SELECT, SHOW DATABASES, and LOCK TABLES privileges. 
'mysql://backup:@localhost/mysql' open as $mysql 
 
# list all databases. 
'SHOW DATABASES' $mysql query as $rs 
 
# loop over each database dumping it to /svn/var-backup/mysql/<database>.sql 
$rs records each 'Database' get as $database 
	$database dup 'mysqldump -Q -u backup %s >/svn/var-backup/%s.sql' format shell drop 
 
# build a list of sql files that have not been previously added to the 
# repository. These databases will be newly created, so add them. 
"svn status /svn/var-backup | grep ? | awk '{print $2}'" 
shell trim LF split each trim 
	'svn add %s' format shell drop

Nightly Backup

Call the various scripts, commit everything, and either dump the Subversion repository images or export the newest revision for external backup.

#!/bin/bash

# /svn/generate-backup

# update the snapshots
/svn/generate-home
/svn/generate-var

# commit everything
svn commit -m "relevant message" /etc
svn commit -m "relevant message" /svn/home-backup
svn commit -m "relevant message" /svn/var-backup

# dump the repositories as svn images. these should be
# shunted off to mass storage or offsite backup mediums.
svnadmin dump /svn/etc  > /tmp/svn-etc.dump
svnadmin dump /svn/home > /tmp/svn-home.dump
svnadmin dump /svn/var  > /tmp/svn-var.dump

Run the whole sequence nightly via crontab.

0 1 * * * /svn/generate-backup >>/var/log/backup.log

Email Updates

Configure Subversion to send daily email updates from all repositories whenever data changes. Subversion usually comes with a Perl script for just this purpose - see contents of /svn/reposname/hooks and /usr/lib/subversion/hook-scripts/ (Debian) for more info on sending plain text emails.

However, I like to make things just a bit more friendly by using a custom script to send HTML emails that:

colour added file names and lines blue.
colour deleted file names and lines red.
separate files with horizontal rules or place them in <div> with CSS for readability.

One issue you may encounter using hook emails is that for backup purposes, you often only want a list of affected files rather than a detailed dump of exactly what changed in each file. Backing up large active locations like /home can quickly lead to huge emails. In this case, you'll want to modify the post-commit script to fit your own requirements.