In an earlier post I laid out why I wanted yet another layer of backups for my important documents and I provided the script without much explanation. As an exercise for myself, I decided to post a step by step explanation for why I wrote the script the way I did. This will be educational for me, and hopefully interesting for you.
Alright, lets start at the beginning.
#! /bin/bash
#
set -ue
I am highly influenced by David Pashley’s superb article “Writing Robust Bash Shell Scripts.” Pashley is not only a great programmer, he is a superb writer who can really explain his reasoning in a way that I find appealing.
Since I want this script to be robust, adding “set -ue “on the first line is a must. The “u” breaks the script if a variable is unset. This is critical as human error can easily turn a script into a vicious beast that will consume your data. The “e” breaks the script on any error. This ensures that everything behaves as expected. Unless there is a good reason not to, always “set -ue”
For reference here is the explain shell for the command.
Here is the first function:
function build_vars
{
# Current Date / Time
epoch=$(date +%s)
month=$(date +%m)
year=$(date +%Y)
day=$(date +%d)
# Building Environment: Directories
configdir="$config_dir"
yeardir="$backup_dir"/"$year"
monthdir="$yeardir"/"$month"
daydir="$monthdir"/"$day"
backupdir="$daydir"
# Building Environment: File Names
backupfile="$backupdir"/"$prefix"."$epoch".tar.bz2
logfile="$configdir"/"$prefix".log
lastepochfile="$configdir"/"$prefix".last
}
Basically this function sets up all of the variables which will be used in the rest of the script. Putting them in a single function improves readability and makes maintenance easy.
Here we also see the first crumbs of how the backup system works. By finding the current Unix epoch, the script discovers the current time in a non-timezone dependent way. It can then use this as reference point and as a means of ultimately building unique file names.
In addition, this function builds all of the directory names that will be used throughout the remained of the script.
I could probably make this more efficient by only running “date” once, but all of the target systems are multi-Ghz so I can endure a couple of wasted cycles.
Here is the second function:
function check_epoch
{
#Checking Time Since Last Run
if [ -n "$(find "$lastepochfile" -mtime -1)" ]; then
echo "Less than 24 hours since last archive"
exit 0
fi
}
This function uses “find” to check if the file “$lastepochfile” is older than 1 day. As “$lastepochfile” is the last thing created in the script this effectively checks to see if the script successfully completed in the last 24 hours. Since I ended up setting a cron job that runs once an hour, this ensures that even if a system is only powered on for short times during the day the job will still get done but not OVER done which would waste cycles and file space.
For reference here is the explain shell for the find command used.
Here is the third function:
function check_env
{
#Checking Configuration Directory
if [ ! -d "$configdir" ]; then
mkdir "$configdir"
echo "$epoch" Making "$configdir" >> "$logfile"
fi
#Checking Year Directory
if [ ! -d "$yeardir" ]; then
mkdir "$yeardir"
echo "$epoch" Making "$yeardir" >> "$logfile"
fi
#Checking Month Directory
if [ ! -d "$monthdir" ]; then
mkdir "$monthdir"
echo "$epoch" Making "$monthdir" >> "$logfile"
fi
#Checking Day Directory
if [ ! -d "$daydir" ]; then
mkdir "$daydir"
echo "$epoch" Making "$daydir" >> "$logfile"
fi
}
While this covers a lot of lines, it is really doing something very simple, it checks to see if the year / month / day directory structure exists for the backup snap-shot that will ultimately be produced. If it does not exist then it makes it.
Effectively what this means is that each snapshot will be in its own directory which will be easy to browse to and find.
Here is the fourth function:
function write_file
{
#Incremental Tar & Compression
echo "$epoch" Writing "$backupfile" >> "$logfile"
find "$target_dir" -type f ! -name ".*" -newer "$lastepochfile" -print0 | \
tar czvf "$backupfile" --null -T -
#Log Success
echo "$epoch" Success "$backupfile" >> "$logfile"
}
This is the payload of the entire script. Everything else has just been leading up to this. First “find” is used to search “$target_dir” for all files that are newer than “$lastepochfile” and do not match the pattern .* . The reasoning for this should be pretty obvious, but I should not that I added the exclusion (by using the !) so that I wouldn’t needlessly copy lockfiles etc.
To ensure proper streaming of the files to tar I used the -printO.
It is important to note that this only searches out changed files and copies only their directory structure. My first approach copied the entire directory structure resulting in mostly empty directories… this resulted in overly large files as all those empty directories eat up space.
The tar line is pretty standard, note I use –null to ensure that it properly deals with null-delimited output and -T sets tar up to receive the filenames.
For reference here is the explain shell for the find and tar commands used.
Success is then logged.
Here is the fifth function
function write_epoch
{
# Unprotect Last Epoch File
chmod 600 "$lastepochfile"
# Write Last Epoch File
echo "$epoch" > "$lastepochfile"
# Protect Last Epoch File
chmod 400 "$lastepochfile"
}
In a way this is the most important part. First it unprotects “$lastepochfile” then it writes the epoch to it and finally the script protects “$lastepochfile.” As the entire system depends on the accuracy of “$lastepochfile” the protection should prevent casual deletion. Worst thing comes to worst, the epoch of the last run is saved in the log file and a new “$lastepochfile” could be created with the date.
Writing the epoch to the file isn’t necessary, but I think it provides a nice backup record and isn’t really that different than simply touching the file.
There is, likely, a way of doing this that does not require an external file (instead referencing the last created archive) but I like that this approach is easy to debug and trivial to understand.
Finally the actual program
# Configuration
prefix=important
target_dir=/home/user/Important
backup_dir=/home/user/ImportantArchive
config_dir=/home/user/ImportantArchive/.kb
#Program
build_vars
check_epoch
check_env
write_file
write_epoch
exit 0
As you can see, I’m really just using functions as a way of organizing the script. I prefer to think in terms of modules which I can replace as needed and this provides that. Also I like that “forcing” an update would be as simple as commenting out “check_epoch”.
So there you have it, it is no masterwork of programming but this script has now been comfortably running on two different systems automatically using CRON for 4 months without a single glitch. I can’t really imagine a better testament to its stability than that!
So what could I improve about it? I’m no expert at scripting so I’d love any feedback that anyone has.
Linux & Liberal Arts 