Backing up Docker
Alright, backups! Super, super important. It’s a critical part of a functioning docker environment, and a part that most people skip. Don’t skip it!
Now if we are going to back up our system, it needs to be on a remote target (no point backing up to the same drive). Which means we need a remote target of some description. Hmm… Hold on.
Shuffles around in the cable box
We have a… raspberry pi 3b and an old usb drive. Yeah that sounds pretty typical for a homelab backup.
In this section, we will:
- Set up a scheduled backup with btrbk and verify that it functions
- An external BTRFS based linux server (another fedora server works great!)
- This guide will use a Raspberry Pi 3b with an external btrfs formatted USB drive
Choosing a Backup Software
Backing up is where a common programming adage is especially relevant: Keep it Simple Stupid.
There’s all sorts of fancy ways to back up. You can do it incrementally, you can encrypt it, you can encode backups into cryptocurrency… oops went off track.
The best, and easiest backup, is the kind you can easily restore from. The simplest method is, well, a file copy. You copy the file, and now you have a copy of the file. Restoring the file involves just copying it back.
As it turns out you can achieve that simplicity, and have the bells and whistles: by leveraging the features of btrfs.
Why BTRFS is amazing
btrfs is a significant reason we chose to use Fedora. Fedora (workstation), Synology and Suse all default to using btrfs
btrfs based backups to Fedora and Suse should work, but btrfs backups to Synology will not (they use a custom build of btrfs)
Btrfs is a copy-on-write filing system designed to allow point-in-time snapshots of the filing system (or individual subvolumes, a term we used earlier in the guide). It’s genius in that you can instantly snapshot your data into a separate folder, and transfer that data to another btrfs filing system at a pure block level (IE: very efficiently).
With all this combined we can:
- Create point-in-time snapshots of our running docker containers without disturbing their operation
- Incrementally transfer these snapshots to an external btrfs storage, or over SSH to another btrfs based computer
- Incrementally transfer and store these snapshots in an efficient, block based, deduped manner
- Perform trivial restores because it’s all transparent at the user level: the snapshots and backups are just folders and files on a filesystem
Where BTRFS may not be appropriate
The big caveat with btrfs backups is.. well.. it has to be on btrfs. at source and destination. If you want to use a more generic storage target, you are better off using a chunk based backup technology, like restic. Restic will also work fine (as long as you back up btrfs snapshots and not the running folder structure), but it’s at the cost of a significant increase of complexity.
Setting up the Raspberry pi
We are going to start off with a raspberry pi running raspberry pi OS desktop edition
running the desktop edition will allow using graphical partition editor, which the server edition won’t do by default
Start by installing some pre-requisites:
Open gparted and unmount your USB drive
Format the drive to BTRFS after unmounting
If this isn’t obvious, overwriting your filing system will destroy ALL data on the USB drive. Make sure it’s not important data, and make SURE you chose the right drive. Do not do this step with ANY important data plugged into the computer. you have been WARNED.
Also label the drive something descriptive (like backup). Press apply (the green check box) to wipe your drive.
Select the device in the filing system, it should mount at
Unplug and plug your USB drive back in to make sure it shows up and mounts automatically.
Now we have two cockpit interfaces: one for our docker host and one for our backup pi. For us to successfully transfer backups over SSH without requiring a password, we need to perform a key exchange. I have written a guide on doing a key exchange here. For convenience, here is the process on an any% speedrun:
We’re done with the raspberry pi for now. Back at your docker host in cockpit, install btrbk:
btrbk is a backup utility to leverages the btrfs filing system to perform snapshots and replication
(/doing-more-with-docker/08-Backing-up-Docker-07.png)For btrbk, we set up a simple configuration file (at
etc/btrbk/btrbk.conf) that describes source, destination, and retention policies. We also create a btrfs subvolume at
/mnt/container_snapshots for local snapshots. run the following, changing your
target to use your destination ip address:
#create a location to house our snapshots btrfs subvolume create /mnt/container_snapshots #place a preformatted config file at /etc/btrbk/btrbk.conf cat > /etc/btrbk/btrbk.conf <<EOF transaction_log /var/log/btrbk.log stream_buffer 512m snapshot_preserve 24h 7d 4w 3m snapshot_preserve_min latest target_preserve 24h 7d 4w 3m target_preserve_min latest volume /mnt subvolume containers snapshot_create ondemand snapshot_dir container_snapshots ssh_identity /root/.ssh/id_rsa target ssh://10.20.10.230/media/pi/backup EOF
You can find an example config here with many more configuration options.
Alright, let’s give it a test! run the following:
easy! We now have a point-in-time snapshot in
/mnt/container_snapshots, and that same snapshot has been replicated to the raspberry pi on the USB drive.
by default, we will be retaining 24 hourly snapshots, 7 daily snapshots, 4 weekly snapshots, and 3 monthly snapshots. This should reasonably cover you for 3 months of rotating backups.
btrbk run again, you’ll get the same result: a second timestamped folder with all of your stuff in it. However, behind the scenes
btrbk will deduplicate the contents at the filesystem level. Which means that both folders are a full backup of your data: however they are transferred and stored in an incremental fashion behind the scenes. Awesome!!!
Setting a Schedule
Last step remaining: let’s set up an hourly backup.
- in cockpit, navigate to the Services category, then to Timers. Create timer.
btrbk already has an (inactive) timer here, but we’ll set up our own
- Create a time called
btrbk-docker, set the command to
/usr/sbin/btrbk run, and set the timer to
And, that’s it! You’re done. You can also find the
btrbk-docker.service in the service tab to run manually and test.
The logs tab in cockpit should warn you if there was a failure in the backup. You can also set up alerting using normal systemd alerting functions.
Before you Forget
Document your backup settings!
So far, we are being pretty cavalier with our use of
root accounts and default admin accounts. Let's fix that with Central Authentication and SSO.