Backing Up Docker

Introduction

Alright, backups! Super, super important. It’s a critical part of a functioning docker environment, and a part that most people skip. Don’t skip it!

Now if we are going to back up our system, it needs to be on a remote target (no point backing up to the same drive). Which means we need a remote target of some description. Hmm… Hold on.

Shuffles around in the cable box

We have a… raspberry pi 3b and an old usb drive. Yeah that sounds pretty typical for a homelab backup.

TL;DR

In this section, we will:

  • Set up a scheduled backup with btrbk and verify that it functions

Pre-Requisites

  • An external BTRFS based linux server (another fedora server works great!)
    • This guide will use a Raspberry Pi 3b with an external btrfs formatted USB drive

Choosing a Backup Software

Backing up is where a common programming adage is especially relevant: Keep it Simple Stupid.

There’s all sorts of fancy ways to back up. You can do it incrementally, you can encrypt it, you can encode backups into cryptocurrency… oops went off track.

The best, and easiest backup, is the kind you can easily restore from. The simplest method is, well, a file copy. You copy the file, and now you have a copy of the file. Restoring the file involves just copying it back.

As it turns out you can achieve that simplicity, and have the bells and whistles: by leveraging the features of btrfs.

Why BTRFS is amazing

btrfs is a significant reason we chose to use Fedora. Fedora (workstation), Synology and Suse all default to using btrfs

btrfs based backups to Fedora and Suse should work, but btrfs backups to Synology will not (they use a custom build of btrfs)

Btrfs is a copy-on-write filing system designed to allow point-in-time snapshots of the filing system (or individual subvolumes, a term we used earlier in the guide). It’s genius in that you can instantly snapshot your data into a separate folder, and transfer that data to another btrfs filing system at a pure block level (IE: very efficiently).

With all this combined we can:

  • Create point-in-time snapshots of our running docker containers without disturbing their operation
  • Incrementally transfer these snapshots to an external btrfs storage, or over SSH to another btrfs based computer
  • Incrementally transfer and store these snapshots in an efficient, block based, deduped manner
  • Perform trivial restores because it’s all transparent at the user level: the snapshots and backups are just folders and files on a filesystem

Where BTRFS may not be appropriate

The big caveat with btrfs backups is.. well.. it has to be on btrfs. at source and destination. If you want to use a more generic storage target, you are better off using a chunk based backup technology, like restic. Restic will also work fine (as long as you back up btrfs snapshots and not the running folder structure), but it’s at the cost of a significant increase of complexity.

Setting up the Raspberry pi

We are going to start off with a raspberry pi running raspberry pi OS desktop edition

running the desktop edition will allow using graphical partition editor, which the server edition won’t do by default

Start by installing some pre-requisites:

sudo apt install gparted cockpit btrfs-progs usbmount mbuffer

Open gparted and unmount your USB drive

Format the drive to BTRFS after unmounting

If this isn’t obvious, overwriting your filing system will destroy ALL data on the USB drive. Make sure it’s not important data, and make SURE you chose the right drive. Do not do this step with ANY important data plugged into the computer. you have been WARNED.

Also label the drive something descriptive (like backup). Press apply (the green check box) to wipe your drive.

Select the device in the filing system, it should mount at /media/pi/<label>

Unplug and plug your USB drive back in to make sure it shows up and mounts automatically.

Key Exchange

Now we have two cockpit interfaces: one for our docker host and one for our backup pi. For us to successfully transfer backups over SSH without requiring a password, we need to perform a key exchange. I have written a guide on doing a key exchange here. For convenience, here is the process on an any% speedrun:

Using btrbk

We’re done with the raspberry pi for now. Back at your docker host in cockpit, install btrbk:

btrbk is a backup utility to leverages the btrfs filing system to perform snapshots and replication

dnf install -y btrbk

For btrbk, we set up a simple configuration file (at etc/btrbk/btrbk.conf) that describes source, destination, and retention policies. We also create a btrfs subvolume at /mnt/container_snapshots for local snapshots. run the following, changing your target to use your destination ip address:

#create a location to house our snapshots
btrfs subvolume create /mnt/container_snapshots

#place a preformatted config file at /etc/btrbk/btrbk.conf
cat > /etc/btrbk/btrbk.conf <<EOF
transaction_log      /var/log/btrbk.log
stream_buffer        512m
snapshot_preserve    24h 7d 4w 3m
target_preserve      24h 7d 4w 3m
volume               /mnt
subvolume            containers
snapshot_create      ondemand
snapshot_dir         container_snapshots
ssh_identity         /root/.ssh/id_rsa
target               ssh://10.20.10.230/media/pi/backup
EOF

You can find an example config here with many more configuration options.

Alright, let’s give it a test! run the following:

btrbk run

easy! We now have a point-in-time snapshot in /mnt/container_snapshots, and that same snapshot has been replicated to the raspberry pi on the USB drive.

by default, we will be retaining 24 hourly snapshots, 7 daily snapshots, 4 weekly snapshots, and 3 monthly snapshots. This should reasonably cover you for 3 months of rotating backups.

If you btrbk run again, you’ll get the same result: a second timestamped folder with all of your stuff in it. However, behind the scenes btrbk will deduplicate the contents at the filesystem level. Which means that both folders are a full backup of your data: however they are transferred and stored in an incremental fashion behind the scenes. Awesome!!!

Setting a Schedule

Last step remaining: let’s set up an hourly backup.

  • in cockpit, navigate to the Services category, then to Timers. Create timer.

btrbk already has an (inactive) timer here, but we’ll set up our own

  • Create a time called btrbk-docker, set the command to /usr/sbin/btrbk run, and set the timer to hourly. Save!

And, that’s it! You’re done. You can also find the btrbk-docker.service in the service tab to run manually and test.

btrbk rocks.

The logs tab in cockpit should warn you if there was a failure in the backup. You can also set up alerting using normal systemd alerting functions.

Before you Forget

Document your backup settings!

Moving On

So far, we are being pretty cavalier with our use of root accounts and default admin accounts. Let’s fix that with Central Authentication and SSO.