Skip to content

Ransomware Resistant Backups with btrbk

Introduction

In the previous article we had set up a very resilient and efficient backup to an external hard-drive. Fantastic! However, as self hosted connoisseurs, we are also typically exposing some level of attack surface out to the public. Could be immich, could be this blog, could be your printer (if you have accidentally let UPnP stay enabled on your router).

Regardless of the risk, it's no longer "good enough" to let your backup go to an external drive and call it a day. An attacker can just as easily ransomware your external drive as they can ransomware your server if they manage to get past the gates. How can we make our backups not just robust, but also protected from malicious intrusion?

TL;DR

In this guide, we will:

  • Set up a separate linux instance and allow root ssh access to the primary server
  • Set up pull backups using btrbk and ssh

Prerequisites

  • A primary docker instance that you wish to back up on a btrfs subvolume (see Live Backups with btrbk)
  • A secondary linux computer (could be a raspberry pi!) with btrfs storage for backups

This guide will also make the assumption we are continuing from the previous article, however it can still be applied without following the initial steps.

Ransomware Protection Theory

So what is ransomware protection? The concept is that your backups must land in a location that, should your primary server (or corporate network segment in enterprise land) get compromised, that the attacker does not have a path of access to encrypt/delete the backups as well.

Offline Backups

The oldest (and to some extent, simplest) way to ensure this are with offline backups. If we were pushing backups to an external drive, well, maybe we just disconnect the external drive when we're done! That is a valid, albeit extremely manual method of protection.

The primary issue with this method is fatigue. Anything that's manual causes people to become complacent over time, and people will inevitably stop verifying or checking that the offline backups are doing their job. Offline backups can also be automated (though not with a homelab setup). This is the primary use case for tape backups, where enterprises can instruct a robot to physically rotate high density tapes that backups get written to.

Immutable Backups

The next option (and a very valid and currently used option) are immutable backups. This is where you entrust a third party to force append only backups, so even if you choose that you want to delete a backup, you can't! There requires an out of band or break glass method to erase any data once it's got written. This is a common method of protection employed by SaaS backup/remote operators. The biggest downside is that the backup software you use must be aware of this method of backup, otherwise the software will try to clean up and start failing almost immediately.

Chain of Custody Backups

This option is similar to the immutable backups, but unlike an immutable backup, does not require the backup software to be aware of the retention rules. In this situation you are still backing up to a third party, but the remote target also performs its own versioning of the backup independently. That way if you are (say) backing up to a bucket, and tomorrow an attacker encrypts that whole target, you have an independent way to get back in and restore a previous version.

The biggest downside to this option is that… well the option has to exist! Not all remote providers have this versioning or restore option. Some bucket providers can version files within the bucket, but that is still visible to the end user and not effective at preventing malicious activity.

Pull Backups

The final (and the method we will use) option are pull backups. This is where a system (with an independent set of credentials, access, or both) will reach in and pluck out the data required to be backed up (or even backing up the backup server for additional protection). This is beneficial in that it is very straightforward to setup and also extremely effective, as compromising the primary systems give no access to the backup systems.

The primary downside to this backup method is that the system pulling the backups must be separated (at minimum with credentials, but preferably by network as well) from the primary set of systems. If your ransomware resistant backup server is domain joined, and your primary server is domain joined, you just defeated the purpose of the whole system!

There is also the expectation that the backup server is isolated from the internet, corporate network (except for that pull access), or both. If an attacker manages to breach the backup server, they now have access to the primary server as a result! Therefore there should be as little exposure of the backup server as possible.

Setting up the Pull Backup

This time around, we have two devices: The one with our container data, and the server that will pull that data. Both have btrbk installed, and have btrfs volumes at /mnt/containers (source) and /mnt/backup (target) respectively:

Showing source and target dashboards

Showing source and target dashboards

First step is to have the target be able to perform root actions on the source. We do this by generating an ssh key on the target and adding it to the authorised keys of the root in the source.

Warning

This step is essentially giving the backup computer root privileges on the source computer. This is why pull backups require high levels of security for the backup machine

Root access isn't always required for this backup method, but as we are doing block level transactions, root access is required for btrbk to operate

Once that's completed, now we can swap to visual studio code to see what files/folders we're backing up.

You will notice that we still have our (operational!) backup going to the external drive from the previous article. We can stop that one (by turning off the systemd timer), but we don't necessarily have to: both can work in parallel. Instead, let's create a new snapshot folder called @snapshots-remote on the source.

btrfs subvolume create /mnt/containers/@snapshots-remote

Configuring btrbk

Now on the target, set up the btrbk config file, changing the following as required:

For the Source information:

  • volume is the btrfs volume path (accessed from the target via ssh)
  • subvolume is the btrfs subvolume being backed up (relative to the volume)
  • snapshot_dir is where the btrfs snapshots should get created (relative to the volume)

For the Target information:

  • target is the fully qualified path (volume + subvolume) of the subvolume where the snapshots will get backed up
cat > /etc/btrbk/btrbk.conf << '_EOF_'
# Specify SSH private key for remote connections
ssh_identity               /root/.ssh/id_ed25519
ssh_user                   root

# Enable stream buffer. Adding a buffer between the sending and
# receiving side is generally a good idea.
# NOTE: If enabled, make sure to install the "mbuffer" package!
stream_buffer              256m

snapshot_dir               mnt/containers/@snapshots-remote
snapshot_create            onchange

snapshot_preserve          8h 7d 0w 1m 1y
snapshot_preserve_min      latest
target_preserve            8h 7d 0w 1m 1y
target_preserve_min        latest

volume ssh://cblab01.gurucomputing.lan/
  target /mnt/backup
  subvolume mnt/containers
_EOF_

Running btrbk

Alright, let's give it a go! add the configuration on the target (either by creating it via vscode or running the above shell script) and run with btrbk run:

If all goes well, btrbk will remote into the first computer, generate a snapshot (found in this case under /mnt/containers/@snapshots-remote and efficiently transfer it over to the backup machine. Awesome!

Finishing up

A backup that can only be ran manually isn't great, so like our previous article, let's generate a schedule to run it hourly.

  • Create the systemd service
cat > /etc/systemd/system/btrbk-hourly.service << '_EOF_'
[Unit]
Description=btrbk-hourly

[Service]
Type=oneshot
ExecStart=/bin/btrbk run
WorkingDirectory=/root
_EOF_

cat > /etc/systemd/system/btrbk-hourly.timer << '_EOF_'
[Unit]
Description=btrbk-hourly

[Timer]
OnCalendar=hourly
Persistent=true

[Install]
WantedBy=timers.target
_EOF_
  • Test the same backup and check the logs
systemctl daemon-reload
systemctl start btrbk-hourly
journalctl -u btrbk-hourly

Moving On

Alright, now we have configured btrbk to push to an external drive, or pull from one server to an independently accessed server. What we don't have yet is an offsite target.

Info

The above backup system can also work just fine remotely, if we so choose.

The next (upcoming) article will discuss just that: How do we set up our backups to go remote, or for that matter, be encrypted and go remote to potentially untrusted destinations?