Now that I've started to migrate my stuff to Docker, I just wanted to share how I've configured the backup solution for them. I've made the move from Duplicity, which was causing some of my backups to become corrupt. Instead, I'm now using Kopia, which not only handles many of the settings I want for my backups in the backend, but also serves a nice graphical frontend.

Even though this screenshot make it seem like each snapshot is over 100GB each, Kopia actually handles deduplication between snapshots, meaning if a file hash matches from any previous snapshot, it will create links instead of making copies which significally reduces the backups in size.

Kopia also indexes each file it backs up, so that you can restore single files instantanously, unlike Duplicity where you had to first download the manifest before you could even list the contents of the backup before performing a restore. Here, I'll just go to the snapshot in time from where I want to restore, browse to the file and just download it. It's as simple as that!

As Kopia is rather easy to setup, I won't bore you with a tutorial, and let this blog entry focus more on the Docker side of the setup.

Kopia Docker container

First I installed Kopia as a docker container alongside my other containers. This was as simple as setting up a docker compose file. Here's my current file:

services:
    kopia:
        image: kopia/kopia:latest
        hostname: fukusha
        container_name: fukusha
        restart: unless-stopped
        ports:
            - 51515:51515
        command:
            - server
            - start
            - --disable-csrf-token-checks
            - --insecure
            - --address=0.0.0.0:51515
            - --server-username=${SERVER_USERNAME}
            - --server-password=${SERVER_PASSWORD}
        environment:
            USER: ${REPO_USER}
            KOPIA_PASSWORD: ${REPO_PASSWORD}
        volumes:
            - ./config:/app/config
            - ./cache:/app/cache
            - ./logs:/app/logs
            - ./data:/data:ro
            - ./repository:/repository
            - ./tmp:/tmp:shared
            - ./rclone/rclone.conf:/app/rclone/rclone.conf:ro

This setup uses an .env file for storing passwords, so be sure to set one up with the appropriate variables.

WARNING: SIDE NOTE
I named the host fukusha (複写), which translates to "duplication" or "copy" in japanese. I like to give my hostnames rememberable
names which also represents their functions. For example, my apache server's hostname is geronimo, and my Mastodon is named manny (even though the Ice Age character is actually a mammoth, but tomato/tomato.)

For my off-site storage, I'm using a simple 2TB Dropbox account, which can be uploaded to using rclone, included in the Kopia image. Since I previously also used rclone for my old backup I simply copied over my configuration and added it as a volume bind in the docker compose file. If you want to configure this yourself you can follow the guide in the official rclone documentation like I did.

Starting the container with docker compose up -d should get you up and running and you can log onto the GUI using the SERVER_USERNAME account you configured earlier in the .env file.

Helper scripts

In most cases, we want to do some actions before taking the backup, for example dumping a database or shutting down a container. For this, I've created a helper script which can be run inside the Kopia container. The helper script is used to start or stop containers or execute commands inside of them. The helper script is then used as source for other service specific backup scripts, which are the ones executed by Kopia before starting the backup process. For it to function properly, we need to give access to it in the Kopia docker compose file, as well as to the host's Docker socket (so that we can communicate with the Docker API):

        volumes:
            - ./config:/app/config
            - ./cache:/app/cache
            - ./logs:/app/logs
            - ./data:/data:ro
            - ./repository:/repository
            - ./tmp:/tmp:shared
            - ./rclone/rclone.conf:/app/rclone/rclone.conf:ro
            
            # Docker socket and helper script
            - /var/run/docker.sock:/var/run/docker.sock:ro
            - ./backup/docker-helper.sh:/app/backup/docker-helper.sh:ro

I will use Mastodon as an example. We will have to add some access to the files from the Mastodon container we want to backup:

        volumes:
            - ./config:/app/config
            - ./cache:/app/cache
            - ./logs:/app/logs
            - ./data:/data:ro
            - ./repository:/repository
            - ./tmp:/tmp:shared
            - ./rclone/rclone.conf:/app/rclone/rclone.conf:ro
            
            # Docker socket and helper script
            - /var/run/docker.sock:/var/run/docker.sock:ro
            - ./backup/docker-helper.sh:/app/backup/docker-helper.sh:ro
        
        #-- Backup endpoints --#

            # Mastodon
            - ../mastodon/backup:/app/backup/mastodon
            - ../mastodon/public:/app/backup/mastodon/data/files/public:ro
            - ../mastodon/docker-compose.yml:/app/backup/mastodon/docker-compose.yml:ro
            - ../mastodon/.env.production:/app/backup/mastodon/.env.production:ro

Here we tell Kopia to bind the backup folder in the Mastodon docker working directory to the mountpoint /app/backup/mastodon in the Kopia container. We also bind the public folder containing user files and cache to the /app/backup/mastodon/data/files mountpoint. For good measure, I also mount the docker compose file for Mastodon as well as the environment variables so those can be backed up as well.

Here's the initial folder structure for the backup setup on the Docker host machine:

$HOME
└── docker
    ├── kopia
    │   └── backup
    │       └── docker-helper.sh
    └── mastodon
        └── backup
            └── dump_db.sh

The dump_db.sh script makes calls to the Docker API in order to execute commands within the database instances:

#!/bin/bash

WORKDIR=$(dirname "$(readlink -f "$0")")
source $WORKDIR/../docker-helper.sh
mkdir -p $WORKDIR/data/databases

# Postgres dump
echo -n "Creating Postgres dump... "
container_exec mastodon-db-1 "pg_dump -U postgres mastodon_production" > ${WORKDIR}/data/databases/postgres-dump.sql
echo "Done!"

# Redis dump
echo -n "Creating Redis dump... "
container_exec mastodon-redis-1 "redis-cli SAVE"
container_exec mastodon-redis-1 "cat /data/dump.rdb" > "$WORKDIR/data/databases/redis-dump.rdb"

Next thing I need to setup the actual backup in Kopia, called a snapshot. Using my Dropbox repository, I create a snapshot pointing to the /app/backup/mastodon folder in the Kopia container, and add the /app/backup/mastodon/dump_db.sh script as pre-execution (Snapshot Actions ⇛ Before Snapshot) to its policy. The dumps of PostgreSQL and Redis databases will be performed inside respective container and write them to the $HOME/docker/mastodon/backup/databases/.

If I wanted I could also create a post-backup script that removes the database dumps after backup has completed, but I figured there's no harm in having them locally available should I ever need to quickly restore the databases.

Conclusion

The neat thing with this setup, is that it should work for any Docker container. I'll just mount to the Kopia container what I want to backup, make snapshot for the top folder and presto. In fact, I'm using the same setup for several other containers already.

I haven't gone through the Kopia documentation that thoroughly, but from what I gather it seems like Kopia is meant to backup files from local sources or mounted paths. I haven't seen anything noteworthy or any best practices regarding handling Docker containers, so until I find anything that can improve my setup this is how I'll keep it for now.

Links

• Kopia
• rclone