Since I have serveral physical machines and virtual ones backups are a pressing point since their are running critical infrastructure for my company. I needed a fairly complicated setup with two backup layers and two steps as requested by the management. The lower layer is a regulary backup of the virtual machines and the higher layer is a regulary backup of the main data from some machines. These includes the main file store and the mail servers. The second step is some sort of off-site backup for the main data. I have a Synology NAS DS920+ running in a seperate location which is the target for all backups and the off-site backups are done from this machine on a weekly basis.

I use borg for these backup tasks. borg is nearly a fire and forget solution but depending on your setup you should have some sort of documentation about the keys and higher level progress. I recommend borgmatic which is a simple wrapper around borg and allows simple configuration and automation of the whole process. Since we have a Client-Server model borg needs to be installed on both sides.

Server side

In this setup the Synology NAS is the server which accepts incoming ssh connections from the clients and stores the backups in repositories. The SynoCommunity provides additional packages for Synology devices since Synology uses its own custom linux. To install the community repository go to Menu -> Package Center -> Settings and set the trust level to accept trusted publishers. Afterwards, add in the Package Sources tab a new source with some name and https://packages.synocommunity.com/ as location. Now you can install the borg package.

To get ssh access enable it under System Control -> Terminal & SNMP. I recommend to add a new backup user which has lower privileges. Now ssh login should be functional with password authentication. For public key authentication create a new key on the client with ssh-keygen -b 4096 -t rsa -f ~/.ssh/<filename>. Copy the *.pub part into the ~/.ssh/authorized_keys file on the Synology device for your backup users. Enable public key access for ssh within the daemon configuration. Switch to root, change the configuration with vim /etc/ssh/sshd_config and enable the line PubkeyAuthentication yes. Reload the service with synoservicectl --reload sshd. Now the backup user should login with public key authentication. In my case the Synology device was a little bit unruly and I needed to disable password authentication on the client side by adding the line PasswordAuthentication no in the client configuration (~/.ssh/config).

This enables the Synology device to accept requests from a client borg and to store the repository and created archives on the server device.

Off-Site

The second step of the two step backup solution is to store the data files on an external drive. To get a backup everytime I plugin the harddrive I use an udev rule. First start the udev monitor on the Synology device with udevadm monitor --kernel --property --subsystem-match=usb and plugin in the harddrive. Thr output should be something like this:

ERNEL[171776.047360] add      /devices/pci0000:00/0000:00:15.0/usb2/2-2 (usb)
ACTION=add
BUSNUM=002
DEVNAME=/dev/bus/usb/002/006
DEVNUM=006
DEVPATH=/devices/pci0000:00/0000:00:15.0/usb2/2-2
DEVTYPE=usb_device
MAJOR=189
MINOR=133
PHYSDEVBUS=usb
PRODUCT=bc2/231a/712
SEQNUM=2130
SUBSYSTEM=usb
TYPE=0/0/0

KERNEL[171776.047652] add      /devices/pci0000:00/0000:00:15.0/usb2/2-2/2-2:1.0 (usb)
ACTION=add
DEVNAME=/dev/2-2:1.0
DEVPATH=/devices/pci0000:00/0000:00:15.0/usb2/2-2/2-2:1.0
DEVTYPE=usb_interface
INTERFACE=8/6/80
MODALIAS=usb:v0BC2p231Ad0712dc00dsc00dp00ic08isc06ip50in00
PHYSDEVBUS=usb
PRODUCT=bc2/231a/712
SEQNUM=2131
SUBSYSTEM=usb
TYPE=0/0/0

To create a new udev rule take some unique property to match on. In my case the property PRODUCT=bc2/231a/712 to match my external harddrive is sufficient. With this information create a new rule with sudo vim /lib/udev/rules.d/90-offsite-backup.rules and the following content:

ACTION=="add", SUBSYSTEM=="usb", ENV{PRODUCT}=="bc2/231a/712", RUN=="/volume1/borg/nn_offsite.sh"

This rule matches on the add action of the usb subsystem if the eviroment contains the specified product. If the rule is matched it runs the script nn_offsite.sh. The script is fairly simple and against some borg rules (See FAQ).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

#!/bin/bash
#
# This small script creates the offsite-backup from the main data pool
# to an external hard-drive connected via usb-1. It uses rsync to exactly
# clone the repository since borg provides no distinct copy command for such a case
# and this method is discouraged.
#
set -eu pipefail

BORG_REPO="/volume2/files"
HDD="/volumeUSB1/usbshare1-2/offsite"

# Only run if path exists
if [[ -d "${HDD}" ]]; then
  # Execute the sync
  logger -s "Running off-site backup, this may take a while."
  rsync -az --delete "${BORG_REPO}" "${HDD}" && {
    echo $(date +%y-%m-%d:%H-%M) > "${HDD}/last_backup";
    /bin/borg check --repository-only "${HDD}";
    echo -e "Subject: Off-site backup done" | ssmtp <mail@domain.de>
  }
else
  logger -s "Expected path ${HDD} not found skipping backup."
fi

This creates a copy of the borg repository on the external harddrive. Mention that this is discouraged by borg and the prefered way is to make a second backup directly to the device. But for my case it is sufficient. Configure e-mail if ssmtp should be used.

Client side

After we setup the server side with borg and the possibility for off-site backups we can configure the client to do backups on a daily basis for the data and the virtual machines. We start with the data backups. This is fairly simple and I use the borgmatic wrapper for easy configuration. To install the tools run apt install borgbackup borgmatic on a debian machine. Generate the dummy configration with generate-borgmatic-config and a changed configuration can be validated with validate-borgmatic-config. Adjust the dummy configuration to your needs with mg /etc/borgmatic/config.yaml.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

# A minimal configuration for a mail server
location:
      source_directories:
        - /etc
        - /var/vmail

    repositories:
        - borg.backup:/volume2/mail
storage:
    compression: zstd,11
retention:
    # Number of daily archives to keep.
    keep_daily: 7
    # Number of weekly archives to keep.
    keep_weekly: 4
    # Number of monthly archives to keep.
    keep_monthly: 6
consistency:
    checks:
        - repository

This minimal configuration defines two folders to backup and one repository on the Synology NAS (I use ssh configurations for shorter ssh commands.). It uses zstd for compression which can be tuned between 1 (fast) and 21 (high compression). As retention policy a whole week of backups is kept and the last 4 weeks and 6 month. Afterwards, the repository needs to be initialised for the first time with borgmatic init -e none where -e none means no encryption and is discouraged. Use some sort of passphrase or keyfile for encrypting your backup. To run the backup manually use borgmatic --verbosity 1 --files. Check the remote repository with borgmatic info for general informations and borgmatic list for archive informations.

For virtual machines borgmatic can also be used. My virtual machines store their data in plain LVM volumes and the vm manager does regulary snapshots from all virtual machines which are suffixed by _ss%Y%m%d-%H%M. So adjust the borgmatic configuration for a hypervisor like the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

location:
    source_directories:
        - '/dev/vg/*_ss[0-9]*'
    repositories:
        - borg.backup:/volume2/vms
    read_special: true
...
hooks:
    after_backup:
         - /usr/local/bin/nn_after_borg

The rest of the file is likewise to the other one. The read_special: true enables borg to backup from special devices and block devices. Be careful with symbolic links and special block devices like /dev/zero. I use the after_backup hook from borgmatic to change the snapshot state from active to disabled. I leave at least a couple of days of vm snapshots on the main machine for fast repair. The nn_after_borg script is fairly easy:

1
2
3
4
5
6
7

#!/bin/bash
#
# Run after borg backup und disable active snapshots

# 1 Find snaptshots with the current date and print them
# 2 Pass them to a compund command of xargs which change the lvm state from a to k
find /dev/vg/ -name "*_ss[0-9]*" -print | { xargs -I{} lvchange -an "{}"; xargs -I{} lvchange -ky "{}"; }

Daily backups

To run borgmatic on a regular basis I use Systemd timers and services. The service is taken from the official repository, mg /etc/systemd/system/borgmatic.service.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

[Unit]
Description=borgmatic backup
Wants=network-online.target
After=network-online.target
# Prevent borgmatic from running unless the machine is plugged into power. Remove this line if you
# want to allow borgmatic to run anytime.
ConditionACPower=true
OnFailure=status_mail@%n.service

[Service]
Type=oneshot

# Security settings for systemd running as root, optional but recommended to improve security. You
# can disable individual settings if they cause problems for your use case. For more details, see
# the systemd manual: https://www.freedesktop.org/software/systemd/man/systemd.exec.html
LockPersonality=true
# Certain borgmatic features like Healthchecks integration need MemoryDenyWriteExecute to be off.
# But you can try setting it to "yes" for improved security if you don't use those features.
MemoryDenyWriteExecute=no
NoNewPrivileges=yes
PrivateDevices=yes
PrivateTmp=yes
ProtectClock=yes
ProtectControlGroups=yes
ProtectHostname=yes
ProtectKernelLogs=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK
RestrictNamespaces=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM
# To restrict write access further, change "ProtectSystem" to "strict" and uncomment
# "ReadWritePaths", "ReadOnlyPaths", "ProtectHome", and "BindPaths". Then add any local repository
# paths to the list of "ReadWritePaths" and local backup source paths to "ReadOnlyPaths". This
# leaves most of the filesystem read-only to borgmatic.
ProtectSystem=full

CapabilityBoundingSet=CAP_DAC_READ_SEARCH CAP_NET_RAW
# Lower CPU and I/O priority.
Nice=19
CPUSchedulingPolicy=batch
IOSchedulingClass=best-effort
IOSchedulingPriority=7
IOWeight=100

Restart=no
# Prevent rate limiting of borgmatic log events. If you are using an older version of systemd that
# doesn't support this (pre-240 or so), you may have to remove this option.
LogRateLimitIntervalSec=0

# Delay start to prevent backups running during boot. Note that systemd-inhibit requires dbus and
# dbus-user-session to be installed.
ExecStartPre=sleep 1m
ExecStart=systemd-inhibit --who="borgmatic" --why="Prevent interrupting scheduled backup" /usr/bin/borgmatic --verbosity -1 --syslog-verbosity 1

The service file should be self-explaining or with the help of the Systemd man pages. I only fixed the path and added the possibility to send a mail if the service fails (See my former post). The corresponding timer is mg /etc/systemd/system/borgmatic.timer:

1
2
3
4
5
6
7
8
9

[Unit]
Description=Run borgmatic backup

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

Enable the timer with systemctl daemon-reload && systemctl enable --now borgmatic.timer and borg does the backup once a day. If you want to specify a certain time adjust the OnCalendar entry.

Conclusion

This posts showed my incarnation of a backup setup with borg saving data and virtual machines on external servers and external harddrives. Following this template a new setup should be possible within less than a hour. The setup is nearly fire and forget. For recovery methods from borg use the official documentation. Leave me a comment about your setup or recommendations to my setup.