Get assimilated
Since I have serveral physical machines and virtual ones backups are a pressing point since their are running critical infrastructure for my company. I needed a fairly complicated setup with two backup layers and two steps as requested by the management. The lower layer is a regulary backup of the virtual machines and the higher layer is a regulary backup of the main data from some machines. These includes the main file store and the mail servers. The second step is some sort of off-site backup for the main data. I have a Synology NAS DS920+ running in a seperate location which is the target for all backups and the off-site backups are done from this machine on a weekly basis.
I use borg for these backup tasks. borg is nearly a fire and forget solution but depending on your
setup you should have some sort of documentation about the keys and higher level progress.
I recommend borgmatic which is a simple wrapper around borg and allows simple configuration and
automation of the whole process. Since we have a Client-Server model borg needs to be installed
on both sides.
Server side
In this setup the Synology NAS is the server which accepts incoming ssh connections from the clients
and stores the backups in repositories. The SynoCommunity provides additional packages for Synology devices
since Synology uses its own custom linux. To install the community repository go to Menu -> Package Center -> Settings
and set the trust level to accept trusted publishers. Afterwards, add in the Package Sources tab a new source with
some name and https://packages.synocommunity.com/ as location. Now you can install the borg package.
To get ssh access enable it under System Control -> Terminal & SNMP. I recommend to add a new backup user
which has lower privileges. Now ssh login should be functional with password authentication. For public key authentication
create a new key on the client with ssh-keygen -b 4096 -t rsa -f ~/.ssh/<filename>.
Copy the *.pub part into the ~/.ssh/authorized_keys file on the Synology device for your backup users. Enable
public key access for ssh within the daemon configuration. Switch to root, change the configuration with
vim /etc/ssh/sshd_config and enable the line PubkeyAuthentication yes.
Reload the service with synoservicectl --reload sshd. Now the backup user should login with public key authentication.
In my case the Synology device was a little bit unruly and I needed to disable password authentication on the client side by adding
the line PasswordAuthentication no in the client configuration (~/.ssh/config).
This enables the Synology device to accept requests from a client borg and to store the repository and created archives on the server device.
Off-Site
The second step of the two step backup solution is to store the data files on an external drive. To get a backup everytime I plugin
the harddrive I use an udev rule. First start the udev monitor on the Synology device with udevadm monitor --kernel --property --subsystem-match=usb
and plugin in the harddrive. Thr output should be something like this:
ERNEL[171776.047360] add /devices/pci0000:00/0000:00:15.0/usb2/2-2 (usb)
ACTION=add
BUSNUM=002
DEVNAME=/dev/bus/usb/002/006
DEVNUM=006
DEVPATH=/devices/pci0000:00/0000:00:15.0/usb2/2-2
DEVTYPE=usb_device
MAJOR=189
MINOR=133
PHYSDEVBUS=usb
PRODUCT=bc2/231a/712
SEQNUM=2130
SUBSYSTEM=usb
TYPE=0/0/0
KERNEL[171776.047652] add /devices/pci0000:00/0000:00:15.0/usb2/2-2/2-2:1.0 (usb)
ACTION=add
DEVNAME=/dev/2-2:1.0
DEVPATH=/devices/pci0000:00/0000:00:15.0/usb2/2-2/2-2:1.0
DEVTYPE=usb_interface
INTERFACE=8/6/80
MODALIAS=usb:v0BC2p231Ad0712dc00dsc00dp00ic08isc06ip50in00
PHYSDEVBUS=usb
PRODUCT=bc2/231a/712
SEQNUM=2131
SUBSYSTEM=usb
TYPE=0/0/0
To create a new udev rule take some unique property to match on. In my case the property PRODUCT=bc2/231a/712 to match
my external harddrive is sufficient. With this information create a new rule with sudo vim /lib/udev/rules.d/90-offsite-backup.rules and
the following content:
ACTION=="add", SUBSYSTEM=="usb", ENV{PRODUCT}=="bc2/231a/712", RUN=="/volume1/borg/nn_offsite.sh"
This rule matches on the add action of the usb subsystem if the eviroment contains the specified product. If the rule is matched it runs
the script nn_offsite.sh. The script is fairly simple and against some borg rules (See FAQ).
1#!/bin/bash
2#
3# This small script creates the offsite-backup from the main data pool
4# to an external hard-drive connected via usb-1. It uses rsync to exactly
5# clone the repository since borg provides no distinct copy command for such a case
6# and this method is discouraged.
7#
8set -eu pipefail
9
10BORG_REPO="/volume2/files"
11HDD="/volumeUSB1/usbshare1-2/offsite"
12
13# Only run if path exists
14if [[ -d "${HDD}" ]]; then
15 # Execute the sync
16 logger -s "Running off-site backup, this may take a while."
17 rsync -az --delete "${BORG_REPO}" "${HDD}" && {
18 echo $(date +%y-%m-%d:%H-%M) > "${HDD}/last_backup";
19 /bin/borg check --repository-only "${HDD}";
20 echo -e "Subject: Off-site backup done" | ssmtp <mail@domain.de>
21 }
22else
23 logger -s "Expected path ${HDD} not found skipping backup."
24fi
This creates a copy of the borg repository on the external harddrive. Mention that this is discouraged by borg and the prefered way
is to make a second backup directly to the device. But for my case it is sufficient. Configure e-mail if ssmtp should be used.
Client side
After we setup the server side with borg and the possibility for off-site backups we can configure the client to do backups on a daily basis for
the data and the virtual machines. We start with the data backups. This is fairly simple and I use the borgmatic wrapper for easy configuration.
To install the tools run apt install borgbackup borgmatic on a debian machine. Generate the dummy configration with generate-borgmatic-config and a changed
configuration can be validated with validate-borgmatic-config. Adjust the dummy configuration to your needs with mg /etc/borgmatic/config.yaml.
1# A minimal configuration for a mail server
2location:
3 source_directories:
4 - /etc
5 - /var/vmail
6
7 repositories:
8 - borg.backup:/volume2/mail
9storage:
10 compression: zstd,11
11retention:
12 # Number of daily archives to keep.
13 keep_daily: 7
14 # Number of weekly archives to keep.
15 keep_weekly: 4
16 # Number of monthly archives to keep.
17 keep_monthly: 6
18consistency:
19 checks:
20 - repository
This minimal configuration defines two folders to backup and one repository on the Synology NAS (I use ssh configurations for shorter ssh commands.).
It uses zstd for compression which can be tuned between 1 (fast) and 21 (high compression). As retention policy a whole week of backups is kept and the last 4 weeks
and 6 month. Afterwards, the repository needs to be initialised for the first time with borgmatic init -e none where -e none means no encryption and is discouraged.
Use some sort of passphrase or keyfile for encrypting your backup. To run the backup manually use borgmatic --verbosity 1 --files. Check the remote repository
with borgmatic info for general informations and borgmatic list for archive informations.
For virtual machines borgmatic can also be used. My virtual machines store their data in plain LVM volumes and the vm manager does regulary snapshots from all
virtual machines which are suffixed by _ss%Y%m%d-%H%M. So adjust the borgmatic configuration for a hypervisor like the following:
1location:
2 source_directories:
3 - '/dev/vg/*_ss[0-9]*'
4 repositories:
5 - borg.backup:/volume2/vms
6 read_special: true
7...
8hooks:
9 after_backup:
10 - /usr/local/bin/nn_after_borg
The rest of the file is likewise to the other one. The read_special: true enables borg to backup from special devices and block devices. Be careful with symbolic links and special
block devices like /dev/zero. I use the after_backup hook from borgmatic to change the snapshot state from active to disabled. I leave at least a couple of days of vm snapshots on
the main machine for fast repair. The nn_after_borg script is fairly easy:
1#!/bin/bash
2#
3# Run after borg backup und disable active snapshots
4
5# 1 Find snaptshots with the current date and print them
6# 2 Pass them to a compund command of xargs which change the lvm state from a to k
7find /dev/vg/ -name "*_ss[0-9]*" -print | { xargs -I{} lvchange -an "{}"; xargs -I{} lvchange -ky "{}"; }
Daily backups
To run borgmatic on a regular basis I use Systemd timers and services. The service is taken from the official repository, mg /etc/systemd/system/borgmatic.service.
1[Unit]
2Description=borgmatic backup
3Wants=network-online.target
4After=network-online.target
5# Prevent borgmatic from running unless the machine is plugged into power. Remove this line if you
6# want to allow borgmatic to run anytime.
7ConditionACPower=true
8OnFailure=status_mail@%n.service
9
10[Service]
11Type=oneshot
12
13# Security settings for systemd running as root, optional but recommended to improve security. You
14# can disable individual settings if they cause problems for your use case. For more details, see
15# the systemd manual: https://www.freedesktop.org/software/systemd/man/systemd.exec.html
16LockPersonality=true
17# Certain borgmatic features like Healthchecks integration need MemoryDenyWriteExecute to be off.
18# But you can try setting it to "yes" for improved security if you don't use those features.
19MemoryDenyWriteExecute=no
20NoNewPrivileges=yes
21PrivateDevices=yes
22PrivateTmp=yes
23ProtectClock=yes
24ProtectControlGroups=yes
25ProtectHostname=yes
26ProtectKernelLogs=yes
27ProtectKernelModules=yes
28ProtectKernelTunables=yes
29RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK
30RestrictNamespaces=yes
31RestrictRealtime=yes
32RestrictSUIDSGID=yes
33SystemCallArchitectures=native
34SystemCallFilter=@system-service
35SystemCallErrorNumber=EPERM
36# To restrict write access further, change "ProtectSystem" to "strict" and uncomment
37# "ReadWritePaths", "ReadOnlyPaths", "ProtectHome", and "BindPaths". Then add any local repository
38# paths to the list of "ReadWritePaths" and local backup source paths to "ReadOnlyPaths". This
39# leaves most of the filesystem read-only to borgmatic.
40ProtectSystem=full
41
42CapabilityBoundingSet=CAP_DAC_READ_SEARCH CAP_NET_RAW
43# Lower CPU and I/O priority.
44Nice=19
45CPUSchedulingPolicy=batch
46IOSchedulingClass=best-effort
47IOSchedulingPriority=7
48IOWeight=100
49
50Restart=no
51# Prevent rate limiting of borgmatic log events. If you are using an older version of systemd that
52# doesn't support this (pre-240 or so), you may have to remove this option.
53LogRateLimitIntervalSec=0
54
55# Delay start to prevent backups running during boot. Note that systemd-inhibit requires dbus and
56# dbus-user-session to be installed.
57ExecStartPre=sleep 1m
58ExecStart=systemd-inhibit --who="borgmatic" --why="Prevent interrupting scheduled backup" /usr/bin/borgmatic --verbosity -1 --syslog-verbosity 1
The service file should be self-explaining or with the help of the Systemd man pages. I only fixed the path and added the possibility to send a mail
if the service fails (See my former post). The corresponding timer is mg /etc/systemd/system/borgmatic.timer:
1[Unit]
2Description=Run borgmatic backup
3
4[Timer]
5OnCalendar=daily
6Persistent=true
7
8[Install]
9WantedBy=timers.target
Enable the timer with systemctl daemon-reload && systemctl enable --now borgmatic.timer and borg does the backup once a day.
If you want to specify a certain time adjust the OnCalendar entry.
Conclusion
This posts showed my incarnation of a backup setup with borg saving data and virtual machines
on external servers and external harddrives. Following this template a new setup should be possible
within less than a hour. The setup is nearly fire and forget. For recovery methods from borg use
the official documentation. Leave me a comment about your setup or recommendations to my setup.