I use Let’s encrypt to obtain valid TLS cerftificates for various services including mail. Each service has its own virtual machine or is a seperate physical machine. While having only one public IP address I use nginx as reverse proxy and TLS termination point for traffic running over HTTP/HTTPS. Since this machine is the only one accessible from the world wide web it also handles the certificate request for all other machines which need valid TLS certificates. After some web research I noticed that there is no common way for distributing certificates across a network of machines. So the following is my way to handle this problem. Comments with improvements or recommendations for other ways are highly welcome.

Providing certificates

I use certbot to request new certificates. The possible ways to get a new one are well described within the manual so I jump straight to providing certificates to other machines. There are multiple possible ways to get the certificates to other machines. On way is to use ssh to copy the certificates directly to other machines but not all of my machines are running ssh. Another way is to use the nginx to provide the encrypted certificates and keys via HTTPS within the local network.

I wrote a small script which parses a simple configuration, encrpyts the selected certificates and copies them within the document root of some nginx virtual host. The script resides as nn_cert_provisioning within /usr/local/bin/.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

#!/bin/env bash
set -eu pipefail

# Define conf folder, password file and document root
CONF="${HOME}"/certs
PASS="${CONF}"/passfile
DEPLOY=/var/www/deploy

# Create deployment folder and check important files.
mkdir -p "${DEPLOY}"
if [ ! -d "${CONF}" ]; then logger -s "No configuration found under ${CONF}" && exit 1; fi
if [ ! -w "${DEPLOY}" ]; then logger -s "${DEPLOY} not writable" && exit 1; fi
if [ ! -e "${PASS}" ]; then logger -s "No file ${PASS}, creating one" && $(< /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c16 > "${PASS}") && echo "" >> "${PASS}"; fi

# Set trap for cleanup
TEMPDIR="$(mktemp -d)"
cleanup() {
    echo    rm -rf -- "${TEMPDIR}"
}
trap cleanup EXIT

for DOMAIN in $(find "${CONF}" -type l -printf '%f\n'); do
  # prepare certificate for distribution as tar archive
  logger -s "Provision ${DOMAIN}"
  tar hczfv "${TEMPDIR}"/"${DOMAIN}".tgz "${CONF}/${DOMAIN}"

  # encrypt symmetric with aes and a custom password
  # use symmetric aes with 256 bits and sha512 hashing, use initial salting and password based key derivation
  openssl enc -aes-256-cbc -md sha512 -pbkdf2 -salt -in "${TEMPDIR}"/"${DOMAIN}".tgz -out "${TEMPDIR}"/"${DOMAIN}".enc -pass file:"${PASS}"

  # deploy cert to /var/www/deploy which is used to serve new certs
  cp "${TEMPDIR}"/"${DOMAIN}".enc "${DEPLOY}"
  chown www-data:www-data -R "${DEPLOY}"
done

# inform the nginx
systemctl reload nginx

logger -s "Successfully refreshed provisioned certificates."
exit 0

The script assumes that a folder ~/certs exists within your home directory which contains a passfile with the shared password for encrpytion. Otherwise, the script creates a random password. Now, you link every certificate in the directory which shall be distributed, for example ln -s /etc/letsencrypt/live/mail.domain.com ~/certs/mail.domain.com. The script follows every link in the certs directory and creates a tar archive which gets encrpyted with openssl and the password from the passfile. This is copied to the document root used for provisioning and nginx gets informed about the new files.

To automate the process one can use either the certbot deploy hook which runs after every successfull renew of a certificate or with a timer. I use systemd-timers for easier monitoring with monit and sending mails with the result to my account.

A systemd-timer is a special systemd unit which runs on specific time points and triggers normal systemd service units. It is recommended that the timer and the service unit have the same name. Systemd provides extensive documentation via man systemd.service and man systemd.timer such that I don’t explain to much on that.

First create the service file /etc/systemd/system/nn_provision_certs.service with the following content:

[Unit]
Description=Provision Let's encrypt certifictes via nginx

[Service]
Type=oneshot
ExecStart=/usr/local/bin/nn_provision_cert
Environment=HOME='/root'

[Install]
WantedBy=default.target

Adjust the Environment if your home directory is not root. To test the new unit file run systemctl daemon-reload && systemctl start nn_provision_certs.service. Now create the timer file /etc/systemd/system/nn_provision_certs.timer with the following content:

[Unit]
Description=Provision certificates once a day

[Timer]
# Run daily at 2 am
OnCalendar=*-*-* 02:00:00
# Execute if job was missed due to machine was offline
Persistent=true

[Install]
WantedBy=timers.target

This timer runs every day at 2 AM and will be made up when the machine was turned off. Enable the timer with sysemctl daemon-reload, systemctl enable --now nn_provisioning_certs.timer. You can check that the timer is active with systemctl list-timers.

Notification time

This runs the certificate provisioning every day but if something happens it remains undetected. This can be changed fairly easy by sending a mail if the job fails. I use msmtp to send mails from a machine to my mail server. The setup is straight foreward and well described within the Debian and Arch Linux wiki. After msmtp works and can send mails create a new script /usr/local/bin/nn_systemd_mail with the following content and make it executable:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

#!/bin/bash

/usr/bin/msmtp -t <<ERRMAIL
To: $1
From: systemd <root@$HOSTNAME>
Subject: $2
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=UTF-8

$(systemctl status --full "$2")
ERRMAIL

This script can be used to send informations to about failed units to some recipient. To be used within systemd we need another service file /etc/systemd/system/status_mail@.service:

[Unit]
Description=status email for %i to the admins

[Service]
Type=oneshot
ExecStart=/usr/local/bin/nn_systemd_mail admins@domain.de %i
User=nobody
Group=systemd-journal

This unit file can now send mails for every other unit file. You can test the unit with something like systemctl daemon-reload && systemctl start status_mail@dbus.service. You should get a mail with the otput of systemctl status --full dbus.service. Now, attach this file to the provisioning unit file by adding the line OnFailure=status_mail@%n.service to the [Unit] section. The %n passes the name of the failed unit file to the mail service.

Now, the provisioning of Let’s encrypt certificates is mostly done. The backend automatically renews the certificates (thanks to certbot), the nn_provision_certs script encrypts the certificates and moves them to the correct place. Also, if some error happens we get a mail.

Make the certificates accessible

At a last step on the provisioning side we need to inform nginx that it provides our certificates for the internal network. I took the virtual host configuration of a simple proxy and added the following lines to the configuration:

# More configuration for the virtual host

# certificate deployment here
location /deploy {
        alias /var/www/deploy/;

        # allow head requests from all networks
        # and the other requests only from the server internal network
        limit_except HEAD {
                allow <internal network ip>/24;
                deny all;
        }
}

# More configuration for the virtual host

This inserts a new location to the virtual host which is called /deploy and the document root is an alias path to /var/www/deploy/ which is used by the provisioning script. The limit_except HEAD directive informs the webserver that only the internal network is allowed and otherwise only HEAD requests are allowed. Now, you can use curl or some other command to download the certificates within your internal network.

On the target machine

With this setup every machine that needs a Let’s encrypt certificate can just download it and deploy it to the correct place. The following script is fairly complicated but does more than downloading the certificates.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

#!/usr/bin/env bash
set -eu pipefail

# User variables to the certificate configuration
CONF="$HOME/certs/"
PASS="$HOME/certs/passfile"
URL="https://<domain.de>/deploy"

# Internal variables
SSL="/etc/ssl"
# Assure correct file permissions
umask 0277

# Exit message
bail(){
  logger -s "$1"; exit 1;
}

# Show help messages
help() {
    echo "Usage: $(basename $0) [-hlu]"
    echo "Manage certificates provided by a web server."
    echo -e "\n\t-l - Show active certificates and end date"
    echo -e "\n\t-u - Update certificates"
}

The first part defines the configuration directory and pass file as well as the download url. Adjust these settings to your needs and copy the password defined on your provisioning machine to match the one used by the client.

1
2
3
4
5
6
7
8

# List certificates deployed on the local machine
list() {
  echo "Deployed certificates"
  for domain in $(find "${base_dir}" -type f \( -iname "*" ! -iname "passfile" -and ! -iname "*.enc*" \) -printf '%f\n'); do
      echo -n "$domain "
      openssl x509 -enddate -noout -in "${ssl_base}/certs/${domain}.pem"
  done
}

The next part is only a utility function which prints for each certificate the valid until date.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69

update() {
  logger -s "Deploy certificates"
  # Set trap for cleanup
  tempdir="$(mktemp -d)"
  cleanup() {
    echo    rm -rf -- "$tempdir"
  }
  trap cleanup EXIT

  for domain in $(find "${CONF}" -type f \( -iname "*" ! -iname "passfile" -and ! -iname "*.enc*" \) -printf '%f\n'); do
      # Create old file in case of non-existence
      touch "${SSL}/certs/${domain}.pem" "${SSL}/private/${domain}.key.pem"
      logger -s "Downloading $domain"
      curl -s -L "${URL}/${domain}.enc" > "${tempdir}/${domain}.enc" || bail 'cannot download file'
      file "${tempdir}/${domain}.enc" | grep -q openssl || bail 'file downloaded is not an openssl encrypted tar archive'

      # Decrypt cert bundle
      openssl enc -d -md sha512 -pbkdf2 -salt -aes-256-cbc -in ${tempdir}/${domain}.enc  -out "${tempdir}"/mail.out.tgz -pass file:${PASS} || \
        bail 'failed to decrypt ${domain}'

      # Extract cert bundle and get the cert and key
      tar -xf "${tempdir}"/mail.out.tgz -C "${tempdir}" || bail 'failed to extract mail.out.tgz'
      CERT="$(find "${tempdir}" -name 'fullchain.pem')"
      KEY="$(find "${tempdir}" -name 'privkey.pem')"

       # Only update if the cert or key differs
      cmp "${CERT}" "${SSL}/certs/${domain}.pem" && cmp "${KEY}" "${SSL}/private/${domain}.key.pem" && { logger -s 'No certificate update needed'; exit 0; } || {
        logger -s "Update certificates"

        # Rotate fullchain
        test '-----BEGIN CERTIFICATE-----' = "$(head -n 1 "${CERT}")"  || bail 'invalid or missing new fullchain.pem'
        test -f ${SSL}/certs/${domain}.pem && cp ${SSL}/certs/${domain}.pem "${SSL}/certs/${domain}.pem.last"
        cat "$CERT" > ${SSL}/certs/${domain}.pem

        # Rotate key
        test '-----BEGIN PRIVATE KEY-----' = "$(head -n 1 "$KEY")"  || bail 'invalid or missing new privkey.pem'
        test -f ${SSL}/private/${domain}.key.pem && cp ${SSL}/private/${domain}.key.pem "${SSL}/private/${domain}.key.pem.last"
        cat "$KEY" > ${SSL}/private/${domain}.key.pem
        logger -s "updated certificates $(date -Iseconds), new certificate valid until $(openssl x509 -enddate -noout -in ${SSL}/certs/${domain}.pem)"
      }
  done

  # Restart the mail services
  systemctl reload dovecot postfix
}

# Main entry point
while getopts ":hlu" opts; do
  case ${opts} in
    l)
      list
      ;;
    u)
      update
      ;;
    h)
      help
      exit 0
      ;;
    \?)
      echo "Invalid option: -$OPTARG" >&2
      exit 1
      ;;
    :)
      echo "Option -$OPTARG requires an argument." >&2
      exit 1
      ;;
  esac
done

The last part is quite a bit longer but the option parsing should be familiar. The update function starts with a trap which removes the temporary directory at the end or if the script is interrupted. Now to the hot part. Likewise to the provisioning script this one uses the ~/certs folder and takes every file execept the passfile as domain name. So to add a domain to get deployed just run touch ~/certs/<domain.name>. The script tries to download the new certificate and checks if the file is properly encrypted. Afterwards, it tries to decrypt the file within a temp folder and checks if the key and fullchain differs from the one installed locally. If so the fullchain and key gets rotated such that the current and last key and chain remains on the machine. If everything runs well the mail services or something else gets informed to load the new certificates.

I use systemd-timers again to run the script on a daily basis and the mail mechanic to get informed if some error happens.

Conclusion

Okay, this post got longer than expected and contains a lot more code than I expected. I’m running this system since half a year without complications or certificate problems. Feel free to leave me comment about your implementations or improvements to my solution.