When running servers I want to encrypt the data stored on them. The problem you then pretty quickly run into is that it’s hard to actually boot with an encrypted root. I’ve solved this problem in the past by having a tinysshd in my initramfs which prompts me for a password to unlock the volumes. Though this works, it’s annoying in that the server isn’t able to boot at all, causing any additional monitoring I have to not work. There are also some services on machines that don’t need any encrypted storage and that I’d be happy to have start unattended.
It turns out though, systemd provides us all the tools we need to achieve this kind of split boot.
Setting up the encrypted disks
In my servers there’s usually 2 disks, with a bunch of partitions including a data partition. The data partition is first ecnrypted with LUKS and then turned into a btrfs RAID1.
As a consequence, I have something along the following in /etc/crypttab:
data1 UUID=<UUID> none luks,noauto
data2 UUID=<UUID> none luks,noauto
The noauto is very important here, we don’t want to attempt to unlock these
disks at boot. The UUID comes from blkid /dev/nvme0n1pX etc.
This also provides us with our first building block. For every of these disks,
systemd will automatically generate a service,
systemd-cryptsetup@<NAME>.service. This is done automatically for you by
systemd-cryptsetup-generator.
Having these services is rather handy. You can start them by hand using
systemctl, and systemd will prompt you for the password on the TTY.
Mounting the encrypted disks
Having these cryptsetup services is rather handy, but we need more than that, we need them mounted. Without them being mounted, we won’t be able to use the data stored on them.
Filesystem mounts are defined in /etc/fstab, and I have something like this
in it:
UUID=<UUID> /var/lib/docker btrfs defaults,noauto,noatime,ssd,subvolid=256,subvol=/docker,x-mount.mkdir,x-systemd.requires=systemd-cryptsetup@data1.service,x-systemd.requires=systemd-cryptsetup@data2.service 0 2
UUID=<UUID> /data btrfs defaults,noauto,noatime,ssd,subvolid=259,subvol=/data,x-mount.mkdir,x-systemd.automount,x-systemd.requires=systemd-cryptsetup@data1.service,x-systemd.requires=systemd-cryptsetup@data2.service 0 2
UUID=<UUID> /var/cache/pacman/pkg btrfs defaults,noauto,noatime,ssd,subvolid=260,subvol=/pkgcache,x-mount.mkdir,x-systemd.automount,x-systemd.requires=systemd-cryptsetup@data1.service,x-systemd.requires=systemd-cryptsetup@data2.service 0 2
Make sure that the UUID here is that of blkid /dev/mapper/<NAME>, using the
names you used for devices in /etc/crypttab.
Much like in /etc/crypttab, we need to pass noauto to our mount options,
to ensure we don’t attempt to mount these filesystems on boot.
The big trick here is that we can declare dependencies using x-systemd.requires
as part of our mount options. /etc/fstab in turn is parsed by
systemd-fstab-generator,
yielding .mount units. Their names is the mount point, which each /
replaced by -, so for example var-lib-docker.mount. I strongly suggest
you take a stroll through systemd.mount
to understand all the options at your disposal.
The x-mount.mkdir option will automatically create the target directory
for us if it doesn’t exist, prior to mounting. This avoids silly scenarios
like a mount failing because we forgot to create /data.
Don’t worry about the x-systemd.automount option, we’ll discuss that a bit
later.
Depending on the encrypted mount
Next up, we’ll need to update a service. What we want to do is ensure that a service is only started after that encrypted filesystem has been mounted. If the disk is already unlocked this will be a no-op, if not, it should trigger the system to prompt us for the password and mount the filesystem.
We achieve this through the Requires and After of a service unit. In them
we specify additional dependencies on mount units (and anythign else you’d
like). You can edit one using systemctl edit, lets try with docker.service:
[Unit]
Requires=var-lib-docker.mount
After=var-lib-docker.mount
All together now
With all of this in place, when you systemctl start docker.service systemd
will now try to start var-lib-docker.mount. var-lib-docker.mount is
generated by systemd-fstab-generator. We’ve specified through
x-systemd.requires that in order to start var-lib-docker.mount, we need
systemd-cryptsetup@<NAME> services started. This in turn will cause systemd
to prompt you for the decryption passwords on the TTY. Once the filesystems
have been unlocked and successfully mounted our docker.service will start.
Since the disks are now decrypted, any other .mount units won’t prompt you
for a password and will just mount instead.
Isn’t that lovely?
Be careful not to start these services at boot, i.e don’t systemctl enable
them. If you don’t want to have to start all services individually, make a
custom .target that you then start, and override the WantedBy of the
units so they don’t get started as part of multi-user.target. Only once
you’ve done that should you systemctl enable the services.
Bonus section
Now one last thing, that x-systemd.automount thing. automount is another
type of unit. systemd effectively watches for file access to the mount point,
and when it happens it will automatically start the associated mount unit,
causing the cryptsetup services to start making systemd prompt you for the
password. This means that if you were to cd /data based on the setup I’ve
showed you here, and the disks aren’t decrypted and mounted yet, you’ll be
prompted for you decryption passwords at that point.
You can see automount units in the mount output, as there will be an
additional entry for them:
systemd-1 on /data type autofs (rw,relatime,fd=46,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18402)
/dev/mapper/data1 on /data type btrfs (rw,noatime,ssd,space_cache,subvolid=259,subvol=/data)
The second mount, the type btrfs one, only exists after data.mount has
started, but the autofs one exists on boot.
Automounts are usually only used together with network mounted filesystems, but it provides a nice little usability improvement in our case too.
Now you might be wondering, why don’t you have the automount option on
/var/lib/docker? Well, unfortunately because it trips up Docker. I run
Docker with the btrfs volume driver, but unfortunately at startup Docker
only notices the autofs mount and concludes it shouldn’t load the btrfs
snapshotter plugin. In order to avoid that, and since realistically Docker
is the only thing that should be doing stuff under /var/lib/docker, I omit
the x-systemd.automount.