Proxmox: Encrypt data at rest using ZFS

Publication date: 2021-07-06

Issue: VMs and containers shall be kept on encrypted storage

Encrypting data at rest protects data in case of stolen physical disks. ZFS has native encryption capabilities but it is not easily usable in Proxmox.

Solution: Provide encrypted ZFS dataset for use by Proxmox

We use native ZFS encryption with an encryption key stored on another machine. This way, the Proxmox host can be started without requiring entering an encryption key on startup of the machine but still get benefits of having encrypted data at rest.

Here is a howto:

Create an encryption key and store it on your key server (here: mykeyserver.local):

openssl rand -out zfs-testkey.raw 32

Make sure to restrict access so that it can only be retrieved by the host for which the key is relevant. For the example, we make the key accessible via https (https://mykeyserver.local/zfs-testkey.raw). An alternative (I like that better) would be to make it available via ssh access so that it can be retrieved using scp.

Create an encrypted dataset on the Proxmox host like this:

wget --directory-prefix=/dev/shm https://mykeyserver.local/zfs-testkey.raw
zfs create -o encryption=aes-256-gcm -o keyformat=raw -o keylocation=file:///dev/shm/zfs-testkey.raw rpool/pvestorage

"/dev/shm" is volatile storage (RAM disk). Instead of creating a new dataset and making that available in Proxmox, you may also delete the default dataset "rpool/data" and recreate it with encryption.

Encryption cannot be activated on an existing dataset, only on dataset creation.Encryption is inherited to child datasets. This means if we encrypt the dataset in which Proxmox stores disks of virtual machines or containers, these disks also get encrypted.

You can check the encryption status using:

zfs get -p encryption,keystatus,keylocation,encryptionroot

To load the encryption key automatically on machine startup, create a systemd service file

nano -w /etc/systemd/system/zfs-prepare-key.service

with the following content:

[Unit]
Description=Load ZFS encryption key and mount datasets
DefaultDependencies=false
Before=pve-guests.service
After=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=wget --directory-prefix=/dev/shm https://mykeyserver.local/zfs-testkey.raw
ExecStart=/usr/sbin/zfs load-key -a
ExecStart=/sbin/zfs mount -a

[Install]
WantedBy=multi-user.target

We make sure to start this service after the network is available but before Proxmox attempts to automatically start any VM or containers.

Finally we need to make sure that this systemd service is loaded at boot time:

systemctl daemon-reload
systemctl enable zfs-prepare-key