Reflection
Deploying · 40 minutes

Secrets with sops-nix

Prerequisite: NixOS and the module system.

Outcome: you understand what .sops.yaml, an age key, and sops.secrets.* are; you know how a secret gets from your laptop to /run/secrets/<name> on a NixOS host; you know how this repo’s install-host and update-secrets scripts orchestrate the flow.

Managing secrets on Nix is a solved problem, and sops-nix is the solution this fleet base picks. If you have used HashiCorp Vault, Ansible Vault, or age on its own, the mental model is not far off. If not, this chapter builds it from scratch.

Why this problem is hard

You have secrets: an API token, a TLS private key, a database password. You want:

  1. to keep them in git alongside the rest of your infrastructure (so history and review work), but
  2. never to have them in plaintext there, and
  3. to have them appear as plaintext files on the target host so that services can read them, and
  4. to control which host can decrypt which file (a compromised web server should not be able to decrypt the database node’s secrets).

sops (Secrets OPerationS, originally by Mozilla, now maintained under the CNCF getsops org) provides (1) and (2). age is the modern encryption format sops uses (small keys, no PKI ceremony). sops-nix glues sops into NixOS’s module system to give you (3) and (4).

The 90-second picture

                       .sops.yaml
                       (recipients + rules)
                             |
             sops encrypts   v
   plaintext values ----> secrets/.yaml   ---- committed to git
                             |
                             |  (nixos-anywhere ships host's
                             |   age key at first install;
                             |   sops-nix reads it on boot)
                             v
                       /run/secrets/
                       (plaintext, on the target host, root-only)
                             |
                             v
                   consumed by services
                   (e.g. systemd EnvironmentFile,
                    sshd, wireguard, ...)

age, briefly

age is a modern file-encryption tool built by Filippo Valsorda. Keys are short strings starting with AGE-SECRET-KEY- (private) or age1 (public / recipient). No PKI, no key servers, no expirations – you generate a key, share the public half, keep the private half.

$ age-keygen -o ~/.config/sops/age/keys.txt
Public key: age1abc123...

The public key (“recipient”) is what you hand out. The private key lives in that file; nothing else needs to know it.

Never cat or read the private-key file directly. To display the public key from an existing private-key file:

$ age-keygen -y ~/.config/sops/age/keys.txt
age1abc123...

age-keygen -y derives the public part without echoing the private key. This repo’s scripts and this tutorial follow that rule; you should too.

Reference: age(1).

Deriving age from SSH host keys

Here is a clever trick this repo relies on: NixOS servers already have an ed25519 host key (/etc/ssh/ssh_host_ed25519_key) generated at install time. That key is mathematically compatible with age, and the tool ssh-to-age converts it to an age recipient in one line:

$ ssh-to-age -i /etc/ssh/ssh_host_ed25519_key.pub
age1def456...

That means every NixOS host can be an age recipient without any extra key material: sops-nix decrypts at boot using the SSH host key that is already there. The modules/sops.nix module says exactly that:

sops.age.sshKeyPaths = [ "/etc/ssh/ssh_host_ed25519_key" ];

install-host bootstraps this by generating the host’s ed25519 key locally, deriving its age recipient, adding it to .sops.yaml, and shipping the private key to the target as --extra-files during nixos-anywhere. That first-boot has the key in place, so sops-nix can decrypt from the very first boot.

.sops.yaml

The rulebook. Two sections:

  • keys: – a list of anchor-named recipients you can reference by alias further down.
  • creation_rules: – for each path pattern, which recipients can decrypt files created there.

The scaffold from templates/default/.sops.yaml:

keys:
  - &admin_you   age1REPLACE_WITH_YOUR_LAPTOP_AGE_PUBLIC_KEY

creation_rules: []

After install-host web-1 runs, the file grows into something like:

keys:
  - &admin_you   age1youdefinitely...
  - &web-1       age1web1derivedfromitssshkey...

creation_rules:
  - path_regex: secrets/web-1\.yaml$
    key_groups:
      - age: [ *admin_you, *web-1 ]

Read: “files matching secrets/web-1.yaml are encrypted for the admin recipient and for web-1’s own age recipient”. You (on your laptop) can decrypt because your admin key is a recipient; web-1 can decrypt at boot because it holds the private half of its age identity.

If you later add a database node, install-host db-1 appends a similar block with secrets/db-1\.yaml and web-1 will not be a recipient there. Least-privilege is the default.

Reference: getsops/sops – README and Mic92/sops-nix – README.

Encrypted files

An encrypted secrets/web-1.yaml looks like:

# somewhat trimmed
db_password: ENC[AES256_GCM,data:...,iv:...,tag:...]
api_token:   ENC[AES256_GCM,...]
sops:
  age:
    - recipient: age1youdefinitely...
      enc: |
        -----BEGIN AGE ENCRYPTED FILE-----
        ...
        -----END AGE ENCRYPTED FILE-----
    - recipient: age1web1derivedfromitssshkey...
      enc: |
        ...
  lastmodified: ...
  mac: ENC[...]
  • Each secret value is encrypted with a randomly-generated data key.
  • The data key is then wrapped for every recipient in the sops: block.

To decrypt as an admin from your laptop:

$ sops secrets/web-1.yaml

You get an interactive editor with the plaintext. Save and exit; sops re-encrypts. Adding, editing, or removing individual entries this way is safe.

Adding secrets in this repo

Two workflows, pick per situation:

  • From a file (set-secret <host> <key> <file>). Reads the value from a file (or - for stdin) and writes it through sops --set. This is the recommended path: multi-line values (PEMs, private keys, JSON blobs) survive intact, and scripted bootstraps have a single-line command. If .sops.yaml has no creation_rule for the target file yet, set-secret adds one with the admin recipient(s), so it works even before the first install-host for that host.
  • Manual with sops secrets/<host>.yaml. Full editor, edits every field at once. Handy for rare surgical fixes; make sure the file exists and .sops.yaml has a matching creation_rule.

Both orderings work: set-secret before or after install-host. When install-host runs, it adds the host’s own recipient to the creation_rule and calls sops updatekeys – any values you set earlier are re-encrypted for the new recipient set.

sops.secrets.* – the NixOS-side view

On the NixOS side, sops-nix gives you a module option:

sops.secrets.db_password = {
  sopsFile = ./secrets/web-1.yaml;
  owner    = "myservice";
  group    = "myservice";
  mode     = "0400";
};

At boot, sops-nix will:

  1. read /etc/ssh/ssh_host_ed25519_key;
  2. decrypt every sops.secrets.<key> value from the given file;
  3. write plaintext to /run/secrets/<key> with the requested owner, group, and mode.

Your service module then references the plaintext path:

systemd.services.myservice.serviceConfig.EnvironmentFile =
  config.sops.secrets.db_password.path;

config.sops.secrets.db_password.path evaluates to /run/secrets/db_password. Never hard-code the path yourself; let sops-nix give you the option value.

Full option reference: sops-nix README – NixOS options.

Shared secrets across hosts (manual)

The default topology in this repo is one file per host (secrets/<host>.yaml). Every secret a host needs lives in that file, encrypted for that host alone. That is the safest posture: least-privilege by default, and rotating a host’s SSH key rotates its recipient without touching anything else.

Sometimes you have a secret several hosts need: a shared CloudFlare API token for DNS, a monitoring token, a backup encryption passphrase. Duplicating the plaintext into every per-host file works but rotates poorly. sops supports a shared-file pattern natively; the repo doesn’t automate it (yet), so this section is the manual recipe.

Step 1 – declare a shared file in .sops.yaml. Add a keys group and a creation_rule listing every host that should be able to decrypt it:

keys:
  - &admin_you   age1youradminkey...
  - &web-1       age1web1derivedfromssh...
  - &web-2       age1web2derivedfromssh...
  - &db-1        age1db1derivedfromssh...

creation_rules:
  # per-host files (managed by install-host, unchanged)
  - path_regex: secrets/web-1\.yaml$
    key_groups:
      - age: [ *admin_you, *web-1 ]
  - path_regex: secrets/web-2\.yaml$
    key_groups:
      - age: [ *admin_you, *web-2 ]
  - path_regex: secrets/db-1\.yaml$
    key_groups:
      - age: [ *admin_you, *db-1 ]

  # shared file -- readable by whichever hosts you list here
  - path_regex: secrets/shared\.yaml$
    key_groups:
      - age: [ *admin_you, *web-1, *web-2 ]

Step 2 – create the file and set values. set-secret’s first argument is normally a host name; for shared files pass the basename you used in the path_regex above:

$ set-secret shared cloudflare_api_token cf-token.txt

Step 3 – reference it from any host module that needs it:

sops.secrets."cloudflare_api_token" = {
  sopsFile = ../../secrets/shared.yaml;
  owner    = "caddy";
  group    = "caddy";
  mode     = "0400";
};

Both web-1 and web-2 will decrypt the same file at boot; db-1 will not (it isn’t in the shared rule).

Adding a new host to a shared file. When you add-host web-3 and want it to see secrets/shared.yaml too:

  1. Run install-host web-3 first – gets the &web-3 anchor into .sops.yaml.
  2. Hand-edit the shared rule’s age: [...] list to include *web-3.
  3. sops updatekeys secrets/shared.yaml – re-encrypts the file for the new recipient set.
  4. deploy web-3 – the new host now has the secret at boot.

Removing a host from shared. Reverse: remove *web-3 from the shared rule, sops updatekeys secrets/shared.yaml. The next deploy web-3 (or a rebuild) will fail sops-nix decryption for that key – move any dependent modules off shared before revoking.

Why not automate this? The one manual step (editing the age: list) forces the operator to consciously grant a new host access to shared material. Automation here would silently expand the blast radius on every add-host.

update-secrets – recipient resync

update-secrets [host] in the devShell is a thin wrapper over sops updatekeys --yes secrets/<host>.yaml. It re-encrypts a file’s recipient wrappers to match the current recipient set in .sops.yaml; values themselves are untouched.

Called without an argument, it walks every secrets/*.yaml in the fleet and resyncs each.

Use it when:

  • You rotate an admin key (edit .sops.yaml, then update-secrets – every file gets re-encrypted for the new admin).
  • You add or remove a shared-file recipient by hand (edit the rule’s age: [...] list, then update-secrets shared).
  • .sops.yaml and an encrypted file’s sops: block have drifted (for example after a merge conflict).

install-host calls sops updatekeys for you when it modifies .sops.yaml, so you rarely need to reach for update-secrets directly.

Rotating a recipient

Two cases:

  • A host’s age recipient changed (rare – would mean regenerating its SSH host key). Re-run install-host --force after updating .sops.yaml, or run sops updatekeys secrets/<host>.yaml yourself.
  • An admin key was added or removed. Edit .sops.yaml, then sops updatekeys secrets/*.yaml. Every file gets re-encrypted for the new recipient set. Commit.

install-host does the updatekeys step for you when it modifies .sops.yaml.

Reference: getsops/sops – Adding and removing keys.

Common gotchas

  • sops-nix fails at boot with “no age key found”. The host’s SSH host key is missing or does not match the recipient in .sops.yaml. Check ssh-to-age -i /etc/ssh/ssh_host_ed25519_key.pub against what .sops.yaml lists.
  • “Recipient not found in creation rule”. You created a new encrypted file whose path does not match any creation_rules path_regex. Add a rule to .sops.yaml.
  • “Cannot decrypt on the laptop”. Your ~/.config/sops/age/keys.txt is either missing or holds a different key than .sops.yaml’s admin recipient expects. Fix the recipient (and sops updatekeys every file) or fix which key you use.
  • Committing an unencrypted secrets file. Never; use git diff before commits. sops shows the encrypted form; if the file has raw plaintext values, sops was skipped somewhere.

Alternative secret backends

sops-nix also supports GnuPG, AWS KMS, GCP KMS, Azure Key Vault, and HashiCorp Vault – you list them in .sops.yaml alongside age. This tutorial sticks to age because it is the simplest, needs no external service, and works for the “small fleet, personal admin laptop” case this repo targets.

For a cluster with strict compliance where the encryption key must live in an HSM, swap age for KMS in .sops.yaml; the NixOS side does not change. Details: getsops/sops – KMS.

What next

You know the secret pipeline. Next is the first-boot pipeline: how a bare Linux target becomes a NixOS host without a rescue image.

Next: Remote install with nixos-anywhere

References for this chapter

Esc
Start typing to search.