Reflection
Operating · 45 minutes

Writing host-specific modules

Prerequisite: Day-two operations.

Outcome: you can add a service to a single host; you can write a disko layout; you can extract a common module across hosts; you can override one of the nixops defaults for a single host.

The nixops base gives you a hardened but minimal NixOS: sshd, firewall, root user with your keys, weekly GC, UTC. Everything past that is your fleet’s own. This chapter shows how to add it.

Everything below lives in hosts/<name>/*.nix, unless it is a common piece worth extracting.

Add a service: nginx, once

Say web-1 should serve HTTP. In hosts/web-1/default.nix:

{ config, pkgs, ... }: {
  imports = [ ./hardware-configuration.nix ];

  services.nginx = {
    enable = true;
    recommendedGzipSettings   = true;
    recommendedOptimisation   = true;
    recommendedProxySettings  = true;
    recommendedTlsSettings    = true;
    virtualHosts."example.com" = {
      enableACME = true;                  # ACME (Let's Encrypt) TLS
      forceSSL   = true;
      root       = "/var/www/example.com";
    };
  };

  security.acme = {
    acceptTerms = true;
    defaults.email = "ops@example.com";
  };

  nixops.firewall.allowedTCPPorts = [ 80 443 ];
}

Deploy:

$ deploy web-1

That is it. No apt-get install nginx, no certbot, no templating an nginx.conf. NixOS’s services.nginx module generates the config from the option values, the ACME module requests certs, and the firewall module opens the ports.

Every option has documentation at search.nixos.org/options – look up services.nginx and see the full menu.

Add a service to many hosts: extract a role

Two hosts both need a node_exporter for Prometheus scraping. Instead of duplicating in each hosts/<name>/default.nix, extract:

# roles/node-exporter.nix
{ ... }: {
  services.prometheus.exporters.node = {
    enable = true;
    port   = 9100;
  };
  nixops.firewall.allowedTCPPorts = [ 9100 ];
}

Then pull it in from each host:

# hosts.nix
web-1 = { ip = "1.2.3.4"; modules = [ ./hosts/web-1 ../roles/node-exporter.nix ]; };
db-1  = { ip = "5.6.7.8"; modules = [ ./hosts/db-1  ../roles/node-exporter.nix ]; };

The paths above are illustrative; the exact layout is up to you. Some fleets prefer a single default.nix per host that itself imports role modules; both work.

Override a nixops default for one host

Every base module has an enable option (NixOS and the module system). Turn one off for a specific host:

# hosts.nix
console-1 = {
  ip = "9.9.9.9";
  modules = [ ./hosts/console-1 ];
  nixops.firewall.enable = false;              # this host is behind a physical firewall
};

Or override an option value:

edge-1 = {
  ip = "10.11.12.13";
  modules = [ ./hosts/edge-1 ];
  nixops.ssh.port = 2222;
};

Remember: the fields on the inventory entry become an inline module fragment for that host (Anatomy of an instance repo). If the change is more than a few lines, write it in hosts/edge-1/default.nix instead and reference it via modules = [ ./hosts/edge-1 ].

Write a disko layout

nixops pulls disko in as a flake input and imports its module for every host, so disko.devices is available out of the box. During add-host you either pick a preset (e.g. hetzner-vm) and get a ready hosts/<name>/disko.nix, or pick custom and drop your own.

Full walkthrough with examples (single-disk ext4, ZFS mirror, escape hatches, contributing a preset back) lives in Host files: disko, hardware-config, default.nix.

Add a user

Non-root users are the fleet’s own concern (see modules/users.nix – only root’s authorized_keys is managed by the base). Add users in a per-host or per-role module:

# roles/app-user.nix
{ ... }: {
  users.users.app = {
    isSystemUser = true;
    group        = "app";
    home         = "/var/lib/app";
    createHome   = true;
  };
  users.groups.app = { };
}

For interactive human accounts:

users.users.alice = {
  isNormalUser = true;
  extraGroups  = [ "wheel" ];
  openssh.authorizedKeys.keys = [ "ssh-ed25519 ..." ];
};

Wire up a systemd service using a secret

Full pattern: declare the secret, write a service unit that consumes it, restart when the secret changes.

# hosts/web-1/default.nix
{ config, pkgs, ... }: {
  imports = [ ./hardware-configuration.nix ];

  sops.secrets.stripe_secret = {
    sopsFile = ../../secrets/web-1.yaml;
    owner    = "web-app";
    mode     = "0400";
  };

  systemd.services.web-app = {
    wantedBy = [ "multi-user.target" ];
    after    = [ "network.target" ];
    serviceConfig = {
      User            = "web-app";
      EnvironmentFile = config.sops.secrets.stripe_secret.path;
      ExecStart       = "${pkgs.web-app}/bin/web-app";
      Restart         = "on-failure";
    };
    restartTriggers = [ config.sops.secrets.stripe_secret.sopsFile ];
  };

  users.users.web-app.isSystemUser = true;
  users.users.web-app.group        = "web-app";
  users.groups.web-app = { };
}
  • sops.secrets.stripe_secret.path is /run/secrets/stripe_secret.
  • restartTriggers = [ config.sops.secrets.stripe_secret.sopsFile ] makes the service restart when the encrypted secret file changes (Nix hashes the .sopsFile path).
  • After adding this to a module, run set-secret web-1 stripe_secret <file> to store the value, then deploy web-1.

Public HTTPS via caddy + Cloudflare Origin CA

The most common “serve traffic to the internet” recipe: caddy terminates TLS with a Cloudflare Origin CA cert (issued from the Cloudflare dashboard, valid only when the client is CF’s proxy) and CF handles the public-facing cert. No ACME/Let’s Encrypt required, DNS-01 not required, no ports opened outbound for cert renewal – just the cert files.

# hosts/demo/default.nix
{ config, ... }: {
  imports = [ ./hardware-configuration.nix ];

  sops.secrets."cloudflare_origin_cert" = {
    sopsFile = ../../secrets/demo.yaml;
    owner = "caddy";
    group = "caddy";
    mode  = "0400";
  };
  sops.secrets."cloudflare_origin_key" = {
    sopsFile = ../../secrets/demo.yaml;
    owner = "caddy";
    group = "caddy";
    mode  = "0400";
  };

  networking.firewall.allowedTCPPorts = [ 80 443 ];

  services.caddy = {
    enable = true;
    virtualHosts."demo.example.com".extraConfig = ''
      tls ${config.sops.secrets."cloudflare_origin_cert".path} \
          ${config.sops.secrets."cloudflare_origin_key".path}
      respond "hello world" 200
    '';
  };
}

Setup:

  1. In Cloudflare dashboard: SSL/TLS -> Origin Server -> Create Certificate. Download the cert and key files.
  2. Add a DNS A record for demo.example.com pointing at the host, orange-cloud (proxied) ON. CF terminates the public cert; your caddy terminates the internal one.
  3. On the operator VM:
    set-secret demo cloudflare_origin_cert cf-origin.crt
set-secret demo cloudflare_origin_key  cf-origin.key
  4. install-host demo (first time) or deploy demo (updates).

Rotating the Origin CA cert is just steps 3-4 again with the new files – CF issues Origin CA certs valid up to 15 years, so this is rare.

Add a package to a single host

# hosts/web-1/default.nix
{ pkgs, ... }: {
  environment.systemPackages = with pkgs; [
    postgresql
    tmux
  ];
}

For the whole fleet, either extend the base module in nixops itself, or wrap mkNixosConfigs in your fleet with a small adapter that adds a shared modules fragment.

Override a package version

To pin nginx to a specific version, use an overlay:

# flake.nix (outputs)
let
  pkgsOverride = final: prev: {
    nginx = prev.nginx.overrideAttrs (old: { version = "1.24.0"; src = ...; });
  };
in {
  nixosConfigurations = nixops.lib.mkNixosConfigs {
    inherit hosts sshKeys;
    # ... you would extend mkNixosConfigs to accept extraModules and pass
    #     { nixpkgs.overlays = [ pkgsOverride ]; }
  };
}

For a small change, services.nginx.package = pkgs.nginxMainline; is often enough. Overlays get their own manual chapter: Nixpkgs Manual – Overlays.

Add a scheduled task

{
  systemd.timers.backup = {
    wantedBy  = [ "timers.target" ];
    timerConfig.OnCalendar = "daily";
  };
  systemd.services.backup = {
    serviceConfig.Type = "oneshot";
    script = ''
      ${pkgs.restic}/bin/restic backup /var/lib/app
    '';
  };
}

Native systemd timers – no cron file to template. NixOS’s timer module wraps this: see services.cron for classic cron, or the services.borgbackup option tree for a preset.

Add a Docker or Podman container

{
  virtualisation.oci-containers = {
    backend = "podman";
    containers.grafana = {
      image = "grafana/grafana:11.4.0";
      ports = [ "3000:3000" ];
      environment = {
        GF_SECURITY_ADMIN_PASSWORD_FILE = "/run/secrets/grafana_admin";
      };
    };
  };
}

For anything that has a native NixOS module (Grafana does – services.grafana), prefer that; you get option-checked config and no image-provenance headache.

Test locally before deploying

deploy web-1 -- --dry-activate builds + copies the closure to the target but skips the final activation – catches most breakage without touching the running system. For pure-local iteration on your workstation (no target involved yet), use plain Nix builders:

  • nix build .#nixosConfigurations.web-1.config.system.build.toplevel – build the whole system closure locally. Catches evaluation errors, missing packages, type errors on options.
  • nix build .#nixosConfigurations.web-1.config.system.build.vm && ./result/bin/run-web-1-vm – boot the config in a QEMU VM on your laptop. Great for services you can exercise without real hardware. See NixOS Manual – test VMs.

Neither of these needs nixos-rebuild; the devShell already gives you nix.

Documenting host quirks

When a host has something nonstandard, drop a comment in its module explaining why. git blame on hosts/<name>/default.nix should tell the next operator (or you-in-six-months) why every line is there.

What next

You know how to add anything you would add on a Debian box, but now it lives in git and deploys deterministically. Troubleshooting is the failure manual for when things do not work.

Next: Troubleshooting

References for this chapter

Esc
Start typing to search.