Context

I possess a set of internal web applications that are only available from my tailnet (virtual network using tailscale and headscale). Headscale allows the declaration of extra DNS records ¹, for instance:

  services.headscale.settings.dns = {
    extra_records = [
      {
        name = "deluge.net.gq";
        type = "A";
        value = "100.64.0.3";
      }
    ];
  };

Then, I set up a Nginx reverse proxy for that record and allow clients connected to the tailnet to access the service as follows:

  services.nginx.virtualHosts."deluge.net.gq" = {
    locations."/" = {
      # Only allow access from the tailnet.
      extraConfig = "
      allow 100.64.0.0/10;
      deny all;";
      proxyPass = "http://127.0.0.1:8112";
    };
  };

Now, any client connected to my tailnet can access the Deluge web application through the URL: http://deluge.net.gq. However, the connection uses HTTP and because I do not actually own the net.gq domain name, I cannot use Let’s Encrypt to deploy TLS certificates for this endpoint (which is the service used in the background when setting enableACME to true ²).

Hence, to get an encrypted connection for this web application, I need to set up my own Certificate Authority. This post focus on the deployment of the step-ca service using NixOS. We will also detail how to configure NixOS clients to request certificates to our CA.

Deploying a Certificate Authority

step-ca ³ is the self-hosted Certificate Authority service that I will use. It will emit and renew certificates for these custom, private endpoints using the ACME protocol. ACME stands for Automated Certificate Management Environment and is a protocol describing exchanges made between a Certificate Authority, and users requesting certificates. step-ca is packaged in nixpkgs and possess multiple options ⁴, the relevant one for us are the following ones:

• port: Port on which the service will listen to.
• address Address on which the service will listen to.

These will be used to create the URLs for the different ACME operations obtained by clients by sending a query to the Directory URL ⁵. Hence, if you expect other clients than the machine hosting the CA to fetch certificates, the address value must be an IP or FQDN the client can access. Endpoints provided by the directory object will be of the form: https://<address>:<port>/acme/<provisioner-name>/directory.

• intermediatePasswordFile: Path to the clear text file containing the password for the intermediate certificate private key.
• enable Enable the service
• settings Attribute set corresponding to the step-ca service settings. An example of existing options is provided on step-ca’s documentation ⁶.

  services.step-ca = {
    enable = true;

    # Listening address and port of step CA, overrides settings.address.
    # Here, i am using the tailscale interface. 
    address = "100.64.0.5";
    port = 5050;

    # File containing clear-text password for intermediate key passphrase.
    intermediatePasswordFile = config.age.secrets.step-ca-pwd.path;
    settings = {
      root = "/var/lib/step-ca-data/root/ca.crt";
      crt = "/var/lib/step-ca-data/intermediate/im.crt";
      key = "/var/lib/step-ca-data/intermediate/im.key";
      dnsNames = [
        "ca.net.gq"
      ];

      db = {
        type = "badgerv2";
        dataSource = "/var/lib/step-ca/db";
      };
      
      authority = {
        provisioners = [
          {
            # https://smallstep.com/docs/step-ca/provisioners/#example-3
            type = "ACME";
            name = "acme";
          }
        ];
      };
      tls = {
        cipherSuites = [
          "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256"
          "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"
        ];
        minVersion = 1.2;
        maxVersion = 1.3;
        renegotiation = false;
      };
    };
  };

The CA requires the path to a root certificate (this certificate will be the one to add as trusted on clients). The crt and key fields should point to an intermediate certificate ⁷ and its private key used by step-ca to deliver my certificates. Both certificates can be generated using predefined profiles ⁸. For instance, the following command generates the root certificate (You can have access to the* *step* command using *nix-shell -p step-cli**.):

step certificate create --profile=root-ca "My Root CA" ca.crt ca.key --no-password --insecure 

And this one the intermediate certificate:

step certificate create --profile=intermediate-ca "My Root Intermediate CA" im.crt im.key --ca=./ca.crt --ca-key=./ca.key

In my case, I moved the root and intermediary certificates under /var/lib/step-ca-data/ and modified the permissions to give access to the file to the step-ca user.

Next is the dnsNames entry. The documentation states “comma separated list of DNS name(s) for the CA”. In practice, this means that fields contained here will be added as Subject Alternative Name ⁹ for the server certificate.

Because I want step-ca to act as an ACME provisioner ¹⁰, I need to specify so using the authority.provisioners option. I do not use any custom configuration here, I simply declare the basic fields type and name to enable this provisioner.

Then the tls entry specifies the TLS configuration to use between the clients and the CA. Here, I used the example configuration provided in step-ca’s documentation: specific cypher suites, the authorized TLS versions, and disabled renegotiation. These choices align with current best practices to mitigate known protocol vulnerabilities.

With this configuration, the step-ca can be up and running. For the service to be reachable by other machines I need to set up a reverse proxy, an example using nginx is as follows:

  services.nginx.virtualHosts."ca.net.gq" = {
    forceSSL = true; 
    enableACME = true;

    locations."/" = {
      proxyPass = "https://100.64.0.5:6060";
    };
  };

With the virtual host active, clients can query https://ca.net.gq/acme/acme/directory to retrieve the different provider endpoints in the form of a JSON message:

{
    "newNonce": "https://ca.net.gq/acme/acme/new-nonce",
    "newAccount": "https://ca.net.gq/acme/acme/new-account",
    "newOrder": "https://ca.net.gq/acme/acme/new-order",
    "revokeCert": "https://ca.net.gq/acme/acme/revoke-cert",
    "keyChange": "https://ca.net.gq/acme/acme/key-change"
}

Automatically requesting certificates

With the step-ca server running I can now set up my clients to request certificates from my CA. With NixOS, the security.acme.certs.<name>.server option allows to specify for a virtual host which ACME Directory Resource URI to use. For my virtual host deluge.net.ca I specified my CA as follows:

 security.acme.certs."deluge.net.gq".server = "https://ca.net.gq/acme/acme/directory";

Now, rather than trying to reach out Let’s Encrypt to request a certificate for my internal virtual hosts, the acme client will communicate with my new CA.

However, don’t use this specific URL for your virtual host (here, ca.net.gq). The ca.net.gq virtual host needs an ACME certificate. Before obtaining one, on NixOS, the ACME service creates a preliminary self-signed certificate using minica to allow nginx to start. Then the ACME client would perform a request for ca.net.gq to ca.net.gq. The request would fail as the certificate comes from an invalid CA and nginx keeps using the preliminary certificate. We are facing a chicken-and-egg issue. To solve this, point the ACME client directly at step-ca, bypassing nginx. In my case, that would be:

sercurity.acme.certs."deluge.net.gq".server = "https://100.64.0.5:6060/acme/acme/directory";

Adding a trusted certificate to your clients.

The certificates delivered by Let’s Encrypt are automatically trusted by my browser because Let’s Encrypt’s root certificate is trusted. For instance, here is the list of all the trusted CA by Mozilla Firefox : ¹¹. However, the root certificate I generated is not yet trusted by the machines that will access my internal services. This can be done on NixOS machines using the security.pki.certificates and the content of the ca.crt file I generated earlier attribute as follows:

  security.pki.certificates = [
      ''
        -----BEGIN CERTIFICATE-----
        MIIBdTCCARqgAwIBAgIRAPOAKlBcE/h/LuxFpQeINF4wCgYIKoZIzj0EAwIwGDEW
        MBQGA1UEAxMNZ2FybXIgUm9vdCBDQTAeFw0yNTA3MTQxMDExMzRaFw0zNTA3MTIx
        MDExMzRaMBgxFjAUBgNVBAMTDWdhcm2yIFJvb3QgQ0EwWTATBgcqhkjOPQIBBggq
        hkjOPQMBBwNCAASKpNvqsVINura1WrF9bcj9hwTmKlbLZ2PA2Oc7rCROHCvrjAD5
        0D2TFMi/5jHlLbKM5AoYu/4AMrg+EsxmgULGo0UwQzAOBgNVHQ8BAf8EBAMCAQYw
        EgYDVR0TAQH/BAgwBgEB/wIBATAdBgNVHQ4EFgQUi6Hlc5zrjPuDVJkzcliP4OEI
        hIgwCgYIKoZIzj0EAwIDSQAwRgIhAIquFboD0RZbpfRCmQur2qsw8Bk+d504IyNn
        nA6kaXCXAiEAzJj3anHJZxCNi2UpSMfQKyACd/W7c56y+FcTOjvgPjM=
        -----END CERTIFICATE-----
      ''
    ];

Conclusion

Now I have a working private Certificate Authority that can be used to deliver TLS Certificates for my internal services and have HTTPS exchange with them. While some steps such as the root certificate delivery were not completely declarative, you could inject more development time and use nix snippets available on GitHub to do so ¹².

References

Banner image: François Biard (v1825-1850). Magdalena-Bay, vue prise de la presqu’île des Tombeaux, au nord du Spitzberg; effet d’aurore boréale. Louvre, Paris.