Deploying with NixOS - Site Edition

Today, I managed to make this site deployable via NixOS, as well as simplifying the whole way I manage my NixOS servers. In this post, I want to walk you through what I did and how I did it.

Original flake

My original NixOS flakes were… a mess, to put it lightly. They were an amalgamation of other people’s flakes, and it led to things like my note-taking laptop (which should have xournalpp on it and not much else) somehow running libvirtd and Postgresql. Since I first wrote them, I’ve wanted to modularize my home-manager configuration and allow it to be managed outside of the system configuration.

Modularizing

I’m pretty sure that’s not a word, but it is now. The first step was to break my home-manager configuration out of my NixOS system configuration. This would let me switch around my home without needing to use sudo, but more importantly, it would let me include modules properly based on the system hostname.

To do this, I stole wrote a function based on one by my friend Ellie, hmConfig. It’s similar to mkSystem in that it generates a proper configuration, except that this one is for the specific user. You can see my function here, but it’s nothing too complex. It sets my home directory, username, and home state version. It then imports a baseline ./home/home.nix, which sets up some other things like my shell.

The big thing is one I copied from mkSystem, the ++ extraImports. This is used further down in the flake in homeConfigurations, where I define the specific modules that should be included in this home-manager configuration. For example, my laptop (cesium), needs things like my mail setup, TeX, my X11 configuration, and mpd. The servers I run, however, don’t need anything special. This lets each system be narrowed down to exactly what it needs.

Deploy-RS

Two days ago, I converted two servers (kronos and magnesium) to NixOS. I quickly realized how much of a pain updating these was going to be, as they got out of sync with my flake. I could’ve set up a crontab to automatically apply the latest flake every so often, but that would be too simple (and result in too much waiting).

Instead, I found a project called serokell/deploy-rs. This let me make a two-line script in my flake repository:

#!/usr/bin/env sh

set -e
nix run github:serokell/deploy-rs

Every time I make a change to my flake, I can run this script and have it automatically conform every managed system. It sets up my services, and all the stuff I expect from one of my servers. When more servers come into the NixOS fold, I have to do a few things.

First, I need to apply a basic flake. I tend to use the kronos host, since that’s got nothing special attached to it. Quickly rewriting the installed configuration.nix to enable flakes, I can run something like:

nixos-rebuild switch --flake "git+https://git.carathe.dev/muirrum/nix#kronos"

This will install the kronos flake (which has the unfortunate side effect of making some interesting network decisions and setting the hostname to kronos). In the future, I’ll probably write a generic server configuration that doesn’t do anything special but enable SSH and create my user.

After that, I need to add the server to my deployment configuration. Near the bottom of my flake.nix is a deploy output, that currently has two nodes (the two servers I’m managing this way). It sets up the user that should be used to SSH in as, and the nodes. I can conform this to a different configuration at this time.

Now that this is done, I can move on to automatically configuring my services.

Service flakes

I decided to start with my site, because it’s fairly simple as a service. All it needs to do is parse some markdown and serve it. Should be pretty simple, right?

Haha. This took two to three days of working out bugs in my site flake. The first thing I had to do was configure some options for it and create a system module. In my flake.nix, I wrote the following:

nixosModules.site = { config, lib, ... }: {
    options = {
        cara.services.carasite.enable = lib.mkEnableOption "enable cara's site";
        cara.services.carasite.domain = lib.mkOption {
            type = lib.types.str;
            default = "devcara.com";
        };
        cara.services.carasite.port = lib.mkOption {
            type = lib.types.port;
            default = 3000;
        };
    };
};

This defines a few options that I can use in my generated configuration. Whether or not the site should be enabled (that’s the mkEnableOption call), and the domain and port to use for Nginx.

I prefixed my options with “cara” just to avoid collisions with proper nixpkgs modules (not that carasite is ever going to end up in nixpkgs, but you never know).

Next up was writing the “implementation”, which takes the options and generates the way that the system should look to make the service work. My basic one looks like this:

nixosModules.site = { config, lib, ... }: {
    ...
    
    config = lib.mkIf config.cara.services.carasite.enable {
        users.groups.cara-site = {
            ...
        };
        users.users.cara-site = {
            ...
        };
        
        systemd.services.cara-site = {
            ...
        };
        
        networking.firewall.allowedTCPPorts = [ ... ];
        
        services.nginx = {
            ...
        };
    };
};

I’ll go into more details on what each of those sections do in a bit. Basically, this takes the options I’m going to set in my per-host configuration and turn it into a workable service deployment.

Implementation Details

Users & Groups

Each of these attribute sets configures a different aspect of what makes this site run. First up are the user and group management bits, they create a service user and group that the site will run as.

users.groups.cara-site = {
    members = [ "cara-site" ];
}

users.users.cara-site = {
    createHome = true;
    isSystemUser = true;
    home = "/var/lib/cara-site";
    group = "cara-site";
};

Pretty standard stuff, and described more thoroughly in the NixOS manual.

Systemd

This is definitely the place where I had the most trouble. I was running into a few errors related to the way I had set up my serviceConfig, but that was fixed by properly setting the WorkingDirectory.

systemd.services.cara-site = {
    wantedBy = [ "multi-user.target" ];
    
    environment = {
        PORT = "${toString (config.cara.services.carasite.port)}";
    };
    
    serviceConfig = {
        User = "cara-site";
        Group = "cara-site";
        Restart = "always";
        WorkingDirectory = "${defaultPackage}";
        ExecStart = "${defaultPackage}/bin/carasite";
    };
};

In the end, it’s a pretty simple systemd service. It runs in the store directory where the built site is kept, and runs as the user we configured just a moment ago.

Here we see one of the frustrations of the Nix options system. For some reason, the lib.types.port type can’t be automatically made into a string, as an integer. Even though the whole purpose of ports is to be made into strings and then put into configuration files somehow, you still need the whole ${toString(...)} crap.

Nginx

Nginx was fairly simple. All it took was configuring the ACME settings somewhere else in my system flake and opening the right firewall ports.

networking.firewall.allowedTCPPorts = [ 443 80 ];

services.nginx = {
    enable = true;
    recommendedProxySettings = true;
    recommendedTlsSettings = true;
    
    virtualHosts."${config.cara.services.carasite.domain}" = {
        forceSSL = true;
        enableACME = true;
        
        locations."/" = {
            proxyPass = "http://127.0.0.1:${toString (config.cara.services.carasite.port)}";
        };
    };
};

In the future, I should probably move the 127.0.0.1 to something that lets me configure the bind host, but it’s working for now.

Deploying

Now that the flake is all done, it comes time to deploy it to a server. kronos is my static site host, so that’s where I’ll be putting this.

First, I added it to my system flake’s inputs, like this:

inputs = {
    ...
    
    carasite = {
        url = "git+https://git.carathe.dev/muirrum/site";
        inputs.nixpkgs.follows = "nixpkgs";
    };
};

This tells my flake to include carasite as a dependency and make its version of nixpkgs follow the system flake’s version.

Next, I added it to my mkSystem function:

mkSystem = conf:
    nixpkgs.lib.nixosSystem rec {
        modules = [
            ...
            
            carasite.nixosModules.${system}.site
        ];
        ...
    };

Note that this won’t enable it for all systems, it still needs to be enabled with the cara.services.carasite.enable option we defined in the site flake.

Then, I went into my hosts/kronos/default.nix file, and added the configuration snippet:

cara.services.carasite = {
    enable = true;
    
    domain = "devcara.com";
    port = 3030;
};

This tells the flake to enable the site only on the kronos server, and also explicitly sets the defaults just in case.

Now, the only thing left to do is run the deploy script and watch the magic work!

Conclusion

Wow. This method of deploying software is so much easier and more predictable than any other I’ve tried. I used to have to manually SSH in and update the site whenever something changed, and now I can do it from my laptop.

The added convienence of being able to apply the same basic settings to all my servers without needing to use something like ansible is also helpful.

I think a few of my Discord bots are going to be moved over next. Those should present some added challenges in that they all require PostgreSQL to function properly.