Table of Contents
Working on NixOS hosts / Core services
Our 'core services' (the stuff we're more scared of changing), are mostly managed with Nix - a functional programming language, build system, configuration tool, and hard disk stress tester wrapped into one.
A lot of this is specific to Tardis core services nix, and is mostly geared towards people needing to make superficial changes, that probably haven't learnt Nix.
Our core services is setup to deploy whatever's on
main to all of our machines. This means if you know exactly what you need to change, you can probably avoid doing any actual building or deploying locally.
With that said, you should still use a syntax checker / linter. You can download alejandra, our linter. Make your changes, then run
alejandra . to check syntax, and format it. Double check your changes to make sure you haven't completely changed the meaning of the file - if the indentation is the same everywhere else you're probably fine.
Here is a small cheatsheet for the nix language - alternatively, just look at the parts around what you're editing. Here are some common files you might need to edit:
- Email domains (see here for full steps)
You won't be able to merge directly into main, for hopefully obvious reasons. Make a branch and an MR, then ask someone else to approve it.
If you're doing a lot of work and want to deploy and redeploy without pushing to CI, you'll need to do some local setup. Fortunately - you don't need a lot of disk space (all of our hosts take up about 11G). We can do all the actual building remotely, and only do some local config to make Nix forward things properly.
You'll need something running *nix - either a VM or WSL. MacOS (Darwin) should work, but we haven't tested it.
We recommend the determinate systems nix installer, which can be run as simple as:
curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install
It gives you basically the same setup as the official Nix installer (which you can use if you have any problems), but it enables some extra commands, and especially flakes.
If you do use the official Nix installer, add
extra-experimental-features = flakes nix-command so you can use these extra features. Don't worry - they're barely experimental anymore and are pretty widely deployed.
Remote building works like this:
- Upload your sources to our build server
- Have it build it
- When you deploy, everything gets downloaded from the build server
For packages we're not building (most packages that we haven't modified), we want the build server to download it directly from Nix's cache, as opposed to the default where you download it then copy to the build server.
To fix this, add
builders-use-substitutes = true to
Lastly, we need root to have passwordless ssh to
nixbuild.internal.tardisproject.uk. This involves proxying through our SSH bastion. You can find more details here, but you'll need to add something like this to
Host tardis User USERNAME IdentityFile /path/to/your/ssh_key HostName tardisproject.uk Host nixbuild.internal.tardisproject.uk User root HostName nixbuild.internal.tardisproject.uk ProxyJump tardis IdentityFile /path/to/tardis/admin/key
This page has more info on Nix build servers, and how they're used.
Finally, for deployment, you'll need to be able to ssh (from your user) into
<hostname>.tardisproject.uk without any password. This is covered here.
If you've gotten this far, test everything with these commands:
# Test using the remote build server nix store ping --store ssh://nixbuild.internal.tardisproject.uk # Test passwordless SSH to root@ wherever youre deploying ssh web.tardisproject.uk
Doing the thing
Now that you're all setup, you can use the 3 scripts in our repo for most tasks:
scripts/eval.sh <hostname>evaluates a given hostname and does some basic checks like that options exist, etc. Nothing gets built, and this doesnt even require remote building.
scripts/remote_build_deploy.sh <hostname>builds and deploys the given hostname - all remotely. It has auto rollback if any services fail to start, but you should still be careful. Add
–dry-runif you want to just build everything, but not deploy it yet.
scripts/needs_rebuild.sh <hostname>tries to tell you if the given hostname needs rebuilt. It's mostly used in CI, and may be broken rn.
And some routine Nix tasks:
nix flake update- Update all inputs
nix flake lock –update-input <name>- Update a specific input, ie