Nix/NixOS: notes and rambles, chapters 1 to ∞

This article was originally dumped on cohost, half finished. This is barely changed.

So, being a longtime Gentoo Linux user, I got kinda tired of the distro like a year or two ago, and while I coasted along for a bit, I decided to install NixOS on a whim. Just, like, install it without thinking too much and figure stuff out as I go along. That was September 2023.

As every NixOS user knows, after some amount of time using it, you have to post random notes and rambles online. Something you had to research and couldn’t find in the docs or whatever. It doesn’t have to be good but you have to post it. This is that post.

Preface

good os

This was going to be the only line here, but events earlier in 2024 did kind of add one major ass caveat: oh no the organization has problems. There was a community letter at https://save-nix-together.org/ and the response was… kind of bad, actually. I don’t want to cover this in detail because it’s not the point of this post. But it has to be acknowledged that the governance situation leaves a lot to desire.

Auxolotl tried to fork Nixpkgs/NixOS but didn’t get enough steam to fork an entire distro, so now it seems to be on the backburner or something. I really do hope they find a way to succeed, if only to create an alternative space, but I’m skeptical; from what I can tell they mostly do meetings. I would absolutely love to be proven wrong here, though.

The “package manager” providing the nix/nix-* toolkit and the like was forked into Lix, which is proving quite successful. The official implementation has been informally dubbed CppNix, and you’ll see this term from time to time. Lix is also upstreamed as alternative Nix implementation, and you can just nix.package = pkgs.lixVersions.stable; it (on nixos-unstable). I think it’s worth doing. For one, it’s had 100% less serious 0-days in the past 48 hours, and it seems to be a good deal faster at stuff, and the devs are cool, and it’s 100% more lesbian.

Resources

As soon as I got started, I noticed there are a million resources online! Great! Surely that is a good thing.

… well. It sort of is. The nice thing about everyone making “notes and rambles” blog posts is you can find very specific stuff. The bad thing is the overarching documentation remains kind of bad.

There’s also a heavy focus on packaging stuff of your own with nixlang. That’s cool and all, you need to learn the language somehow … but, well, my goal was NixOS, and writing configuration tends to be somewhat different. And besides, it’s an OS. An OS is not a package, right? Right. (Foreshadowing for the only part of this post that isn’t bad.)

Eventually, most of the things I learned came from the following sources:

• nix-starter-configs, particularly the standard version. A repo with a template NixOS system as a flake. A reasonably well-documented and well-organized template… but outdated. This isn’t a learning resource on its own, but a surprising amount of things involving flakes are missing the “where”. In this case it’s because it is very obvious once you know where, which is great if you don’t (the answer is “a flake.nix which can technically go anywhere but canonically lives at /etc/nixos/flake.nix). Also the couple lines of nixosConfigurations.hostname = are good to have.
• search.nixos.org. This is where you look for packages, yes - but, on the options tab, it’s also the right place to look for configuration you throw into configuration.nix. That’s autogenerated and more or less complete as a result. (Flakes tab may also be a last resort if you want something not packaged in nixpkgs.)
• The official pieces of documentation on nixos.org & nix.dev:
  • Nix manual’s “builtins” section. Ever see a weird builtins.blah? It’s here.
  • The rest of the Nix reference manual is detailed but it’s a reference manual; it is for reference so you can reference things, not a main learning source
  • The NixOS manual is acceptable for a lot of basics and for things like how to handle partitioning. A bunch of options are documented quite well here, and seeing how esoteric some of the documented stuff is you’d think it would be complete, but… no. It’s nowhere near as comprehensive as the options search
  • Nix pills is old. The chapters about installing Nix are a bit dated, it’s not flakes-first, but, the stuff about the actual language is good. For better or for worse, it’s written informally. It’s a good place for pieces to click together.
  • nixpkgs manual. You usually want the lib. documentation under that tiny little “Nixpkgs lib > Functions reference” section.
  • There’s two wikis, one official one unofficial, and it’s complicated. Neither are good, but both sometimes contain arcane magic spells you may need, which someone graciously dumped on there after spending hours trying to figure something specific out.
• The Arch and Gentoo wikis remain surprisingly relevant in a lot of ways when it comes to things like configuring individual parts of any program. You’re generally not gonna be touching /etc directly on NixOS, but it can give you an idea of which toggles to flip to achieve whatever you’re after, and that’s always relevant information.
• The NixOS & flakes book. There is so much good stuff here. This wants to be the one-stop shop for everything, it’s not quite there, but hey.
• Julia Evans’ notes on nix flakes and notes on NixOS. Yep it’s notes alright

Flakes: nobody needs another attempt to explain it, but I decide to write anyway

Flakes: the nix/“nix3” command line tool

Aside from that, there is also a reworked nix commandline tool for all the other stuff. This seems to be called “nix3” in some places. And unlike most of the ‘old’ tools, those tools under the nix command are flake-aware. So generally, you use those and discard many of the separate nix-thing binaries:

• You don’t use nix-channel at all, everything you’d think of doing with it should be an input in your flake.nix and/or be updated via nix flake update. There’s even an option to disable this tool entirely, and some options to manage channels via flakes instead. This comes with some not fully documented implications like the default command-not-found handler breaking (it’s ok you want nix-index anyway), but it’s solvable
• nix-env still has some uses, but it’s an increasingly niche tool. nix-env --install/nix-env -iA are pretty much out entirely.
• nix-build, nix-instantiate and nix-shell are probably disused, unless you’re using them on non-flake stuff
• nix-collect-garbage, nix-store, nix-copy-closure, nix-hash, nix-info and nix-prefetch-url are still useful
• nixos-rebuild is flake-aware and stands by itself. you use it all the same, just with --flake added

So what’s the new stuff on a NixOS flake system?

so, you don’t use most of the old stuff. What DO you use?

• nix flake has some subcommands to handle a flake. Update it to the latest version of your current branch, check it for correctness, and more
• nix fmt can run an automatic formatter over your whole flake, if configured with something like formatter.x86_64-linux = nixpkgs.legacyPackages.x86_64-linux.nixfmt-rfc-style; in the flake.nix .
• If you want to run a program “without installation”, nix run nixpkgs#hyfetch . As soon as you finish running it, it goes “poof”³
• If you want to search in the packages without using the online search, nix search nixpkgs hyfetch will slowly do the trick while spitting 100 ignorable warnings at you, and then the next time it’s in cache and it won’t do the warnings thing.
• If you want a shell with a specific program or set of programs available in the path, nix shell nixpkgs#hyfetch nixpkgs#cowsay. You can specify as much programs as you want, and as soon as you close the shell, they go “poof”
• If you want to build a program, nix build nixpkgs#cowsay . You get a symlink called result in your current directory, pointing to the output, which is usually a directory with a bin directory and the like.
• If you want a shell ready to manually build a program, like say you’re working on a patched version of something, nix develop nixpkgs#ripgrep gets you most of the way. Suddenly, you’ll have access to the very same rust compiler that would be used for ripgrep.
• If you wanna try out some very cool one-liner, nix repl gets you a shell, or nix eval runs some commandline stuff or a .nix file of your choosing
• If you want to install packages without editing a config file, nix profile exists but is usually a mistake. It doesn’t get automatically synchronized with anything.
• nix store has a lot of subcommands that do neat stuff. Some overlap with nix-store, some don’t. Not really needed on a single user system.

But what is this nixpkgs#?

It’s a path. Every flake reference with a # in it is a path followed by a package name or the like.

Or, to be more accurate, it’s an installable. Specifically a “flake output attribute”. This is in a section of the manual that a new user is very unlikely to find, yet it is completely central to doing anything on a flakes system.

If you’re running something from a flake in the current directory, this means references to it start with .# . As much as . being the current directory is ingrained in me, that .# confused me to no end for over a hour. I grew up with a discarded MS-DOS 3.3 computer for fuck’s sake. I should have known that it wasn’t in fact some magical Nix syntax but just a path on disk.

But we were talking about nixpkgs#. Well, no, that’s not a path. It’s an alias as listed in the flake registry! And nixpkgs is one of those! … right?

If you make your own flake, you can put the full path to that flake there and use it all the same. nix run https://git.whatever.invalid.example/someusername/mycoolrepo.git#ribbit could do everything you’d ever want to! It’s just annoyingly long unless you add it to your local flake registry; if it’s your own flake, that’s easily done with something like nix.registry = { ribbit.flake = inputs.self; };. But in general, just try not to overdo the whole “running untrusted code off the internet” thing; this is only barely different from curl | bash or similar.

The flake registry has a special nixpkgs# twist

Now, there is one extra important twist to this: NixOS 24.05 includes a bunch of cool new configuration options, namely nixpkgs.flake.setFlakeRegistry (and nixpkgs.flake.setNixPath), which will make all of these nixpkgs# invocations point to the current system-wide nixpkgs version. And it’s set by default. This is good, since staying on your system nixpkgs generally means less downloading and better compatibility. In practice, this means that nix run nixpkgs#cowsay will stay in sync with the NixOS version on your current system if and only if you are running NixOS 24.05+. Otherwise, the version in use may differ on every invocation!

This is something to keep in mind if you also use Nix outside of NixOS somewhere, since there you are not going to get that pinning behavior. This is particularly important if you run NixOS 24.05, since nixpkgs# defaults to unstable elsewhere. At times you may get different node.js major versions and the like.

The flake registry has a special Lix twist

If you’re running Lix, then note that the flake registry is no longer an online resource but instead a locked-in-time copy included with Lix. The default CppNix fetches it from an online URL fairly regularly, which means it’s prone to change, which is pretty much counter to the whole “if it works now it will work 5 years from now” thing Nix tries to achieve.

Non-Nix binary compatibility (like Itch games)

This is the part I feel like i’ve written a comment about a dozen times. Yes, NixOS is somewhat bad at non-Nix compatibility… but actually, no, it’s gotten pretty decent nowadays! Packaging is no longer required for a lot of use cases.

Do you have a single binary you want to run, most particularly one that doesn’t use OpenGL/Vulkan? nix-alien probably got you covered. This will cover most random stuff, like Node.js downloading a random copy of esbuild or whatever. Games are harder.

You want a traditional-ish system as an extra? nix-ld is a few configuration options away. This requires manual configuration.

Finally, there’s the steam-run package, part of nixpkgs. They figured out all the stuff Steam needs, and as it turns out, the needs of Steam and the needs of “pretty much any generic proprietary Linux desktop game/app” have significant overlap. Not much options if you need more, though.

And of course, you can make your own FHS env… which is kinda bad if it’s the first thing you wanna do. It’s not that much work if you already know exactly how to do it. It is a lot of work if you don’t. There are probably examples online.

staging-next is a conspiracy by big everything to sell more everything

One thing I was unprepared for: the “staging-next” updates. By its method of operation, Nix has to rebuild absolutely everything if something below it changes. glibc or gcc get updated to fix a minor bug occurring only on leap days on ARM systems or whatever? Full rebuild of everything.

Of course, you’re not building the bulk of the stuff locally in most cases - you’re downloading prebuilt binaries from the cache server. The build is not your concern, unless you want to custom patch one of those packages systemwide. If you do, though, you are about to enter the Gentoo realm.

The download and disk use might be a problem if you’re on a less great connection or are very low on storage. A decent desktop with a couple of dev tools can be easily ~7GB+ worth of download, and even more disk space; you’ll temporarily end up with two or more copies of everything during the install of such a rebuild. My /nix/store is 17GB with just one. Of course, you can nix-collect-garbage -d the old version of the OS away if everything still works after a reboot, not all staging-next merges are full rebuilds, and I have a bunch of dev stuff installed.

So anyway, they are aware there is a cost to those kinds of updates, so they get batched; big rebuilds have to go to a staging branch and linger there for a couple of weeks until the maintainers decide it’s time to merge it. Then they send it to a branch for the build servers to catch up first. This process happens every ~3 weeks or so. (the process description doesn’t really mention any particular cadence and it seems to be somewhat on an “as-required” basis)

This also sucks if there’s an important update to a system level library that needs fixing immediately. You end up waiting for hydra to rebuild the entire OS from scratch, and just hope that the maintainers decide it’s important enough for some kind of speedy push. The xz situation took like a week to be resolved.

Anyway, basically: if you’re on a slow connection, or one with a tight data cap, NixOS might be a bad fit. If you run multiple similar systems on a connection that’s pushing it, look into setting up a local binary cache for that kinda thing.

The epic tale of the garbage collector and the data hoarder that wants to stop hoarding data for just a brief moment

Or: “why do I still have /nix/store/*-xz.5.6.1-* when all the tooling says nothing depends on it. Actually why I do have like, 20 different copies of xz? And a lot of compilers too. Strange. wow some are from 6 months ago why isn’t it removing them. OK sure you’re not running them but I just want them gone y’know”

Normally, nix-collect-garbage will remove things like compilers and -dev outputs that you may want to keep once downloaded despite not truly installing them. For this reason, developerish people tend to set a combination of nix.settings.keep-outputs = true and nix.settings.keep-derivations = true . Like it’s straight up in the manual as an example of how to set options. If you’re a developer that wants to occasionally patch a .nix package, but doesn’t want to install all the build tooling systemwide for regular interactive use, this is probably a good idea.

But, if you choose to set those options, you should probably place a comment reminding you to run something like this on irregular intervals and/or when something ended up in the store that you strongly desire disposal of:

nix-collect-garbage --option keep-derivations false --option keep-outputs false -d

My first run of this particular version freed up a huge amount of disk space, removing remnants from NixOS 23.05 and NixOS 23.11, too. Quite a lot of garbage can stick around.

The nixos-system-hostname-bla derivation and /run/current-system: what’s a NixOS anyway?

Over at /run/current-system and /run/booted-system lie two very important and often identical symlinks: symlinks to your system’s toplevel, usually named something like nixos-system-funnyhostname-24.11.20241231.abcdef0. There’s a lot of fun stuff in there!

In fact, that is a derivation; a package. A package that contains your entire system. Of course, most everything there is a symlink or hardlink to other stuff in the store, so it’s small (just a couple MB of filesystem metadata, I guess…).

So, what is a NixOS system?

• A couple of scripts to activate, init and bin/switch-to-configuration the system. To be honest I’m not entirely sure what the exact split is on what they do, but these set up all kinds of your configuration and the environment to match. Some populate /etc, others just set environment variables. It’s very rare to use these manually, but it’s good that there’s an unchanging path for it, I guess.
• An etc directory, where all the stuff that nix ends up throwing into your /etc lives, as handled by one of the above shell scripts. The real /etc in NixOS isn’t truly read-only despite most of the contents being explicitly managed and overwritten should you try to change it yourself, but that skeleton one is very much read only.
• There’s sw, sometimes also called the system-path. That, too, is a derivation/package, consisting of more or less everything from environment.systemPackages symlinked together. The end result is that the /run/current-system/sw/bin directory is about as close to any other distro’s /usr/bin as you get, and likewise for the other ones. This is how interactive shells in Nix aren’t terrible. This can also be used in “hacky” ways, and sometimes that happens in Nixpkgs because the alternatives are worse (for example, autostart files should reference that path).
• A boot.json file, describing where to find the kernel and initrd files, though they’re also just sitting there. Of course, these are really written to eventually be consumed by your bootloader, or rather by scripts that set up that bootloader. nixos-rebuild runs all of those for you.
• There’s the firmware and extra kernel-modules for your kernel. I guess this is why there’s a booted-system; you’d still need to use the stuff for your old kernel until you reboot, given Linux cannot reliably load modules for a different kernel version.
• And a couple of other files

Closely related is /run/opengl-driver; this, too, is a symlink. It’s where your OpenGL and Vulkan drivers live. This feels like it should be in the current-system dir but I guess there’s a reason it isn’t. (I don’t know it but I suspect Nvidia is to blame.)

But anyway, this is what NixOS really is: the target directory of that symlink. That’s everything! It’s a derivation - a package. Sure, it’s made out of other packages, but what isn’t? Really it’s weird an OS takes so much work to make. mkdir -p /run/current-system there I made a Linux distro. Only took me 5 seconds.

This realization made a lot of things click for me. In Nix terms, nothing is super special about NixOS. It’s an assembly of a bunch of packages that are in itself assemblies of a bunch of packages and files. No, really.

# nix repl
Lix 2.91.0
Type :? for help.
nix-repl> :load-flake /etc/nixos
warning: Git tree '/etc/nixos' is dirty
Added 16 variables.

nix-repl> outputs.nixosConfigurations.mycoolhostname.config.system.build.toplevel
«derivation /nix/store/aaabbbcccdddeeefffi2rmzv9b4x4y61-nixos-system-mycoolhostname-24.11.20241231.9f4128e.drv»

That one file describes The Whole Fucking System. Sure, it’s not actually the whole thing, since it depends on many many other .drv files that you can’t independently rebuild with just that one, but as long as you have a tool that takes such a file and copies over all of its dependencies over too, it’ll work. nix-copy-closure sits around to do that, and that’s how pretty much anything involving remote building, remote deploy and the like works. As long as you don’t set it as your /run/current-system, you can build for any number of other systems as you’d like.

There’s one more thing: the “generation” system where it keeps old versions of your system. Well, those are just old-school profiles. /nix/var/nix/profiles/system* has all of them, and nix-collect-garbage -d will remove old ones. They’re all symlinks to a toplevel.

So, Nix is all kinds of complex, but there isn’t “a packaging system” and “a general configuring system” and “a /etc managing system” that happen to share a language. It’s the same system and you’re just creating a set of hella complicated packages for /etc, your system-path and a few others. That’s something that wasn’t that obvious to me.