The one thing I do not like about the Nix package manager (and a fix for it)

The nix package manager is an awesome package manager for linux and macos, which focuses on declarative packages. This means that you can dump out all the packages you want into a file, and nix will go out and fetch them for you.

This package manager builds itself on the concept of reproducibility, and it boasts a collection of over 80,000 packages, second only to the AUR! But this blog post is not about why nix-os is great, you'll find many other blog posts and videos if you want to learn why.

Many nix package definitions include compiling from source. This can be tedious, since compiling takes a long time and wastes more bandwidth downloading build dependencies. Hence, the nix team has a binary cache, in which they build binaries for all the systems that nix supports for most of the common packages out there, like web browsers, desktop environments, and many more.

But what if your desired package is not in the binary cache? And what if it takes a long time to compile? This is what happened to me, and this is the one thing I don't like about nix.

The AUR provides binary packages, alongside source packages, for example, yay and yay-bin, but nix only provides source packages for most of its packages. Granted you'll see some exceptions, like firefox and firefox-bin, but those are pretty rare.

The fix

In this article, I'll show you how you can create a binary package for your desired program. I wanted to download the SurrealDB package, but the package on nix was a source package, meaning that I had to spend over 50 minutes waiting for a stupid package to compile.

Surreal provides binaries over at its GitHub releases, which I could've downloaded and ran, but I'd have to manually update the package, and as a 10x developer (I'm not one), I'd never manually do anything, but spend 10x the time trying to automate it. Doing this also defeats the reproducibility of nix, hence it's better to create a nix package so that anyone would be able to download your dependencies without having to do anything extra.

Getting set up

First, make sure you have in hand a name for your package, and its version. Generally, binary packages end with -bin, so I'll call my package surrealdb-bin, and I'll be downloading the binary for the latest version as of the time of writing, which is v1.1.0.

Since I use NixOS, and since I want to install surrealdb globally, I'll create my surrealdb package in /etc/nixos, which is the folder which holds my configuration.nix. You may choose to co-locate your package in the directory with your flake.nix, for example, or even push it to GitHub/Lab so you can make your nix config truly reproducible.

This approach will contain two nix files, one for the package itself, and another for an overlay. An overlay gives you the ability to modify nixpkgs without having to publish your package to the nixpkgs repository, and without having to mess around with inputs. Overlays make it very easy to use your package.

Creating the package

Generally, I put my custom packages in the packages/ subfolder, and custom overlays in the overlays/ subfolder, and I name both these files with the name of the package, in this case surrealdb-bin.nix. I'd recommend you follow the same strategy to avoid clutter, but if you're only going to create one package, you can just keep everything in one directory.

I shall refer to packages/surrealdb-bin.nix as the package declaration. Now, put this code in your package declaration:

{ stdenv, fetchzip, autoPatchelfHook, glibc, gcc-unwrapped }: stdenv.mkDerivation rec {
  pname = "PACKAGE_NAME";
  version = "PACKAGE_VERSION";

  src = fetchzip {
    url = "PATH_TO_YOUR_PACKAGE'S_TARBALL_HERE";
    hash = "A_HASH_OF_YOUR_PACKAGE'S_CONTENTS";
  };

  nativeBuildInputs = [ autoPatchelfHook ];
  buildInputs = [
    glibc gcc-unwrapped
    # Any other system binaries your app may need
  ];

  installPhase = ''
    runHook preInstall
    mkdir -p $out/bin
    install -m755 PACKAGE_BINARY_FILE_NAME $out/bin
    runHook postInstall
  '';
}

Let's go over this code step-by-step. If you're familiar with nix syntax, you'll recognise that we're creating a function which destructures its first argument to accept some fields, and returns the output of the function stdenv.mkDerivation. mkDerivation is the helper function that you use to define nix packages. We give it an argument which defines the basic metadata of our package. Here are the fields it defines:

• pname: This is the name of the package. You could also use name, but you'll have to also specify the package's version in the name. Using pname makes nix automatically generate the name for us.

• version: This is the version of your package. Make sure you're fetching the correct binary!

• src: Now here's the meat of the package. We use nix's fetchers to fetch our package's binary from the internet. There are two mainly used fetchers for binary packages: fetchurl and fetchzip. Let's take a closer look at them.

fetchurl: This fetcher directly downloads the file provided to it by the url field.

fetchzip: This file downloads the archive provided to it by the url field, and unarchives it. The fetcher may be named fetchzip, but it also works for other archives like .tar.gz.

I'll be using the fetchzip fetcher to download the tarball of the correct SurrealDB version using a direct link pointing to its GitHub release. You can use nix's string interpolation to generate a proper link. This is what mine would look like:

https://github.com/surrealdb/surrealdb/releases/download/v${version}/surreal-v${version}.linux-amd64.tgz

I'll replace PATH_TO_YOUR_PACKAGE'S_TARBALL_HERE with the above link.

Now for the hash field. You must specify a checksum hash for the binary that you're downloading so that nix can verify that the correct file is being downloaded. The easiest way to set this value is to set hash to something random, install the package, and set the hash to whatever it says in the error that gets thrown.

If you'd like to write hash yourself, the syntax for it will be:

hashingAlgorithm-base64encodedhash

Many hashing algorithms are supported, but the most commonly used ones are md5 and sha256. Make sure to base64 encode your hash value, since nix only accepts that! Don't give it a hex string.

Now let's get back to our package derivation. The rest of the code is instructing patchelf to automatically patch the binary to make it run with nix. Since nix doesn't work like other package managers, binaries which expect shared libraries to be at one particular location will not work, hence we need to use patchelf to update these locations. Thankfully we won't have to manually run patchelf, since nix provides us with the autoPatchelf package. This package is defined in the nativeBuildInputs.

The additional libraries which the package depends on must be specified in the buildInputs array. The best way to find this out, would again be to install the package and add the dependencies it lists out in the error, but generally the two packages I've included should suffice.

Finally, in the installPhase, we define a shell script that runs. The hooks that are called are all related to patchelf, so just make sure they're present in your code. In between the two hook calls, we write code to create a bin/ directory in our package's nix-store path, and we transfer any binaries the package provides to the bin/ folder. Do update the PACKAGE_BINARY_FILE_NAME variable with the name of the binary that gets downloaded by the fetcher. In my case, that'd be surreal.

The install command is actually just some syntactic sugar for the cp command, they both have the same function. The only extra thing is the -m flag, which sets chmod permissions on the binary to make it readable, writable and executable by the user (the 7), and only readable and executable by the user's group and everyone else (the two 5s).

Finally, this is how your package derivation should look like:

# /etc/nixos/packages/surrealdb-bin.nix 
{ stdenv, fetchzip, autoPatchelfHook, glibc, gcc-unwrapped }: stdenv.mkDerivation rec {
  pname = "surrealdb-bin";
  version = "1.1.0";

  src = fetchzip {
    url = "https://github.com/surrealdb/surrealdb/releases/download/v${version}/surreal-v${version}.linux-amd64.tgz";
    sha256 = "2611de5eb7779dfe3b32bb47833fee2e3e168e39e43d76b47ea649b2f8c407fa";
  };

  nativeBuildInputs = [ autoPatchelfHook ];
  buildInputs = [ glibc gcc-unwrapped ];

  installPhase = ''
    runHook preInstall
    mkdir -p $out/bin
    install -m755 surreal $out/bin
    runHook postInstall
  '';
}

Creating the overlay

Now we need to create an overlay to be able to add our package to the existing list of nixpkgs. I'll call my overlay derivation surrealdb-bin.nix, and place it in the overlays/ folder in the /ext/nixos directory.

If you're using a different path than me, be sure to update the relevant imports.

Add these lines to your overlay definition:

self: super: {
  surrealdb-bin = super.callPackage ../packages/surrealdb-bin.nix {};
}

Now this may seem a little scarier, if you're new to the nix language. Since nix functions can only accept one argument, we use nested functions to declare multiple arguments. self and super, as they're commonly called, or more recently final and prev, are both instances of nixpkgs. It's just that super/prev is the version of nixpkgs before the overlay is applied, and self/final is the version of nixpkgs after all overlays are applied.

You should generally only use the super argument. Read the wiki if you want to learn more about overlays.

The overlay functions return type is a set of packages which will be merged into nixpkgs. Here you can see that I'm defining one package called surrealdb-bin, and I'm calling the super.callPackage function and giving it the location of my surrealdb-bin package derivation as the argument. The callPackage function ensures that the package derivation is called with the proper arguments supplied. The blank argument set at the end is just to specify that I don't want to extend the list of arguments passed to the package derivation any further.

Using the overlay

The only thing left to do now, is to actually use the overlay and extend the nixpkgs definition. Now this depends on where you plan to use the overlay, i.e. in a shell.nix, flake.nix, the nixos configuration.nix, or in a home-manager configuration. The steps are different for all of those.

Check the wiki for instructions pertaining to your specific use case. Since I want to install surreal globally, I'll add the overlay to my configuration.nix. All I have to do is add the below line somewhere, and then add surrealdb-bin to the list of environment/user packages.

nixpkgs.overlays = [ (import ./overlays/surrealdb-bin.nix) ];

Don't forget to update the path to the overlay if you're using a different path!

And with a small nixos-rebuild switch, I have access to the surreal command in my environment! Great success! Now if there's a new surreal version published, all I have to do is change the version in my package derivation and rebuild!

You should push your package derivation and overlay to a centralised place like GitHub, GitLab, or any other place that provides direct download links (GitHub Gist / GitLab snippets would be a great place), so that you can download this overlay in any nix configuration and maintain reproducibility, instead of possibly having to duplicate your overlay and maintain two sources of truths.

What if I don't have access to binaries

If your package only has the source available, with no binaries, and still takes a long time to compile, then you should probably leave the compilation to a GitHub action.

Just create a GitHub repository, create an action to build the package, and then upload it as an artifact/release once done.

Then use Nix to fetch that artifact in your package.

Do note that GitHub-provided hosted action runners have a maximum runtime of 6 hours, so if you have a package that takes longer than that (e.g. a web browser), then you need to use self-hosted runners, which have a max runtime of 35 days!

It is risky to host a self-hosted runner on your own machine, since your machine may go down, and you'll have to restart the build process again

Instead it'd be best to use a cloud VPS service like Digital Ocean. If you use that link, or this one, you can get $200 in credit for upto 60 days, that's literally giving you free access to a 16GiB ram + 8 vCPU instance to build all your hopes and dreams on! (You will have to request access to these machines first though).

If you use my link, I also get some credits, so it helps me out too! Thanks for using my link.

Anyway, custom GitHub actions are out of scope of this article, but do let me know if you want me to create another article on that topic in the comments below!