This is a tutorial and an ad for my favorite software packaging tool, “nix”, which is awesome and which more people should know about.
In this context, nix refers to:
nix tool suiteNix has the highest power-to-weight ratio of any software package management system I have ever used.
Nix is the product of a multi-year European research project which produced several PhDs, including Eelco Dolstra, whose dissertation identified the abstractions underlying nix.
Nix solves hard problems in the field of “running software” better than alternatives.
Nix helps many audiences including solo developers, small development teams, open source communities, and large-scale engineering organizations to converge on common task-specific software toolchains at an affordable price, and with nice desiderata around reproducibility and supply-chain integrity.
In my opinion, the best way to learn nix is to:
nix repl to evaluate nix
expressionsnix repl to explore
nixpkgs and other nix flakes.We’ve already addressed (1) above, so now let’s turn to (2).
The most fundamental idea behind nix is that because almost all software is built by combining small pieces into larger ones, it might be handy to have a language specifically for expressing these combinations that works at all levels of scale: all the way from compilers, linkers, and assemblers at the bottom up to tools for scheduling logical networks on to physical hardware at the top.
The next most fundamental idea behind nix is that the recipes for describing how to combine inputs to make outputs should be expressed as functions, with inputs provided as function arguments and with outputs arising from evaluating function application.
Finally, the last really important idea behind nix is that these recipes should be abstracted over where the inputs and outputs are actually stored until the last possible moment when the time comes to actually building the outputs because this way, we can conveniently solve a whole pile of other software packaging and operations problems just by ensuring that novel combinations of inputs always (and preferably deterministically) give rise to novel outputs, typically via some simple cryptography.
Now let’s turn to (3), “learn, via tutorials, to read the nix expression language”, first by discussing the purpose of the nix expression language and then by discussing how to read nix expressions heuristically.
As discussed immediately above, the nix expression language, colloquially “nix”, is a programming language designed to make it easy to define recipes that express how to combine and process inputs to produce outputs.
Like most programming languages, it has types and syntax for common data structures like strings, numbers, lists, dictionaries (called “attrsets”, in nix), and functions.
Unlike most other programming languages, nix-the-language is specialized for working with files that need to be transformed by external programs (often compilers, linkers, etc) to produce directories full of output files, often in target-architecture-specific ways. That’s what makes it so useful defining “how to build programs”, including at the scale of an entire operating system.
The way that nix is specialized for working with files is that, in
addition to the common kinds of data structures and computations listed
above, nix also includes support for a special kind of computational
value, called a “derivation”, that records how to transform input file
system structures to output file system structures, typically by running
a program (usually a bash shell script) with environment
variables initialized to strings that denote the file-system locations
that are intended to contain the derivation’s “inputs” and
“outputs”.
Thus, the fundamental ideas behind nix so far:
To package software at scale, we need a way to refer to recipes for how to combine inputs to make outputs. Nix doctrine says that we should define “recipes” to be functions that evaluate to derivations or to data structures containing them and ii) that we should “refer” to recipes by i) defining an expression language for expressing recipes and ii) in that language, binding variables to recipes (or to their outputs) as needed.
At the lowest level, “recipes” should be represented by a choice of a program to run to turn inputs into outputs + the paths that the inputs and outputs should be stored in.
(secretly): good things will happen if we also arrange for the nix evaluator to (cryptographically) incorporate provenance information into the actual file/directory-names that go into those input/output storage paths.
Here are some tips for reading nix expressions (nixexprs, or “nix exprs”) heuristically. Warning: this set of tips is not, and is not intended to be, a complete algorithm. In fact, the actual nix expression syntax is more complex than is presented below; however, hopefully, these simplifications will be useful in helping you get started reading nix expressions of practical relevance to you.
With that, here are the most important rules that I use to make sense of nix-exprs that I’m reading or skimming:
Nix exprs can be read left-to-right.
{ tokens mean “attrset” or “attrset pattern”, and
need to be paired with a matching } token to close the
attrset or attrset-pattern. Attrset-entries, called “attributes” or
attribute pairs, are terminated by ; tokens. Attrset
patterns bind variables and are separated by , tokens. In
nix, attrsets are the fundamental tool for naming values (including
functions), and they work a lot like JSON or Javascript dictionaries
wherein attributes have keys and values, the keys are usually strings,
and the values can have heterogenous types.
: tokens mean “function”. All functions in nix take
a single argument, but that argument can have complex internal structure
that can be pattern-matched by the function. Argument patterns go to the
left of the :. All functions in nix have a body that
consists of a single expression, written to the right of the
: function introduction token. As described earlier,
functions are the primary means of abstracting recipes for building
outputs over their inputs, now represented as function
arguments.
[ characters mean “list” and, like attrsets, must be
closed by a matching ]. List entries are
space-separated.
Function application is written as juxtaposition, separated by a space. Parentheses may be used to control order-of-application; otherwise, function application associates to the left.
. tokens occur in three contexts of which the third
is by far the most important: in floating-point literals (1.0), in
path-literals (./., /.), and to define
“attribute lookup paths” (a.b.c) which are used both for
reading and for writing the indicated attributes.
${ tokens in string contexts are called “splices” or
“interpolations”, and need to be closed with matching }
tokens or need to be escaped. Splices are instructions to evaluate the
enclosed expression. Splices can be used in attribute lookup paths, like
so: a."${someFunction someArgument}".c to look up
dynamically selected attributes in a convenient way.
Nix has syntax for both inline and multi-line strings, introduced
by enclosing " or '' tokens, respectively,
with the usual in-line character escaping syntax. Multi-line strings
strip leading indentation, which is handy for generating config files
for typical Unix-style programs.
Nix has additional syntax for managing variable-to-value bindings
and scopes/environments. The main pieces of syntax are:
let BINDINGS... in BODY, with ATTRSET; BODY,
and the rec token for controlling the visibility of sibling
bindings introduced in let and attrset
contexts in the bodies of other sibling bindings; i.e., for making the
sibling bindings visible, which is called making the bindings “mutually
recursive”, hence the rec keyword.
Nix has syntax for referring to file system paths, both absolute as well as relative to the file containing the nix-expr currently being evaluated. Note: at evaluation time, path expressions are treated specially by the nix evaluator, which normally copies any files referred to by path expressions to the nix store, which can have big effects on performance.
Nix has syntax for “importing” and evaluating nix-exprs from
files. Nix-exprs can be imported from static or dynamically calculated
paths. Additionally, imported expressions that evaluate to functions can
be immediately applied to arguments; hence an expression like
import ./f.nix {} implicitly encodes an expectation that
the expression contained in f.nix will evaluate to a
function that can be applied to an attrset argument (which, here, is
empty).
The typical nix evaluator, provided by github.com/nixos/nix, provides many additional builtin operations including some quite complex operations for fetching source code from upstream version control systems, for fetching packages of nix expressions called “flakes”, and for providing runtime services like profiling, debugging, and runtime reflection (some of which are used extensively by the nixpkgs standard library for key functionality like automatic dependency injection based on nixpkgs-provided package definitions.
The nixpkgs standard library, referred to above, defined a broad
and deep range of additional abstractions which are commonly used by
people using nix to package software which, to progress, you will
eventually need to learn. These additional abstractions are often
documented in the nixpkgs manual, in
addition to the underlying language, which is mostly documented in the
separate but adjacent nix language/tooling manual (stable, unstable). Some key
abstractions as of this writing: stdenv.mkDerivation,
callPackage/callPackageWith,
newScope', list/attrset/string manipulation functions, file
reading and writing functions, and the various “overrides” mechanisms
like overrideAttrs, override, and
overrideDerivation.
nix repl to evaluate nix
expressionsNix expressions are made of simple components that interact in meaningful ways to capture the enormous variety and complexity of real software, e.g., an entire Linux distribution + various proprietary company-specific layers, many packages of which can also be built from the same recipes but with different ingredients on other systems of interest like developer laptops running macOS, and coping with this complexity requires tools.
The most important of these tools, in my opinion, are three things [two of which we will discuss later]:
To be set up to use the nix repl successfully, you’ll need to accomplish at least the following on your own, perhaps aided by the indicated resources.
You need a computer that you can install nix on. Typically, you either own this computer outright yourself and so can run what you want to on it, you’ll rent this computer from a cloud vendor, or you’ll borrow this computer from someone nearby with the understanding that you’re going to experiment on it, and that it will likely need to be reset to factory defaults / reimaged after you’re done with it.
If at all possible, it’s best if you either already are, or are willing to become, comfortable with “the command line”, or with “CLI interfaces” more broadly. Note: if this is new for you, or something you’re still working on, https://github.com/nuitrcs/commandlineworkshop looks from casual inspection like it may be a helpful place to get started.
Finally, you’ll need to install nix on your computer if it’s not already installed and usable. For getting-started directions here, please see nix.dev.
To test whether you’ve got things set up well enough to continue, please
nix repl, to start the Nix repl$ nix repl
nix repl
Welcome to Nix version 2.3.8. Type :? for help.
nix-repl> (possibly with a different and more recent nix version number.)
(Note: https://scrive.github.io/nix-workshop/ also looks to be a helpful resource, complmentary to this one, on getting this set up.)