Nix Tutorial

Michael Stone, June 20, 2021, , (src), (all posts)

Contents

Introduction

This is a tutorial and an ad for my favorite software packaging tool, “nix”, which is awesome and which more people should know about.

In this context, nix refers to:

  1. The technology described at nixos.org, developed by the Nix and NixOS community.
  2. The nix tool suite
  3. The “nix” language itself, in which “nix expressions” (nix-exprs) are written
  4. nixpkgs, a community monorepo + standard library that define a collection of tens of thousands of packages, library functions, and OS modules that can be used as-is on Linux or MacOS, and which NixOS is defined in terms of.
  5. The basis of NixOS, an innovative Linux distribution based on nix.

Why Learn Nix

Nix has the highest power-to-weight ratio of any software package management system I have ever used.

Nix is the product of a multi-year European research project which produced several PhDs, including Eelco Dolstra, whose dissertation identified the abstractions underlying nix.

Nix solves hard problems in the field of “running software” better than alternatives.

Nix helps many audiences including solo developers, small development teams, open source communities, and large-scale engineering organizations to converge on common task-specific software toolchains at an affordable price, and with nice desiderata around reproducibility and supply-chain integrity.

How to Learn Nix

In my opinion, the best way to learn nix is to:

  1. Understand what nix is and why it’s useful
  2. Understand the big ideas that make nix interesting
  3. Learn, via tutorials, to read the nix expression language
  4. Learn, hands-on, to use the nix repl to evaluate nix expressions
  5. Learn to navigate nixpkgs – specifically all-packages.nix, how to search NixOS/nixpkgs, and how to use nix repl to explore nixpkgs and other nix flakes.
  6. Learn about the different layers of the process of running software that nix addresses, and how tools built with nix address various problems in this space
  7. Learn to use nix and nix flakes to create reproducible development environments for existing software
  8. Learn to use nix and nix flakes to package novel software for deployment and use by others

We’ve already addressed (1) above, so now let’s turn to (2).

The Big Ideas that Make nix Interesting

The most fundamental idea behind nix is that because almost all software is built by combining small pieces into larger ones, it might be handy to have a language specifically for expressing these combinations that works at all levels of scale: all the way from compilers, linkers, and assemblers at the bottom up to tools for scheduling logical networks on to physical hardware at the top.

The next most fundamental idea behind nix is that the recipes for describing how to combine inputs to make outputs should be expressed as functions, with inputs provided as function arguments and with outputs arising from evaluating function application.

Finally, the last really important idea behind nix is that these recipes should be abstracted over where the inputs and outputs are actually stored until the last possible moment when the time comes to actually building the outputs because this way, we can conveniently solve a whole pile of other software packaging and operations problems just by ensuring that novel combinations of inputs always (and preferably deterministically) give rise to novel outputs, typically via some simple cryptography.

Now let’s turn to (3), “learn, via tutorials, to read the nix expression language”, first by discussing the purpose of the nix expression language and then by discussing how to read nix expressions heuristically.

The Purpose of the nix Expression Language

As discussed immediately above, the nix expression language, colloquially “nix”, is a programming language designed to make it easy to define recipes that express how to combine and process inputs to produce outputs.

Like most programming languages, it has types and syntax for common data structures like strings, numbers, lists, dictionaries (called “attrsets”, in nix), and functions.

Unlike most other programming languages, nix-the-language is specialized for working with files that need to be transformed by external programs (often compilers, linkers, etc) to produce directories full of output files, often in target-architecture-specific ways. That’s what makes it so useful defining “how to build programs”, including at the scale of an entire operating system.

The way that nix is specialized for working with files is that, in addition to the common kinds of data structures and computations listed above, nix also includes support for a special kind of computational value, called a “derivation”, that records how to transform input file system structures to output file system structures, typically by running a program (usually a bash shell script) with environment variables initialized to strings that denote the file-system locations that are intended to contain the derivation’s “inputs” and “outputs”.

Thus, the fundamental ideas behind nix so far:

  1. To package software at scale, we need a way to refer to recipes for how to combine inputs to make outputs. Nix doctrine says that we should define “recipes” to be functions that evaluate to derivations or to data structures containing them and ii) that we should “refer” to recipes by i) defining an expression language for expressing recipes and ii) in that language, binding variables to recipes (or to their outputs) as needed.

  2. At the lowest level, “recipes” should be represented by a choice of a program to run to turn inputs into outputs + the paths that the inputs and outputs should be stored in.

  3. (secretly): good things will happen if we also arrange for the nix evaluator to (cryptographically) incorporate provenance information into the actual file/directory-names that go into those input/output storage paths.

How to Read nix Expressions Heuristically

Here are some tips for reading nix expressions (nixexprs, or “nix exprs”) heuristically. Warning: this set of tips is not, and is not intended to be, a complete algorithm. In fact, the actual nix expression syntax is more complex than is presented below; however, hopefully, these simplifications will be useful in helping you get started reading nix expressions of practical relevance to you.

With that, here are the most important rules that I use to make sense of nix-exprs that I’m reading or skimming:

  1. Nix exprs can be read left-to-right.

  2. { tokens mean “attrset” or “attrset pattern”, and need to be paired with a matching } token to close the attrset or attrset-pattern. Attrset-entries, called “attributes” or attribute pairs, are terminated by ; tokens. Attrset patterns bind variables and are separated by , tokens. In nix, attrsets are the fundamental tool for naming values (including functions), and they work a lot like JSON or Javascript dictionaries wherein attributes have keys and values, the keys are usually strings, and the values can have heterogenous types.

  3. : tokens mean “function”. All functions in nix take a single argument, but that argument can have complex internal structure that can be pattern-matched by the function. Argument patterns go to the left of the :. All functions in nix have a body that consists of a single expression, written to the right of the : function introduction token. As described earlier, functions are the primary means of abstracting recipes for building outputs over their inputs, now represented as function arguments.

  4. [ characters mean “list” and, like attrsets, must be closed by a matching ]. List entries are space-separated.

  5. Function application is written as juxtaposition, separated by a space. Parentheses may be used to control order-of-application; otherwise, function application associates to the left.

  6. . tokens occur in three contexts of which the third is by far the most important: in floating-point literals (1.0), in path-literals (./., /.), and to define “attribute lookup paths” (a.b.c) which are used both for reading and for writing the indicated attributes.

  7. ${ tokens in string contexts are called “splices” or “interpolations”, and need to be closed with matching } tokens or need to be escaped. Splices are instructions to evaluate the enclosed expression. Splices can be used in attribute lookup paths, like so: a."${someFunction someArgument}".c to look up dynamically selected attributes in a convenient way.

  8. Nix has syntax for both inline and multi-line strings, introduced by enclosing " or '' tokens, respectively, with the usual in-line character escaping syntax. Multi-line strings strip leading indentation, which is handy for generating config files for typical Unix-style programs.

  9. Nix has additional syntax for managing variable-to-value bindings and scopes/environments. The main pieces of syntax are: let BINDINGS... in BODY, with ATTRSET; BODY, and the rec token for controlling the visibility of sibling bindings introduced in let and attrset contexts in the bodies of other sibling bindings; i.e., for making the sibling bindings visible, which is called making the bindings “mutually recursive”, hence the rec keyword.

  10. Nix has syntax for referring to file system paths, both absolute as well as relative to the file containing the nix-expr currently being evaluated. Note: at evaluation time, path expressions are treated specially by the nix evaluator, which normally copies any files referred to by path expressions to the nix store, which can have big effects on performance.

  11. Nix has syntax for “importing” and evaluating nix-exprs from files. Nix-exprs can be imported from static or dynamically calculated paths. Additionally, imported expressions that evaluate to functions can be immediately applied to arguments; hence an expression like import ./f.nix {} implicitly encodes an expectation that the expression contained in f.nix will evaluate to a function that can be applied to an attrset argument (which, here, is empty).

  12. The typical nix evaluator, provided by github.com/nixos/nix, provides many additional builtin operations including some quite complex operations for fetching source code from upstream version control systems, for fetching packages of nix expressions called “flakes”, and for providing runtime services like profiling, debugging, and runtime reflection (some of which are used extensively by the nixpkgs standard library for key functionality like automatic dependency injection based on nixpkgs-provided package definitions.

  13. The nixpkgs standard library, referred to above, defined a broad and deep range of additional abstractions which are commonly used by people using nix to package software which, to progress, you will eventually need to learn. These additional abstractions are often documented in the nixpkgs manual, in addition to the underlying language, which is mostly documented in the separate but adjacent nix language/tooling manual (stable, unstable). Some key abstractions as of this writing: stdenv.mkDerivation, callPackage/callPackageWith, newScope', list/attrset/string manipulation functions, file reading and writing functions, and the various “overrides” mechanisms like overrideAttrs, override, and overrideDerivation.

Learn, hands-on, to use the nix repl to evaluate nix expressions

Nix expressions are made of simple components that interact in meaningful ways to capture the enormous variety and complexity of real software, e.g., an entire Linux distribution + various proprietary company-specific layers, many packages of which can also be built from the same recipes but with different ingredients on other systems of interest like developer laptops running macOS, and coping with this complexity requires tools.

The most important of these tools, in my opinion, are three things [two of which we will discuss later]:

  1. search, editing, and collaboration tools for working with the nix and nixpkgs source code and documentation.
  2. experience, to learn what to look for / how to navigate this complexity efficiently, and
  3. observability + the ability to experiment cheaply, which the nix tooling provides today via an interactive “read-eval-print” loop, or “repl”, and adjacent tooling for creating disposable development environments, which we will now introduce.

Prerequisites

To be set up to use the nix repl successfully, you’ll need to accomplish at least the following on your own, perhaps aided by the indicated resources.

  1. You need a computer that you can install nix on. Typically, you either own this computer outright yourself and so can run what you want to on it, you’ll rent this computer from a cloud vendor, or you’ll borrow this computer from someone nearby with the understanding that you’re going to experiment on it, and that it will likely need to be reset to factory defaults / reimaged after you’re done with it.

  2. If at all possible, it’s best if you either already are, or are willing to become, comfortable with “the command line”, or with “CLI interfaces” more broadly. Note: if this is new for you, or something you’re still working on, https://github.com/nuitrcs/commandlineworkshop looks from casual inspection like it may be a helpful place to get started.

  3. Finally, you’ll need to install nix on your computer if it’s not already installed and usable. For getting-started directions here, please see nix.dev.

To test whether you’ve got things set up well enough to continue, please

  1. open a fresh terminal / command-line session in the environment you’ve installed nix into
  2. run nix repl, to start the Nix repl
  3. check to see that you see output that looks like
$ nix repl
nix repl
Welcome to Nix version 2.3.8. Type :? for help.

nix-repl> 

(possibly with a different and more recent nix version number.)

(Note: https://scrive.github.io/nix-workshop/ also looks to be a helpful resource, complmentary to this one, on getting this set up.)