Nix Packaging Tricks: Making Go, R, Python, Haskell, OpenSSL, and Docker Dance

Michael Stone, October 18, 2015, , (src), (all posts)

Contents

Introduction

I occasionally have to deal with software with complicated dependencies written in a variety of languages including Python, Go, R, and Haskell.

Along the way, I have also encountered some helpful idioms for managing these programs’ dependencies using nix, usually by writing or modifying three files called shell.nix, default.nix, and ~/.nixpkgs/config.nix.

Here’s how they work together.

Separation of Concerns

default.nix lets me specify how to build the software that I’m working on.

shell.nix lets me specify what dependencies (including extra development tools) I want the current package to be built with.

~/.nixpkgs/config.nix lets me make certain account-wide overrides to the various interpreters that will be acting on the definitions in the files I’ve provided above.

Snippets

Here are some snippets of nix code that demonstrate idioms I commonly find myself using, contextualized by a few words about the problems that they solve for me.

Customizing R

Whenever I do any graphing or statistics these days, I turn to GNU R and ggplot2, along with whatever problem-specific packages I feel like using in the course of my analysis.

However, whenever I then want to make a dashboard out of whichever images I’ve wound up producing that someone finds helpful, I immediately find that I need a way to bundle up the resulting mess of R packages that I’m using – sometimes from multiple repositories like RForge and BIOConductor in addition to CRAN! – into some kind of repeatable, deployable build, that is ideally kept separate and isolated from the analogous builds (possibly with different package versions!) from my previous projects.

Thus: here’s a typical nixexpr when I’m feeling simple and am willing to make any necessary overrides in ~/.nixpkgs/config.nix:

{ pkgs ? import <nixpkgs> {} }:
with pkgs;
stdenv.mkDerivation {
    name = "foo";
    buildInputs = [
        git

        go_1_4

        (rWrapper.override {
            packages = with rPackages; [
                ggplot2
                gridBase
                gridExtra
                gridSVG
                directlabels

                dplyr
                magrittr
                tidyr

                dlm
                coda
                tseries

                DBI
                RSQLite

                falsy

                showtext
                jsonlite
                RCurl
            ];
        })
    ];
}

(Note: for extra fun, add rstudio and set R_LIBS_SITE and RSTUDIO_WHICH_R.)

(Note 2: for overrides, use buildRPackage!)

(Note 3: CRAN mirrors routinely archive packages, so, if using the approach above, do be prepared to download tarballs by hand, to create directory trees that mirror URL path of the original (failing) download, and then to nix- prefetch-url src/contrib/... to install the tarball at the necessary nix store path!)

Customizing Python

Let’s say that you’ve got a scheduling application that uses Numberjack, which is not yet packaged in nixpkgs. What to do?

Answer: let’s package it locally and keep right on trucking. Thus we get, in numberjack.nix:

{ stdenv, fetchgit, buildPythonPackage, python, swig, libxml2, zlib, gmp}:
buildPythonPackage rec {
    name = "numberjack-${rev}";
    rev = "e32843a5306d09c79c3ca2a9eed9649123f436ec";
    src = fetchgit {
        url = "https://github.com/eomahony/Numberjack";
        inherit rev;
        sha256 = "e3cfcc161a34592fa854f0801c08bff2c0577721632354513560a3ea1b4fc6ce";
    };
    buildInputs = [
        python
        swig
        libxml2
        zlib
        gmp
    ];
    propagatedBuildInputs = [];
}

and in shell.nix:

with import <nixpkgs> {};
let
    numberjack = callPackage ./numberjack.nix {
        inherit swig libxml2 zlib gmp
    };
in stdenv.mkDerivation {
    name = "python-foo";
    version = "0.0.1";
    src = null;
    buildInputs = with pythonPackages [
        bpython
        numberjack
        requests
        ...
    ];
    shellHook = ''
        export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-bundle.crt;
    '';
    extraCmds = ''
        export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-bundle.crt;
    '';
}

(again with the CA certificate store customization!).

Customizing Haskell

Haskell presents a variety of interesting challenges to packagers including:

  1. Haskell binaries are sensitive to locale issues settings, which means you’ll need to include appropriate locale data and environment information to help find them.

  2. You’re probably going to want to have different development tools than package build tools, which means overriding buildInputs in shell.nix.

  3. Finally, you might decide to try to statically link the result (though, if you prefer something else like Docker, I’ll show you a neat trick later on.)

Thus, you might write, in default.nix, something like:

{ stdenv, haskellPackages, glibcLocales }:
let
  spkgs = p: with p; [
    base bytestring text ...
  ];
  env = haskellPackages.ghcWithPackages (p: spkgs p);
in
haskellPackages.mkDerivation {
    pname = "foo";
    version = "0.0.1";
    buildDepends = (spkgs haskellPackages);
    license = ...;
    src = ./.;
    isExecutable = true;
    # enableSharedExecutables = false;
    # configureFlags = [ "--ghc-option=-optl=-static" "--ghc-option=-optl=-pthread" ];
    shellHook = ''
        export NIX_GHC="{env}/bin/ghc"
        export NIX_GHCPKG="{env}/bin/ghc-pkg"
        export NIX_GHC_DOCDIR="{env}/share/doc/ghc/html"
        export NIX_GHC_LIBDIR=$( $NIX_GHC --print-libdir )
        export LOCALE_ARCHIVE="${glibcLocales}/lib/locale/locale-archive";
    '';
}

and in shell.nix, something like:

{ pkgs ? (import <nixpkgs> {}) }:
let
    openssl = pkgs.openssl.overrideDerivation (oldAttrs: {
        dontDisableStatic = true;
        postInstall = ''
            rm -r $out/etc/ssl/misc $out/bin/c_rehash
        '';
    });
    overrideCabal = pkgs.haskell.lib.overrideCabal;
in pkgs.lib.overrideDerivation
    ((import ./default.nix) {
        inherit (pkgs) stdenv glibcLocales;
        haskellPackages = pkgs.haskellPackages.override {
            overrides = self: super: {
                HsOpenSSL = overrideCabal super.HsOpenSSL (drv: {
                    librarySystemDepends = [ openssl ];
                    patches = [ ./hsopenssl-lib.patch ];
                });
            };
        };
    })
    (old: {
        buildInputs = old.buildInputs ++ (with pkgs; [
            haskellPackages.cabal-install
            vim
            less
            git
            docker
            curl
        ])
    })

because

  1. if you want to statically link against openssl, you need to ask for static archives and to avoid deleting them in postInstall,

  2. you’ll then need to plumb your customized openssl into the buildInputs of any Haskell packages that link against it, probably by way of librarySystemDepends,

  3. you’ll then discover that HsOpenSSL’s Extra-Libraries: crypto ssl is fine for dynamic linking but that for static linking, you need it to read Extra-Libraries: ssl crypto so that ld receives -lssl -lcrypto instead of the (fine for order-insensitive dynamic linking!) -lcrypto -lssl that it normally gets, and

  4. finally, you’ll discover at the end of the day that you still need to set your (undocumented?) SYSTEM_CERTIFICATE_PATH environment variable to point to the CAs certs for the custom CA you actually want your app to authenticate against. :-/

Deploying nix-closures to Docker

So, let’s say the static linking thing described above doesn’t work out – your target machine (Docker) doesn’t have the right version of glibc’s NSS plugins in the path your binary is expecting them in – and you find yourself stuck: what’s a poor engineer to do?

Here’s a horrible, hacky, terrible brute-force solution inspired by Sander’s tips on how to back up Nix/Hydra outputs:

  1. nix-build your project to get an output, probably in the form of a symlink named result that is pointing at a store derivation output.

  2. nix-push --dest $(pwd)/cache result to export the your software’s binary closure to a directory full of NARs.

  3. tar cf app.tar cache result to pack up the resulting cache for installation via docker.

  4. (cd docker; cp ../app.tar .; docker build .) with a Dockerfile like this one to install all your software’s dependencies right back into /nix/store inside your lovely new docker image.

FROM nixos/nix
MAINTAINER ...

ADD app.tar /
RUN echo "Priority: 10" >> /cache/nix-cache-info
RUN rm -rf /nix/var/nix/binary-cache*
RUN mkdir -p /etc/nix && echo "binary-caches = file:///cache/" > /etc/nix/nix.conf
RUN ["/bin/sh", "-l", "-c", "nix-store -r /result"]

ENTRYPOINT ["/bin/sh", "-l", "-c", "/..."]
  1. Then, because docker registries terrify you, docker save, rsync, and docker load your way into production.