% Nix Packaging Tricks: Making Go, R, Python, Haskell, OpenSSL, and Docker Dance
% Michael Stone
% October 18, 2015

## Introduction

I occasionally have to deal with software with complicated dependencies
written in a variety of languages including Python, Go, R, and Haskell.

Along the way, I have also encountered some helpful idioms for managing these
programs' dependencies using [nix], usually by writing or modifying three
files called `shell.nix`, `default.nix`, and `~/.nixpkgs/config.nix`.

[nix]: https://nixos.org/nix/

Here's how they work together.

## Separation of Concerns

`default.nix` lets me specify how to build the software that I'm working on.

`shell.nix` lets me specify what dependencies (including extra development
tools) I want the current package to be built with.

`~/.nixpkgs/config.nix` lets me make certain account-wide overrides to the
various interpreters that will be acting on the definitions in the files
I've provided above.

## Snippets

Here are some snippets of nix code that demonstrate idioms I commonly find
myself using, contextualized by a few words about the problems that they solve
for me.

### Customizing R

Whenever I do any graphing or statistics these days, I turn to [GNU R] and
[ggplot2], along with whatever problem-specific packages I feel like using in
the course of my analysis.

However, whenever I then want to make a dashboard out of whichever images I've
wound up producing that someone finds helpful, I immediately find that I need
a way to bundle up the resulting mess of R packages that I'm using --
sometimes from multiple repositories like `RForge` and `BIOConductor` in
addition to `CRAN`! -- into some kind of repeatable, deployable build, that is
ideally kept separate and isolated from the analogous builds (possibly with
different package versions!) from my previous projects.

[GNU R]: https://r-project.org/
[ggplot2]: https://ggplot2.org/

Thus: here's a typical nixexpr when I'm feeling simple and am willing to make
any necessary overrides in `~/.nixpkgs/config.nix`:

~~~~ { .nix }
{ pkgs ? import <nixpkgs> {} }:
with pkgs;
stdenv.mkDerivation {
	name = "foo";
	buildInputs = [
		git

		go_1_4

		(rWrapper.override {
			packages = with rPackages; [
				ggplot2
				gridBase
				gridExtra
				gridSVG
				directlabels

				dplyr
				magrittr
				tidyr

				dlm
				coda
				tseries

				DBI
				RSQLite

				falsy

				showtext
				jsonlite
				RCurl
			];
		})
	];
}
~~~~

(Note: for extra fun, add `rstudio` and set `R_LIBS_SITE` and `RSTUDIO_WHICH_R`.)

(Note 2: for overrides, use `buildRPackage`!)

(Note 3: CRAN mirrors routinely archive packages, so, if using the approach
above, do be prepared to download tarballs by hand, to create directory trees
that mirror URL path of the original (failing) download, and then to `nix-
prefetch-url src/contrib/...` to install the tarball at the necessary nix
store path!)

### Customizing Python

Let's say that you've got a scheduling application that uses
[Numberjack](http://numberjack.ucc.ie), which is not yet packaged in
[nixpkgs](https://github.com/NixOS/nixpkgs). What to do?

Answer: let's package it locally and keep right on trucking. Thus we get, in
`numberjack.nix`:

~~~~ { .nix }
{ stdenv, fetchgit, buildPythonPackage, python, swig, libxml2, zlib, gmp}:
buildPythonPackage rec {
	name = "numberjack-${rev}";
	rev = "e32843a5306d09c79c3ca2a9eed9649123f436ec";
	src = fetchgit {
		url = "https://github.com/eomahony/Numberjack";
		inherit rev;
		sha256 = "e3cfcc161a34592fa854f0801c08bff2c0577721632354513560a3ea1b4fc6ce";
	};
	buildInputs = [
		python
		swig
		libxml2
		zlib
		gmp
	];
	propagatedBuildInputs = [];
}
~~~~

and in `shell.nix`:

~~~~ { .nix }
with import <nixpkgs> {};
let
	numberjack = callPackage ./numberjack.nix {
		inherit swig libxml2 zlib gmp
	};
in stdenv.mkDerivation {
	name = "python-foo";
	version = "0.0.1";
	src = null;
	buildInputs = with pythonPackages [
		bpython
		numberjack
		requests
		...
	];
	shellHook = ''
		export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-bundle.crt;
	'';
	extraCmds = ''
		export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-bundle.crt;
	'';
}
~~~~

(again with the CA certificate store customization!).

### Customizing Haskell

Haskell presents a variety of interesting challenges to packagers including:

  1. Haskell binaries are sensitive to locale issues settings, which means
  you'll need to include appropriate locale data and environment information
  to help find them.

  2. You're probably going to want to have different development tools than
  package build tools, which means overriding `buildInputs` in `shell.nix`.

  3. Finally, you might decide to try to statically link the result (though,
  if you prefer something else like Docker, I'll show you a neat trick later
  on.)

Thus, you might write, in `default.nix`, something like:

~~~~ { .nix }
{ stdenv, haskellPackages, glibcLocales }:
let
  spkgs = p: with p; [
  	base bytestring text ...
  ];
  env = haskellPackages.ghcWithPackages (p: spkgs p);
in
haskellPackages.mkDerivation {
	pname = "foo";
	version = "0.0.1";
	buildDepends = (spkgs haskellPackages);
	license = ...;
	src = ./.;
	isExecutable = true;
	# enableSharedExecutables = false;
	# configureFlags = [ "--ghc-option=-optl=-static" "--ghc-option=-optl=-pthread" ];
	shellHook = ''
		export NIX_GHC="{env}/bin/ghc"
		export NIX_GHCPKG="{env}/bin/ghc-pkg"
		export NIX_GHC_DOCDIR="{env}/share/doc/ghc/html"
		export NIX_GHC_LIBDIR=$( $NIX_GHC --print-libdir )
		export LOCALE_ARCHIVE="${glibcLocales}/lib/locale/locale-archive";
	'';
}
~~~~

and in `shell.nix`, something like:

~~~~
{ pkgs ? (import <nixpkgs> {}) }:
let
	openssl = pkgs.openssl.overrideDerivation (oldAttrs: {
		dontDisableStatic = true;
		postInstall = ''
			rm -r $out/etc/ssl/misc $out/bin/c_rehash
		'';
	});
	overrideCabal = pkgs.haskell.lib.overrideCabal;
in pkgs.lib.overrideDerivation
	((import ./default.nix) {
		inherit (pkgs) stdenv glibcLocales;
		haskellPackages = pkgs.haskellPackages.override {
			overrides = self: super: {
				HsOpenSSL = overrideCabal super.HsOpenSSL (drv: {
					librarySystemDepends = [ openssl ];
					patches = [ ./hsopenssl-lib.patch ];
				});
			};
		};
	})
	(old: {
		buildInputs = old.buildInputs ++ (with pkgs; [
			haskellPackages.cabal-install
			vim
			less
			git
			docker
			curl
		])
	})
~~~~

because

  a) if you want to statically link against openssl, you need to ask for
  static archives and to avoid deleting them in `postInstall`,

  b) you'll then need to plumb your customized openssl into the `buildInputs`
  of any Haskell packages that link against it, probably by way of
  `librarySystemDepends`,

  c) you'll then discover that HsOpenSSL's `Extra-Libraries: crypto ssl` is
  fine for dynamic linking but that for static linking, you need it to read
  `Extra-Libraries: ssl crypto` so that `ld` receives `-lssl -lcrypto` instead
  of the (fine for order-insensitive dynamic linking!) `-lcrypto -lssl` that
  it normally gets, and

  d) finally, you'll discover at the end of the day that you still need to set
  your (undocumented?) `SYSTEM_CERTIFICATE_PATH` environment variable to point
  to the CAs certs for the custom CA you actually want your app to
  authenticate against. :-/

### Deploying nix-closures to Docker

So, let's say the static linking thing described above doesn't work out --
your target machine (Docker) doesn't have the right version of glibc's NSS
plugins in the path your binary is expecting them in -- and you find yourself
stuck: what's a poor engineer to do?

Here's a horrible, hacky, terrible brute-force solution inspired by [Sander's
tips] on how to back up Nix/Hydra outputs:

  1. `nix-build` your project to get an output, probably in the form of a
  symlink named `result` that is pointing at a store derivation output.

  2. `nix-push --dest $(pwd)/cache result` to export the your software's
  binary closure to a directory full of NARs.

  3. `tar cf app.tar cache result` to pack up the resulting cache for
  installation via `docker`.

  4. `(cd docker; cp ../app.tar .; docker build .)` with a `Dockerfile` like
  this one to install all your software's dependencies right back into
  `/nix/store` *inside* your lovely new docker image.

~~~~ { .Dockerfile }
FROM nixos/nix
MAINTAINER ...

ADD app.tar /
RUN echo "Priority: 10" >> /cache/nix-cache-info
RUN rm -rf /nix/var/nix/binary-cache*
RUN mkdir -p /etc/nix && echo "binary-caches = file:///cache/" > /etc/nix/nix.conf
RUN ["/bin/sh", "-l", "-c", "nix-store -r /result"]

ENTRYPOINT ["/bin/sh", "-l", "-c", "/..."]
~~~~

  5. Then, because docker registries terrify you, `docker save`, `rsync`, and
  `docker load` your way into production.

[Sander's tips]: http://sandervanderburg.blogspot.com/2014/07/backing-up-nix-and-hydra-builds.html