I occasionally have to deal with software with complicated dependencies written in a variety of languages including Python, Go, R, and Haskell.
Along the way, I have also encountered some helpful idioms for
managing these programs’ dependencies using nix, usually by writing or modifying
three files called shell.nix, default.nix, and
~/.nixpkgs/config.nix.
Here’s how they work together.
default.nix lets me specify how to build the software
that I’m working on.
shell.nix lets me specify what dependencies (including
extra development tools) I want the current package to be built
with.
~/.nixpkgs/config.nix lets me make certain account-wide
overrides to the various interpreters that will be acting on the
definitions in the files I’ve provided above.
Here are some snippets of nix code that demonstrate idioms I commonly find myself using, contextualized by a few words about the problems that they solve for me.
Whenever I do any graphing or statistics these days, I turn to GNU R and ggplot2, along with whatever problem-specific packages I feel like using in the course of my analysis.
However, whenever I then want to make a dashboard out of whichever
images I’ve wound up producing that someone finds helpful, I immediately
find that I need a way to bundle up the resulting mess of R packages
that I’m using – sometimes from multiple repositories like
RForge and BIOConductor in addition to
CRAN! – into some kind of repeatable, deployable build,
that is ideally kept separate and isolated from the analogous builds
(possibly with different package versions!) from my previous
projects.
Thus: here’s a typical nixexpr when I’m feeling simple and am willing
to make any necessary overrides in
~/.nixpkgs/config.nix:
{ pkgs ? import <nixpkgs> {} }:
with pkgs;
stdenv.mkDerivation {
name = "foo";
buildInputs = [
git
go_1_4
(rWrapper.override {
packages = with rPackages; [
ggplot2
gridBase
gridExtra
gridSVG
directlabels
dplyr
magrittr
tidyr
dlm
coda
tseries
DBI
RSQLite
falsy
showtext
jsonlite
RCurl
];
})
];
}(Note: for extra fun, add rstudio and set
R_LIBS_SITE and RSTUDIO_WHICH_R.)
(Note 2: for overrides, use buildRPackage!)
(Note 3: CRAN mirrors routinely archive packages, so, if using the
approach above, do be prepared to download tarballs by hand, to create
directory trees that mirror URL path of the original (failing) download,
and then to nix- prefetch-url src/contrib/... to install
the tarball at the necessary nix store path!)
Let’s say that you’ve got a scheduling application that uses Numberjack, which is not yet packaged in nixpkgs. What to do?
Answer: let’s package it locally and keep right on trucking. Thus we
get, in numberjack.nix:
{ stdenv, fetchgit, buildPythonPackage, python, swig, libxml2, zlib, gmp}:
buildPythonPackage rec {
name = "numberjack-${rev}";
rev = "e32843a5306d09c79c3ca2a9eed9649123f436ec";
src = fetchgit {
url = "https://github.com/eomahony/Numberjack";
inherit rev;
sha256 = "e3cfcc161a34592fa854f0801c08bff2c0577721632354513560a3ea1b4fc6ce";
};
buildInputs = [
python
swig
libxml2
zlib
gmp
];
propagatedBuildInputs = [];
}and in shell.nix:
with import <nixpkgs> {};
let
numberjack = callPackage ./numberjack.nix {
inherit swig libxml2 zlib gmp
};
in stdenv.mkDerivation {
name = "python-foo";
version = "0.0.1";
src = null;
buildInputs = with pythonPackages [
bpython
numberjack
requests
...
];
shellHook = ''
export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-bundle.crt;
'';
extraCmds = ''
export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-bundle.crt;
'';
}(again with the CA certificate store customization!).
Haskell presents a variety of interesting challenges to packagers including:
Haskell binaries are sensitive to locale issues settings, which means you’ll need to include appropriate locale data and environment information to help find them.
You’re probably going to want to have different development tools
than package build tools, which means overriding
buildInputs in shell.nix.
Finally, you might decide to try to statically link the result (though, if you prefer something else like Docker, I’ll show you a neat trick later on.)
Thus, you might write, in default.nix, something
like:
{ stdenv, haskellPackages, glibcLocales }:
let
spkgs = p: with p; [
base bytestring text ...
];
env = haskellPackages.ghcWithPackages (p: spkgs p);
in
haskellPackages.mkDerivation {
pname = "foo";
version = "0.0.1";
buildDepends = (spkgs haskellPackages);
license = ...;
src = ./.;
isExecutable = true;
# enableSharedExecutables = false;
# configureFlags = [ "--ghc-option=-optl=-static" "--ghc-option=-optl=-pthread" ];
shellHook = ''
export NIX_GHC="{env}/bin/ghc"
export NIX_GHCPKG="{env}/bin/ghc-pkg"
export NIX_GHC_DOCDIR="{env}/share/doc/ghc/html"
export NIX_GHC_LIBDIR=$( $NIX_GHC --print-libdir )
export LOCALE_ARCHIVE="${glibcLocales}/lib/locale/locale-archive";
'';
}and in shell.nix, something like:
{ pkgs ? (import <nixpkgs> {}) }:
let
openssl = pkgs.openssl.overrideDerivation (oldAttrs: {
dontDisableStatic = true;
postInstall = ''
rm -r $out/etc/ssl/misc $out/bin/c_rehash
'';
});
overrideCabal = pkgs.haskell.lib.overrideCabal;
in pkgs.lib.overrideDerivation
((import ./default.nix) {
inherit (pkgs) stdenv glibcLocales;
haskellPackages = pkgs.haskellPackages.override {
overrides = self: super: {
HsOpenSSL = overrideCabal super.HsOpenSSL (drv: {
librarySystemDepends = [ openssl ];
patches = [ ./hsopenssl-lib.patch ];
});
};
};
})
(old: {
buildInputs = old.buildInputs ++ (with pkgs; [
haskellPackages.cabal-install
vim
less
git
docker
curl
])
})because
if you want to statically link against openssl, you need to ask
for static archives and to avoid deleting them in
postInstall,
you’ll then need to plumb your customized openssl into the
buildInputs of any Haskell packages that link against it,
probably by way of librarySystemDepends,
you’ll then discover that HsOpenSSL’s
Extra-Libraries: crypto ssl is fine for dynamic linking but
that for static linking, you need it to read
Extra-Libraries: ssl crypto so that ld
receives -lssl -lcrypto instead of the (fine for
order-insensitive dynamic linking!) -lcrypto -lssl that it
normally gets, and
finally, you’ll discover at the end of the day that you still
need to set your (undocumented?) SYSTEM_CERTIFICATE_PATH
environment variable to point to the CAs certs for the custom CA you
actually want your app to authenticate against. :-/
So, let’s say the static linking thing described above doesn’t work out – your target machine (Docker) doesn’t have the right version of glibc’s NSS plugins in the path your binary is expecting them in – and you find yourself stuck: what’s a poor engineer to do?
Here’s a horrible, hacky, terrible brute-force solution inspired by Sander’s tips on how to back up Nix/Hydra outputs:
nix-build your project to get an output, probably in
the form of a symlink named result that is pointing at a
store derivation output.
nix-push --dest $(pwd)/cache result to export the
your software’s binary closure to a directory full of NARs.
tar cf app.tar cache result to pack up the resulting
cache for installation via docker.
(cd docker; cp ../app.tar .; docker build .) with a
Dockerfile like this one to install all your software’s
dependencies right back into /nix/store inside
your lovely new docker image.
FROM nixos/nix
MAINTAINER ...
ADD app.tar /
RUN echo "Priority: 10" >> /cache/nix-cache-info
RUN rm -rf /nix/var/nix/binary-cache*
RUN mkdir -p /etc/nix && echo "binary-caches = file:///cache/" > /etc/nix/nix.conf
RUN ["/bin/sh", "-l", "-c", "nix-store -r /result"]
ENTRYPOINT ["/bin/sh", "-l", "-c", "/..."]docker save, rsync, and
docker load your way into production.