write a simple shell package with resholve

This past year I've been working on resholve to improve Nix packaging for shell projects, and it's recently landed in nixpkgs-unstable. This is a first-look at how you can put it to use.

Wait, what's wrong with shell packaging in Nix?

The Shell Way conflicts with The Nix Way. Shell scripts need their environment to supply dependencies that aren't formally declared, but Nix wants to avoid leaking package dependencies into the user's environment. To resolve the conflict The Nix Way, we need to: find the dependencies, declare them, and make references to them absolute.

Manual review and the substitute* functions are good enough for small/simple scripts, but I've found at least one dependency missing from the Nix expression for all of the substantive Shell programs I've checked so far.

Our simple shell project

I'll start with a contrived example--an executable shell script which sources its own library script and invokes some other commands. If you want to follow along, clone the source with

1git clone https://gist.github.com/9d5a815b3ac49728a4893b003ab872c0.git

First, let's look at our executable script:

funsies.bash
1#! /usr/local/bin/bash
2source library.bash
3neofetch

It's a simple script--it just sources library.bash and invokes neofetch. But we'll have to take a look at library.bash to be sure we understand correctly.

library.bash
1neofetch(){
2  cowsay -n < <(command neofetch --stdout)
3}

Aha! There's a catch in our library: funsies.bash will run the library's neofetch function--not the executable itself. Naively replacing neofetch with ${neofetch}/bin/neofetch would change the script's behavior. The library also has a dependency on cowsay that we could've missed if we just skimmed the executable script.

Writing a default.nix for our shell project

This is where resholve comes in. Instead of stdenv.mkDerivation, our Nix expression uses resholve.mkDerivation to build the package and tell resholve:

  1. which scripts to resolve
  2. which interpreter (shebang) to use
  3. to look for missing commands in the cowsay and neofetch packages
default.nix
 1{ pkgs ? import <nixpkgs> { } }:
 2
 3with pkgs;
 4resholvePackage rec {
 5  version = "unset";
 6  pname = "funsies";
 7  src = ./.;
 8  installPhase = ''
 9    install -Dv library.bash $out/library.bash
10    install -Dv funsies.bash $out/bin/funsies.bash
11  '';
12  solutions = {
13    # an arbitrary name that you'd use to override this solution
14    # (some packages need multiple conflicting solutions)
15    funsies = {
16      # $out-relative paths
17      scripts = [ "library.bash" "bin/funsies.bash" ]; # 1
18      interpreter = "${bash_5}/bin/sh"; # 2
19      inputs = [ neofetch cowsay ]; # 3
20    };
21  };
22}

That's all--we're ready to nix-build it:

build.console
 1bash-5.1 $ nix-build
 2this derivation will be built:
 3  /nix/store/cgzwhy4wk360kdlidlq9rrpxc03ar1l7-funsies-unset.drv
 4building '/nix/store/cgzwhy4wk360kdlidlq9rrpxc03ar1l7-funsies-unset.drv'...
 5unpacking sources
 6unpacking source archive /nix/store/rqj87qyg55768h6pda28bqx2k5zjazkq-write_a_simple_shell_package_with_resholve
 7source root is write_a_simple_shell_package_with_resholve
 8patching sources
 9configuring
10no configure script, doing nothing
11building
12no Makefile, doing nothing
13installing
14install: creating directory '/nix/store/y764dgi2nnmdgzhp99a2zinhfvl8dwrm-funsies-unset'
15'library.bash' -> '/nix/store/y764dgi2nnmdgzhp99a2zinhfvl8dwrm-funsies-unset/library.bash'
16install: creating directory '/nix/store/y764dgi2nnmdgzhp99a2zinhfvl8dwrm-funsies-unset/bin'
17'funsies.bash' -> '/nix/store/y764dgi2nnmdgzhp99a2zinhfvl8dwrm-funsies-unset/bin/funsies.bash'
18post-installation fixup
19/nix/store/y764dgi2nnmdgzhp99a2zinhfvl8dwrm-funsies-unset /private/tmp/nix-build-funsies-unset.drv-0/write_a_simple_shell_package_with_resholve
20Overwrote '/nix/store/y764dgi2nnmdgzhp99a2zinhfvl8dwrm-funsies-unset/bin/funsies.bash'
21Overwrote '/nix/store/y764dgi2nnmdgzhp99a2zinhfvl8dwrm-funsies-unset/library.bash'
22/private/tmp/nix-build-funsies-unset.drv-0/write_a_simple_shell_package_with_resholve
23strip is /nix/store/rih2pjsr7p9dwwa6j3kdvpw4gdka7ik6-cctools-binutils-darwin-949.0.1/bin/strip
24stripping (with command strip and flags -S) in /nix/store/y764dgi2nnmdgzhp99a2zinhfvl8dwrm-funsies-unset/bin 
25patching script interpreter paths in /nix/store/y764dgi2nnmdgzhp99a2zinhfvl8dwrm-funsies-unset
26/nix/store/y764dgi2nnmdgzhp99a2zinhfvl8dwrm-funsies-unset

Ok, but what did it do?

During the fixup phase, resholve.mkDerivation invokes resholve to overwrite our copies of funsies.bash and library.bash with external dependencies resolved to the appropriate store paths. Let's take a look at the rewritten copies in the result symlink, starting with funsies.bash:

result/bin/funsies.bash
1#!/nix/store/bh237kq8sdsgp8l7m9c846fc9klpi4jn-bash-5.1-p4/bin/sh
2source /nix/store/y764dgi2nnmdgzhp99a2zinhfvl8dwrm-funsies-unset/library.bash
3neofetch
4
5### resholve directives (auto-generated) ## format_version: 2
6# resholve: keep source:/nix/store/y764dgi2nnmdgzhp99a2zinhfvl8dwrm-funsies-unset/library.bash
7

Here you can see that resholve:

  • prepended the interpreter we specified (${bash_5}/bin/sh)
  • resolved the reference to library.bash to an absolute store path
  • was smart enough to avoid resolving neofetch, because it actually just refers to the function defined in library.bash

(It also appended information about how it resolved this script. You don't need to understand these lines. Each time resholve encounters a script that sources another script, it also parses the sourced script. When it does, it uses this list of directives to understand how it originally resolved the script.)

Let's see what it did to the library:

result/library.bash
1#!/nix/store/bh237kq8sdsgp8l7m9c846fc9klpi4jn-bash-5.1-p4/bin/sh
2neofetch(){
3  /nix/store/f7yvzpilpymw8356xm477fz35kmrkhhr-cowsay-3.03+dfsg2/bin/cowsay -n < <(command /nix/store/rbvydki2qkfp4v0qxs0nqiniqwh3cxwq-neofetch-7.1.0/bin/neofetch --stdout)
4}
5
6### resholve directives (auto-generated) ## format_version: 2
7# resholve: keep /nix/store/f7yvzpilpymw8356xm477fz35kmrkhhr-cowsay-3.03+dfsg2/bin/cowsay
8# resholve: keep /nix/store/rbvydki2qkfp4v0qxs0nqiniqwh3cxwq-neofetch-7.1.0/bin/neofetch
9

Most of this should make sense from before, but I'll pull out how it changed the main invocation for reference:

1               cowsay -n < <(command                neofetch --stdout) # source 
2/nix/store/.../cowsay -n < <(command /nix/store/.../neofetch --stdout) # resolved

Finally, let's take it for a spin:

run.console
 1bash-5.1 $ ./result/bin/funsies.bash
 2 _____________________________________________________ 
 3/ abathur@085df404                                    \
 4| ----------------                                    |
 5| OS: macOS Catalina 10.15.7 19H2 x86_64              |
 6| Host: MacBookAir9,1                                 |
 7| Kernel: 19.6.0                                      |
 8| Uptime: 13 days, 2 hours, 26 mins                   |
 9| Packages: 7 (brew), 375 (nix-system), 2 (nix-user)  |
10| Shell: bash 4.4.23                                  |
11| Resolution: 2560x1600                               |
12| DE: Aqua                                            |
13| WM: Quartz Compositor                               |
14| WM Theme: Graphite (Dark)                           |
15| Terminal: perl                                      |
16| CPU: Intel i5-1030NG7 (8) @ 1.10GHz                 |
17| GPU: Intel Iris Plus Graphics                       |
18| Memory: 10976MiB / 16384MiB                         |
19\                                                     /
20 ----------------------------------------------------- 
21        \   ^__^
22         \  (oo)\_______
23            (__)\       )\/\
24                ||----w |
25                ||     ||
Discussing this elsewhere?
Enter 'link' for a markdown link
or 'tweet <message>' for a pre-populated Tweet :)
Want to subscribe? Enter 'rss' for a feed URL.
>