I have recently gotten into declarative system definitions. Not out of nowhere, of course. My standard work VM was happily on fedora for a good few years, before something corrupted the disk. Not sure what, improper shutdown? Cosmic ray? No clue.

But that being the case, I needed a new VM pretty fast. Our deployments at work are done on Rocky Linux, but that had some unusual interactions with VMWare - VMWare+Wayland is a cursed combination and you'll likely run into niche issues like "lmao my cursor is invisible". I remade the VM pretty fast in Alma Linux though, which worked nicely. Very standard drop-in, but I was left thinking "I'm having to do this all manually... that's a pain. Maybe it's time to look into a deployment system or similar to make my config persist even if (when) my VM goes kaboom again".

I tried out NixOS for a week or two and really liked the concepts. The DSL took some getting used to, it feels a little magical at times (in a bad way) and I still don't really get it. It took very little time to convert my old environment into nix though, like less than a few hours from fresh install to having my old setup basically replicated.

Cool huh? Can't say I've fully internalised the syntax, but it's quick enough to learn the very basics.

Then a few days later I was browsing the open slopware list for no particular reason and saw "NixOS - Permissive AI policy. Alternative: GNU Guix". I'm reasonably anti-AI as they go, so yknow what let's give it a go. What's the worst that can happen.

Early exploration, or: the things Guix does better

Home config!!! HOME CONFIG! In nix you have to bind in home-manager as its own thing, and figure out the frankly concerningly long list of options that thing has (5408 at time of writing); in Guix, everyone has their own package store by default, and a home.scm to define your home directory like

(define home-config
    (home-environment 
        (packages
            (list (specifications->packages (list "emacs"))))
        (services
            (list (service home-zsh-service-type)))))

as the fully native equivalent to the home-manager

{
  home.packages = [pkgs.emacs];
  programs.zsh.enable = true;
}

whilst the nix one is more succinct, the Guix one is just pure scheme and can be extended however you'd like.

Also because the per-user config is distinct, you don't have to rebuild your whole system to apply home changes, guix home will just handle it. Very cool I think.

Guix also does not use SystemD. I do not have strong views on SystemD, but I have had to debug why things don't work under it many times and have spent many an hour going "WHY DOES THAT NOT ORDER RIGHT?". Guix has GNU Shepherd, which is more of a pure process supervisor than anything more complex - you point it at an executable in the store and it'll happily run it. Very simple actually.

Scheme is a bit of a hurdle though, understanding quotes, semi-quotes and quasi-quotes took a while. Here's an exercise for the reader, which one of these is correct syntax?

(define cert "mycert")

A:
(environment-variables
  `(("CLIENT_CERT" . cert)))
  
B:
(environment-variables
  `(("CLIENT_CERT" , cert)))
  
C:
(environment-variables
  `(("CLIENT_CERT" . ,cert)))

THAT'S RIGHT IT'S C BECAUSE YOU NEED TO TELL THE INTERPRETER THAT YOU MEAN cert IS A VARIABLE IT SHOULD CARE ABOUT AND NOT A SYMBOL BECAUSE THOSE ARE THINGS AND ALSO YOU NEED THE DOT BECAUSE THAT'S HOW YOU MAKE A KEY-VALUE THINGY OTHERWISE IT'S A LIST.

I am not a lisp-head.

But hey I took my nix config and rewrote it in scheme pretty fast, all good right? Time to get to work!

I use oh-my-zsh on the shell. Not sure why exactly, picked it up about 10 years ago and never put it back down. But hey, I quite like it and it's what I'm used to. And given that I'm now a washed-up dev I guess that means I'm sticking with it.

But how do I get that on Guix?

Oh-my-zsh shennanigans

Over in nix-land, you can specify

{
    programs.zsh.oh-my-zsh.enable = true;
}

and the mystical magic will take care of it. Naturally Guix does not have this so we're going to have to do it ourselves. But it's a simple git clone, right? Can't be that hard.

Literally nobody on the internet seems to have tried this, ever. There is NO PRIOR ART TO PULL FROM D:

But listen here, reader, you forget that I am insane and also evil and also will try things that are a long shot. Get this, I was reading the documentation about packaging and saw that there's a git-fetch method. Surely we can use that, but how?

So I had to do a bigthink. The biggest bigthink. All home-files-service-type wants is to be given a structure that goes (location, file-like). The term file-like is used extensively in the Guix documentation with no satisfactory explanation for what one actually IS. So I'll give the one that I think works - a file-like is any expression which resolves down to a pointer to the Guix store. Anything. (local-file "./path") will shove ./path into the Guix store and give you its new location. (plain-file "string") will take the literal "string", shove it in the store, and give you the path.

SOOOO! We can use ANYTHING that resolves down to that. And origin will! It'll download it from somewhere and put it in the store, so behold!

(define home-config
  (home-environment
    (services
      (list
        (service home-files-service-type
           `((".oh-my-zsh", (origin
                             (method git-fetch)
                             (uri (git-reference
                                   (url "https://github.com/ohmyzsh/ohmyzsh.git")
                                   (commit "964ddf789b022b6958b27d02d7fa54f552de21e9")))
                             (sha256 (base32 "1v7c774sj584yn6x5ivzkjv7jbfq0593igic6p5vjgsf0sfgcva4"))))))))))

This works! origin will go off and git fetch the repository, shove it in the store, and then it'll get symlinked to .oh-my-zsh. Wonderful...

Alright now we can finally get some work done. I just need to install some custom CAs first... huh? How do I do that?

Custom CA NONSENSE

Nix provides security.pki.certificateFiles as a standard option, so this:

security.pki.certificateFiles = [
    (builtins.fetchurl {
      url = "https://your-ca-cert-location";
      sha256 = "thing-here";
    })
];

just works.

Guix has no such option. This is a very corporate thing to need, yknow. Custom CAs. No wonder Guix's standard userbase don't have much of a need for it.

I had to go looking a lot to find grop's blog post about how they achieved this. Since finding that post was a pain in the bum, I'll replicate my config here in case anyone stumbles upon this from le google.

custom-ca-certs.scm (filename IMPORTANT)

(define-module (certs)
  #:use-module (guix build-system trivial)
  #:use-module (guix packages)
  #:use-module (guix download)
  #:use-module (gnu packages perl)
  #:use-module (gnu packages tls)
  #:use-module (guix gexp)
  #:use-module ((guix licenses) #:prefix license:))

(define-public custom-ca-certs
  (package
   (name "floti-certs")
   (version "1")
   (build-system trivial-build-system)
   (source #f)
   (inputs
    `(("ca", (origin
       (method url-fetch)
       (uri "https://your-thing-here")
       (sha256 (base32 "use guix download to get this"))))
      ("ca-2", (more here))))
   (native-inputs
    (list openssl perl))
   (arguments
    (list
     #:modules '((guix build utils))
     #:builder
     #~(begin
	 (use-modules (guix build utils))
         (let ((out (string-append #$output "/etc/ssl/certs"))
               (openssl
                (assoc-ref %build-inputs "openssl"))
               (perl
                (assoc-ref %build-inputs "perl"))
               (ca (assoc-ref %build-inputs "ca"))
	           (ca-2 (assoc-ref %build-inputs "ca-2")) ; add more here if you need them

           (mkdir-p out)
           (for-each
            (lambda (cert)
              (copy-file cert (string-append out "/"
                                             (strip-store-file-name cert))))
            (list ca ca-2)) ; also add them here 
           (chdir out)
           (invoke (string-append perl "/bin/perl")
                   (string-append openssl "/bin/c_rehash")
                   ".")))))
  (home-page "http://localhost")
  (synopsis "Custom certificates")
  (description
    "This package provides a certificate store containing needed CA certificates.")
  (license license:public-domain)))

custom-ca-certs

Then in your system.scm

(add-to-load-path (dirname (current-filename)))
(use-modules ...
    (custom-ca-certs))
(operating-system
  (packages (append
    (list custom-ca-certs)))) ; make sure this isn't wrapped in specification->package or you'll get a funny error

And there ya go, custom certs. Cool!

Now I can finally get some work done...

pnpm test

Docker...

THE DOCKER VERSION PACKAGED BY GUIX IS v20. TWENTY. RELEASED IN 2023. THREE YEARS AGO.

My work tests did not work because v20 is so old that docker-compose hadn't been spun off into docker compose yet. And you can't use a newer client that has it correctly spun off because the daemon does not like that.

Alright I'll have to find a way to get a newer version. How hard can that be? (clueless)

Galaxy-brain idea that did not work

Guix actually packages Nix. You can outright install Nix on a Guix system and it's happy about that. Nix packages docker. See where I'm going with this?

BEHOLD MY MASTERPIECE:

(define docker
  (nix-shell-wrapper "docker"
                     '("github:NixOS/nixpkgs/nixos-26.05#docker")
                     #:run-command '("dockerd")))

...
(operating-system
  (services
   ...
     (simple-service 'dockerd shepherd-root-service-type
                                 (list
                                  (shepherd-service
                                   (documentation "Docker!")
                                   (provision '(dockerd))
                                   (requirement '(user-processes containerd dbus-system networking udev))
                                   (start #~(make-forkexec-constructor
                                             (list #$docker)
                                             #:log-file "/var/log/docker.log"))
                                   (stop #~(make-kill-destructor)))))

Beautiful isn't it? As I mentioned, shepherd can be pointed at any binary in the store. Anything. I was using the really cool nix-in-guix thingy, in Rosenthal, which let me create a "file-like" (anything that resolves to a pointer to the store) from a nix package.

And lo and behold, it RAN! I could docker ps. I could docker compose!

BUT.

runc. Docker's daemon has external binary dependencies that it calls as part of its normal activities, and those will not be in its path, given that we have everything in our Guix store...

Anything encased in a nix-shell-wrapper is effectively isolated, you cannot manipulate its $PATH or anything, so you can't hack it into working.

Sad.

So I had to actually update the Guix docker package

I HAVE NO IDEA WHAT I'M DOING.

Ok I'm not 100% clueless I did just ~~steal~~ write that CA certificate thingy...

These snippets will be way too long to embed, so I'm going to put them on a public repo to refer to it. This one in particular.

When I started, the state of the package was like this. Do you understand any of that? NEITHER DID I.

I tried just bumping the version (define docker-version) to something more sensible, but the build failed for no easily diagnosable reason. It looked like it was something to do with the go.mod? But it didn't give anything useful. I tried patching the build script, had to remove lines from the package that referred to now-gone files, renamed files and so on.

I was stuck doing this for a few hours. Like I was properly in the weeds until I had a brainwave - why don't I just go a few versions at a time instead of trying to jump 9 major versions at once?

Perhaps the best brainwave I've had in a while. Alright, drop down to v23 and we have some WAY MORE understandable errors. Let's walk you through what this package build actually DOES.

We start with the 'unpack phase, a standard for all packages. The source is downloaded and shoved in a temporary directory.

Then we get 'patch-paths - what this is trying to do is find hardcoded binary paths and updating them to point to our Guix Store, because they WILL NOT be in the "normal" locations like a normal system. I largely didn't have to touch the first step of this where it looks in specified locations (default config?) apart from removing a few entries that pointed to files that were gone.

Then the actually fun one. substitute-LookPath* and substitute-Command*. These are very well named actually - the function is going to look for the golang calls exec.LookPath (I assume this is go's version of which) and exec.Command, a shell call. These are trying to find executables that we also need to rewrite to point to the store. Immediately after this call is a check for any unsubstituted commands, which will fail the build if it finds any.

The final step is just purging tests that fail in the builder before the 'check step which was supposed to run unit tests that I disabled :)

The step I had to interact most with was the 'patch-paths, docker introduced a bunch of new dependencies and ALL OF THEM must be passed as inputs to the build to have access to them.

An easy one is slirp4netns - docker uses this for its user-space networking, and it needs to have the binary. Guix packages this, so fixing this is as simple as adding

(substitute-Command*
    ("slirp4netns" "/bin/slirp4netns"))
    
(inputs
    ...
    slirp4netns)

to the build. Now our build will replace /bin/slirp4netns with the appropriate store path and we'll be all good.

A few more of these, some require you to figure out where a command comes from. id for example - this comes from shadow so you need to pass in that as well.

How about vpnkit though? This is a docker-provided tool for... something I didn't look. But it can be called if slirp4netns isn't found. But it WILL be found, because we built it in. So I guess we can tell vpnkit to go die in a ditch!

("vpnkit" "/dev/null")

Bye :) You will not be missed. This is safe actually because we've ensured that we'll never reach this branch...

But yes we just loop this process. Run build. Check message. Patch in extra binaries if needed.

Once I had a successful v23 build, I jumped to v25. Same process again, add in a few deps blah blah. v25 working!

Let's try v27... oops this one calls out to buildkit. Some other random docker binary thing that looks like hell to build, so let's not do that! v25 is new enough that it works with the tests I wanted to run, wahoo! Like I could take the time to figure out how buildkit is... built, but it's non-obvious. And I'm gonna be honest here this has taken too much time already, so I'll call it quits here.

Should I contribute this upstream? MAYBE! I won't though because I am self-conscious about being stupid and doing stupid things and also I disabled all tests to make this work and I don't want to fix that.

Sorry GNU people.

So right, I got docker working, everything runs... but...

What i'm gonna do

I think for modern corporate development (sniff sniff) nix is the play right now. Honestly Guix's fundamentals are way better, and scheme is a much better language to express your configuration in than a subject-to-change DSL.

But currently Guix suffers the same fate as many GNU projects - the pains of getting it to work in a modern way are just too great to justify, as much as I think it's a fantastic project.

Also I wrote this post in Emacs. Somewhere along the way I decided yknow what let's also switch my editor at the same time as my OS. I'm a 15-year-long vim user, so this feels like a betrayal of sorts... evil mode my beloved. So at least I got something to take with me from my foray into GNU-land, a new personality disorder disguised as a text editor!

If anyone reading this feels the urge to try out Guix, I've shoved my working config over on my forgejo so you can yoink concepts or whatever from it yesyes.