Using Guix despite uncooperative HPC admins—the ultimate solution

Using Guix despite uncooperative HPC admins—the ultimate solution

Published by Arun Isaac on

In other languages: தமிழ்

Tags: guix, software

How do you use Guix on HPC when your friendly HPC admins are uncooperative? This is the ultimate solution.

Guix is extremely useful on HPC. But often, your friendly HPC admins are uncooperative. guix packs are an idea, but they only work from /gnu/store. There are ways to make relocatable packs using -R, -RR, etc. But, when these hacks do not work, there is an ultimate solution—rebuild all packages from scratch to work from your HPC home folder. Here's how.

A guix-daemon with a different store and state directory

First, create the path you want on your HPC (say, /home/hpcuser) on your own machine.

$ sudo mkdir /home/hpcuser

Then, build a guix-daemon that puts its store and state directories in that path. To do so, use the following guix-daemon.scm. Put your path of choice in the %relocate-path variable.

(use-modules (gnu packages package-management)
             (guix gexp)
             (guix packages)
             (guix utils))

(define %relocate-path
  "/home/hpcuser")

(package
  (inherit guix)
  (arguments
   (substitute-keyword-arguments (package-arguments guix)
     ;; Skip tests to save time.
     ((#:tests? tests? #f)
      #f)
     ;; Set store and state directories to be inside %relocate-path.
     ((#:configure-flags flags '())
      #~(cons #$(string-append "--with-store-dir=" %relocate-path "/store")
              (map (lambda (flag)
                     (if (string=? flag "--localstatedir=/var")
                         #$(string-append "--localstatedir=" %relocate-path "/var")
                         flag))
                   #$flags))))))

Build and run the guix-daemon.

$ sudo $(guix build -f guix-daemon.scm)/bin/guix-daemon --build-users-group=guixbuild --listen=/tmp/guix-daemon-socket --no-substitutes

Use the new guix-daemon to build yourself a guix pack

Use the new guix-daemon to build yourself a guix pack; no need to use -R, -RR, etc.

(use-modules ((gnu packages base) #:select (hello))
             (guix packages)
             (guix utils))

;; Skip tests to save time.
(define (package-without-tests p)
  (package/inherit p
    (arguments
     (substitute-keyword-arguments (package-arguments p)
       ((#:tests? _ #f) #f)))))

(packages->manifest
 (map (package-mapping package-without-tests)
      (list hello)))

With the above manifest.scm, run guix pack like so.

$ NIX_STORE_DIR=/home/hpcuser/store GUIX_DAEMON_SOCKET=/tmp/guix-daemon-socket guix pack -S /home/hpcuser/bin=bin -m manifest.scm

Now, this guix pack will work on our HPC, specifically from the one path that we chose. This will work no matter what!

$ scp /home/hpcuser/store/…-guix-pack.tar.gz hpc:~
$ ssh hpc
[hpc]~$ tar --strip-components=3 xf …-guix-pack.tar.gz
[hpc]~$ ./bin/hello
Hello, world!

Bonus: Save more build time!

Even though the above manifest.scm will work, we can save a little more build time with the following.

(use-modules ((gnu compression) #:select (compressor))
             ((gnu packages base) #:select (hello tar))
             ((gnu packages compression) #:select (gzip))
             (guix gexp)
             (guix monads)
             (guix packages)
             (guix profiles)
             (guix scripts pack)
             (guix store)
             (guix utils))

(define %relocate-path
  "/home/hpcuser")

;; Skip tests to save time.
(define (package-without-tests p)
  (package/inherit p
    (arguments
     (substitute-keyword-arguments (package-arguments p)
       ((#:tests? _ #f) #f)))))

(define hpc-profile
  (profile
   (content (packages->manifest
             (map (package-mapping package-without-tests)
                  (list hello))))
   ;; Disable hooks and locales? to save on package building. Hope
   ;; that these are irrelevant to us.
   (hooks '())
   (locales? #f)
   (allow-collisions? #t)))

(with-store store
  (run-with-store store
    (with-monad %store-monad
      (>>= (self-contained-tarball "hpc-pack"
                                   hpc-profile
                                   #:compressor (compressor "gzip"  ".gz"
                                                            #~(list #+(file-append (package-without-tests gzip)
                                                                                   "/bin/gzip")
                                                                    "-9n"))
                                   #:symlinks `((,(string-append %relocate-path "/bin")
                                                 ->
                                                 "/bin"))
                                   #:archiver (package-without-tests tar))
           (lambda (drv)
             (return drv))))))

Put this in hpc-pack.scm and build like so.

$ NIX_STORE_DIR=/home/hpcuser/store GUIX_DAEMON_SOCKET=/tmp/guix-daemon-socket guix build -f hpc-pack.scm

A different approach to avoid rebuilding the world

Rebuilding all packages is a waste of time and energy. There is a way to avoid this however—download Guix substitutes and patch them replacing the /gnu/store path with a path of our choice. Patching binaries this way is not bullet-proof. It is hard to support all kinds of binary files. But, it is another worthy choice. Pjotr describes this further.

Acknowledgment

Thanks to Christopher Baines for suggesting this technique in a Guix London meetup!