Using Guix despite uncooperative HPC admins—the ultimate solution
Published by Arun Isaac on
In other languages: தமிழ்
Guix is extremely useful on HPC. But often, your friendly HPC admins are uncooperative. guix packs are an idea, but they only work from /gnu/store. There are ways to make relocatable packs using -R, -RR, etc. But, when these hacks do not work, there is an ultimate solution—rebuild all packages from scratch to work from your HPC home folder. Here's how.
A guix-daemon with a different store and state directory
First, create the path you want on your HPC (say, /home/hpcuser) on your own machine.
$ sudo mkdir /home/hpcuser
Then, build a guix-daemon that puts its store and state directories in that path. To do so, use the following guix-daemon.scm. Put your path of choice in the %relocate-path variable.
(use-modules (gnu packages package-management)
(guix gexp)
(guix packages)
(guix utils))
(define %relocate-path
"/home/hpcuser")
(package
(inherit guix)
(arguments
(substitute-keyword-arguments (package-arguments guix)
;; Skip tests to save time.
((#:tests? tests? #f)
#f)
;; Set store and state directories to be inside %relocate-path.
((#:configure-flags flags '())
#~(cons #$(string-append "--with-store-dir=" %relocate-path "/store")
(map (lambda (flag)
(if (string=? flag "--localstatedir=/var")
#$(string-append "--localstatedir=" %relocate-path "/var")
flag))
#$flags))))))
Build and run the guix-daemon.
$ sudo $(guix build -f guix-daemon.scm)/bin/guix-daemon --build-users-group=guixbuild --listen=/tmp/guix-daemon-socket --no-substitutes
Use the new guix-daemon to build yourself a guix pack
Use the new guix-daemon to build yourself a guix pack; no need to use -R, -RR, etc.
(use-modules ((gnu packages base) #:select (hello)) (guix build-system) (guix packages) (guix utils)) ;; Skip tests to save time. (define (package-without-tests p) ;; Some build systems (raw and trivial) do not support a #:tests? ;; argument. So, don't modify the package. (if (memq (build-system-name (package-build-system p)) (list 'raw 'trivial)) p ;; For all other packages, disable tests. (package/inherit p (arguments (substitute-keyword-arguments (package-arguments p) ((#:tests? _ #f) #f)))))) (packages->manifest (map (package-mapping package-without-tests) (list hello)))
With the above manifest.scm, run guix pack like so.
$ NIX_STORE_DIR=/home/hpcuser/store GUIX_DAEMON_SOCKET=/tmp/guix-daemon-socket guix pack -S /home/hpcuser/bin=bin -m manifest.scm
Now, this guix pack will work on our HPC, specifically from the one path that we chose. This will work no matter what!
$ scp /home/hpcuser/store/…-guix-pack.tar.gz hpc:~ $ ssh hpc [hpc]~$ tar --strip-components=3 xf …-guix-pack.tar.gz [hpc]~$ ./bin/hello Hello, world!
Bonus: Save more build time!
Even though the above manifest.scm will work, we can save a little more build time with the following.
(use-modules ((gnu compression) #:select (compressor)) ((gnu packages base) #:select (hello tar)) ((gnu packages compression) #:select (gzip)) (guix build-system) (guix gexp) (guix monads) (guix packages) (guix profiles) (guix scripts pack) (guix store) (guix utils)) (define %relocate-path "/home/hpcuser") ;; Skip tests to save time. (define (package-without-tests p) ;; Some build systems (raw and trivial) do not support a #:tests? ;; argument. So, don't modify the package. (if (memq (build-system-name (package-build-system p)) (list 'raw 'trivial)) p ;; For all other packages, disable tests. (package/inherit p (arguments (substitute-keyword-arguments (package-arguments p) ((#:tests? _ #f) #f)))))) (define hpc-profile (profile (content (packages->manifest (map (package-mapping package-without-tests) (list hello)))) ;; Disable hooks and locales? to save on package building. Hope ;; that these are irrelevant to us. (hooks '()) (locales? #f) (allow-collisions? #t))) (with-store store (run-with-store store (with-monad %store-monad (>>= (self-contained-tarball "hpc-pack" hpc-profile #:compressor (compressor "gzip" ".gz" #~(list #+(file-append (package-without-tests gzip) "/bin/gzip") "-9n")) #:symlinks `((,(string-append %relocate-path "/bin") -> "/bin")) #:archiver (package-without-tests tar)) (lambda (drv) (return drv))))))
Put this in hpc-pack.scm and build like so.
$ NIX_STORE_DIR=/home/hpcuser/store GUIX_DAEMON_SOCKET=/tmp/guix-daemon-socket guix build -f hpc-pack.scm
A different approach to avoid rebuilding the world
Rebuilding all packages is a waste of time and energy. There is a way to avoid this however—download Guix substitutes and patch them replacing the /gnu/store path with a path of our choice. Patching binaries this way is not bullet-proof. It is hard to support all kinds of binary files. But, it is another worthy choice. Pjotr describes this further.
Acknowledgment
Thanks to Christopher Baines for suggesting this technique in a Guix London meetup!
All content is licensed under a