Using Guix despite uncooperative HPC admins—the ultimate solution
Published by Arun Isaac on
In other languages: தமிழ்
Guix is extremely useful on HPC. But often, your friendly HPC admins are uncooperative. guix packs are an idea, but they only work from /gnu/store. There are ways to make relocatable packs using -R, -RR, etc. But, when these hacks do not work, there is an ultimate solution—rebuild all packages from scratch to work from your HPC home folder. Here’s how.
A guix-daemon with a different store and state directory
First, create the path you want on your HPC (say, /home/hpcuser) on your own machine.
$ sudo mkdir /home/hpcuser
Then, build a guix-daemon that puts its store and state directories in that path. To do so, use the following guix-daemon.scm. Put your path of choice in the %relocate-path variable.
(use-modules (gnu packages package-management)
(guix gexp)
(guix packages)
(guix utils))
(define %relocate-path
"/home/hpcuser")
(package
(inherit guix)
(arguments
(substitute-keyword-arguments (package-arguments guix)
;; Skip tests to save time.
((#:tests? tests? #f)
#f)
;; Set store and state directories to be inside %relocate-path.
((#:configure-flags flags '())
#~(cons #$(string-append "--with-store-dir=" %relocate-path "/store")
(map (lambda (flag)
(if (string=? flag "--localstatedir=/var")
#$(string-append "--localstatedir=" %relocate-path "/var")
flag))
#$flags))))))
Build and run the guix-daemon.
$ sudo $(guix build -f guix-daemon.scm)/bin/guix-daemon --build-users-group=guixbuild --listen=/tmp/guix-daemon-socket --no-substitutes
Use the new guix-daemon to build yourself a guix pack
Use the new guix-daemon to build yourself a guix pack; no need to use -R, -RR, etc.
(use-modules ((gnu packages base) #:select (hello))
(guix build-system)
(guix packages)
(guix utils))
;; Skip tests to save time.
(define (package-without-tests p)
;; Some build systems (raw and trivial) do not support a #:tests?
;; argument. So, don't modify the package.
(if (memq (build-system-name (package-build-system p))
(list 'raw 'trivial))
p
;; For all other packages, disable tests.
(package/inherit p
(arguments
(substitute-keyword-arguments (package-arguments p)
((#:tests? _ #f) #f))))))
(packages->manifest
(map (package-mapping package-without-tests)
(list hello)))
With the above manifest.scm, run guix pack like so.
$ NIX_STORE_DIR=/home/hpcuser/store GUIX_DAEMON_SOCKET=/tmp/guix-daemon-socket guix pack -S /home/hpcuser/bin=bin -m manifest.scm
Now, this guix pack will work on our HPC, specifically from the one path that we chose. This will work no matter what!
$ scp /home/hpcuser/store/…-guix-pack.tar.gz hpc:~
$ ssh hpc
[hpc]~$ tar --strip-components=3 xf …-guix-pack.tar.gz
[hpc]~$ ./bin/hello
Hello, world!
Bonus: Save more build time!
Even though the above manifest.scm will work, we can save a little more build time with the following.
(use-modules ((gnu compression) #:select (compressor))
((gnu packages base) #:select (hello tar))
((gnu packages compression) #:select (gzip))
(guix build-system)
(guix gexp)
(guix monads)
(guix packages)
(guix profiles)
(guix scripts pack)
(guix store)
(guix utils))
(define %relocate-path
"/home/hpcuser")
;; Skip tests to save time.
(define (package-without-tests p)
;; Some build systems (raw and trivial) do not support a #:tests?
;; argument. So, don't modify the package.
(if (memq (build-system-name (package-build-system p))
(list 'raw 'trivial))
p
;; For all other packages, disable tests.
(package/inherit p
(arguments
(substitute-keyword-arguments (package-arguments p)
((#:tests? _ #f) #f))))))
(define hpc-profile
(profile
(content (packages->manifest
(map (package-mapping package-without-tests)
(list hello))))
;; Disable hooks and locales? to save on package building. Hope
;; that these are irrelevant to us.
(hooks '())
(locales? #f)
(allow-collisions? #t)))
(with-store store
(run-with-store store
(with-monad %store-monad
(>>= (self-contained-tarball "hpc-pack"
hpc-profile
#:compressor (compressor "gzip" ".gz"
#~(list #+(file-append (package-without-tests gzip)
"/bin/gzip")
"-9n"))
#:symlinks `((,(string-append %relocate-path "/bin")
->
"/bin"))
#:archiver (package-without-tests tar))
(lambda (drv)
(return drv))))))
Put this in hpc-pack.scm and build like so.
$ NIX_STORE_DIR=/home/hpcuser/store GUIX_DAEMON_SOCKET=/tmp/guix-daemon-socket guix build -f hpc-pack.scm
A different approach to avoid rebuilding the world
Rebuilding all packages is a waste of time and energy. There is a way to avoid this however—download Guix substitutes and patch them replacing the /gnu/store path with a path of our choice. Patching binaries this way is not bullet-proof. It is hard to support all kinds of binary files. But, it is another worthy choice. Pjotr describes this further.
Acknowledgment
Thanks to Christopher Baines for suggesting this technique in a Guix London meetup!
All content is licensed under a