Deploy scripts using G-expressions
Published by Arun Isaac on
In other languages: தமிழ்
Tags: lisp, scheme, guix, software
When deploying scripts to remote machines, one also needs to install all the necessary dependencies. These dependencies need to keep up-to-date and of the right version. When some dependencies are no longer needed, they need to be uninstalled. This is a pain and getting things wrong is a common problem. Guix G-expressions can be used to alleviate the problem. Let's see how.
Deploy a pure Guile script using a G-expression
Web users and other software often use htpasswd files to store usernames and passwords. We may use the following bash script to add and delete users from such a htpasswd file.
passwd_file=/etc/web/passwd case $1 in add) htpasswd $passwd_file $2 ;; delete) htpasswd -D $passwd_file $2 ;; esac
We may rewrite this bash script in Guile as a Guix G-expression. Noteworthy below are the use of file-append and program-file. file-append is used to refer to the /bin/htpasswd file of the httpd package. program-file is used to intern our G-expression into the Guix store.
(use-modules (gnu packages web)
(guix gexp))
(define webuser-gexp
(with-imported-modules '((guix build utils))
#~(begin
(use-modules (guix build utils)
(ice-9 match))
(define passwd-file "/etc/web/passwd")
(match (program-arguments)
((_ "add" user)
(invoke #$(file-append httpd "/bin/htpasswd")
passwd-file user))
((_ "delete" user)
(invoke #$(file-append httpd "/bin/htpasswd")
"-D" passwd-file user))))))
(program-file "webuser" webuser-gexp)
Say this is written into a file webuser.scm, we may build it using
$ guix build -f webuser.scm /gnu/store/1d8qs814gi1vgf52n7x0s43ix6rc5qzc-webuser
Let's see what is in the built store item.
$ cat /gnu/store/1d8qs814gi1vgf52n7x0s43ix6rc5qzc-webuser
#!/gnu/store/cnfsv9ywaacyafkqdqsv2ry8f01yr7a9-guile-3.0.7/bin/guile --no-auto-compile !# (eval-when (expand load eval) (let ((extensions (quote ())) (prepend (lambda (items lst) (let loop ((items items) (lst lst)) (if (null? items) lst (loop (cdr items) (cons (car items) (delete (car items) lst)))))))) (set! %load-path (prepend (cons "/gnu/store/pgj8653w17hsapbd1srlvd44rlnhbx8n-module-import" (map (lambda (extension) (string-append extension "/share/guile/site/" (effective-version))) extensions)) %load-path)) (set! %load-compiled-path (prepend (cons "/gnu/store/fwzac0aal1y24rx2vn8yhw6hf45kk46j-module-import-compiled" (map (lambda (extension) (string-append extension "/lib/guile/" (effective-version) "/site-ccache")) extensions)) %load-compiled-path)))) (begin (use-modules (guix build utils) (ice-9 match)) (define passwd-file "/etc/web/passwd") (match (program-arguments) ((_ "add" user) (invoke "/gnu/store/xkmpq6vwhml48bwfn2fxjdvqfmzf9ccb-httpd-2.4.52/bin/htpasswd" passwd-file user)) ((_ "delete" user) (invoke "/gnu/store/xkmpq6vwhml48bwfn2fxjdvqfmzf9ccb-httpd-2.4.52/bin/htpasswd" "-D" passwd-file user))))
The first paragraph sets up paths to import various Guile modules. The second paragraph is our G-expression only that the file-append calls are replaced with absolute paths to htpasswd.
To deploy this to a remote machine, install it at /usr/local/bin/webuser and protect it from garbage collection by registering it as a garbage collector root, we run
$ guix copy --to=remote /gnu/store/1d8qs814gi1vgf52n7x0s43ix6rc5qzc-webuser $ ssh remote ln --symbolic --force /gnu/store/1d8qs814gi1vgf52n7x0s43ix6rc5qzc-webuser /usr/local/bin/webuser $ ssh remote ln --symbolic --force /usr/local/bin/webuser /var/guix/gcroots
Here, remote is a machine that has been configured to accept guix copy requests. See guix copy documentation for more.
So, what is the use of doing all this? The G-expression we installed on the remote machine didn't go alone. It went carrying along the htpasswd utility and all the other required dependencies. In total, this amounted to 407.9 MiB!
$ guix size /gnu/store/1d8qs814gi1vgf52n7x0s43ix6rc5qzc-webuser store item total self /gnu/store/hy6abswwv4d89zp464fw52z65fkzr7h5-perl-5.34.0 147.7 58.6 14.4% /gnu/store/rc781v4k0drhaqn90xfwwpspki5x0bvf-binutils-2.37 92.7 54.4 13.3% /gnu/store/cnfsv9ywaacyafkqdqsv2ry8f01yr7a9-guile-3.0.7 129.1 52.0 12.8% /gnu/store/1kws5vkl0glvpxg7arabsv6q9vazp0hx-guile-3.0.7 129.1 52.0 12.8% /gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33 38.3 36.6 9.0% /gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib 71.7 33.4 8.2% /gnu/store/xkmpq6vwhml48bwfn2fxjdvqfmzf9ccb-httpd-2.4.52 350.1 29.7 7.3% /gnu/store/vqdsrvs9jbn0ix2a58s99jwkh74124y5-coreutils-minimal-8.32 88.0 16.4 4.0% /gnu/store/d251rfgc9nm2clzffzhgiipdvfvzkvwi-coreutils-8.32 88.0 16.4 4.0% /gnu/store/fnr1z6xsan0437r0yg48d0y8k32kqxby-glibc-utf8-locales-2.33 13.9 13.9 3.4% /gnu/store/4jdghmc65q7i7ib89zmvq66l0ghf7jc4-glibc-2.33-static 46.9 8.6 2.1% /gnu/store/4ic6244i3ca4b4rxc2wnrgllsidyishv-file-5.39 78.2 6.6 1.6% /gnu/store/lk3ywzavgz30xrlfcmx2x9rfz3cs7xq6-openssl-1.1.1s 77.2 5.5 1.4% /gnu/store/690qz3fg334dpwn3pn6k59n4wc943p2b-gawk-5.1.0 76.0 3.3 0.8% /gnu/store/hkhbq2q1gfs970gsp2nhsmcqb4vmv2xr-libunistring-0.9.10 74.0 2.3 0.6% /gnu/store/ckh3gy7gpvd4b65s6jsm6f1y9bp460ji-libunistring-0.9.10 74.0 2.3 0.6% /gnu/store/720rj90bch716isd8z7lcwrnvz28ap4y-bash-static-5.1.8 1.7 1.7 0.4% /gnu/store/di5bqb45hi5lvp2q08hlxqjdcl9phjb1-pcre-8.45 73.4 1.7 0.4% /gnu/store/rbb9h501zyf8mg1hz47plql80gsl99za-apr-1.7.0 312.1 1.5 0.4% /gnu/store/2b3blhwbag1ial0dhxw7wh4zjxl0cqpk-pkg-config-0.29.2 72.8 1.1 0.3% /gnu/store/lk24spr6hbkzh68s79nzqp9z36nx0m1f-pkg-config-0.29.2 72.8 1.1 0.3% /gnu/store/c8isj4jq6knv0icfgr43di6q3nvdzkx7-xz-5.2.5 73.7 1.1 0.3% /gnu/store/chfwin3a4qp1znnpsjbmydr2jbzk0d6y-bash-minimal-5.1.8 72.7 1.0 0.2% /gnu/store/4y5m9lb8k3qkb1y9m02sw9w9a6hacd16-bash-minimal-5.1.8 39.3 1.0 0.2% /gnu/store/xjwp2hsd9256icjjybfrmznppjicywf6-grep-3.6 73.5 0.8 0.2% /gnu/store/wxgv6i8g0p24q5gcyzd0yr07s8kn9680-sed-4.8 72.5 0.8 0.2% /gnu/store/r3lv5k4mxaz53f4sr4wf9dqkqadcpms6-apr-util-1.6.1 313.2 0.8 0.2% /gnu/store/2lczkxbdbzh4gk7wh91bzrqrk7h5g1dl-libgc-8.0.4 72.4 0.7 0.2% /gnu/store/3pylba5sjy3r7b8fjm9yxz24751721ff-libgc-8.0.4 72.4 0.7 0.2% /gnu/store/s3hl12jxz9ybs7nsy7kq7ybzz7qnzmsg-bzip2-1.0.8 73.1 0.4 0.1% /gnu/store/52zhpralb3iimrm7xbc1vf3qsy4gy1vl-expat-2.4.9 72.0 0.4 0.1% /gnu/store/fwzac0aal1y24rx2vn8yhw6hf45kk46j-module-import-compiled 0.3 0.3 0.1% /gnu/store/wgqhlc12qvlwiklam7hz2r311fdcqfim-libffi-3.3 71.8 0.2 0.0% /gnu/store/g000a71kc336795axa3hh1xhd3mfq083-libffi-3.3 71.8 0.2 0.0% /gnu/store/2i5alw7qcp35x0rn0yqxmvxv3pd6ln3w-libltdl-2.4.6 71.8 0.1 0.0% /gnu/store/s2pg5k98fl2g2szg9dykxyd9zl3xihv9-ld-wrapper-0 183.5 0.1 0.0% /gnu/store/k04pcgdvdipdc37cc6xvm33pcglbb8rz-libsigsegv-2.13 71.7 0.1 0.0% /gnu/store/pgj8653w17hsapbd1srlvd44rlnhbx8n-module-import 0.1 0.1 0.0% /gnu/store/1d8qs814gi1vgf52n7x0s43ix6rc5qzc-webuser 407.9 0.0 0.0% total: 407.9 MiB
So, using G-expressions, we need only concern ourselves with installing the G-expression alone. Guix automatically takes care of the required dependencies. If any dependencies that are required today are no longer required in the future, no sweat—they will automatically be erased in the next guix gc run.
Deploy scripts written in other languages
Deploy a bash script using a G-expression
Don't like Guile and would prefer to write your scripts in bash? No problem! G-expressions can deploy bash scripts too. To do that, we must wrap our bash script in a G-expression. Here, we show a G-expression wrapping a bash script webuser.sh. Noteworthy here is local-file used to intern webuser.sh into the Guix store, and set-path-environment-variable used to make htpasswd visible on PATH.
(use-modules (gnu packages bash)
(gnu packages web)
(guix gexp))
(define webuser-gexp
(with-imported-modules '((guix build utils))
#~(begin
(use-modules (guix build utils)
(ice-9 match))
(set-path-environment-variable
"PATH" (list "bin") (list #$httpd))
(apply invoke
#$(file-append bash "/bin/bash")
#$(local-file "webuser.sh")
(match (program-arguments)
((arg0 . rest-args)
rest-args))))))
(program-file "webuser" webuser-gexp)
Deploy a python script using a G-expression
bash is not special, and we can indeed deploy scripts written in other language such as python too! Here is an example.
(use-modules (gnu packages python)
(guix gexp))
(define foo-gexp
(with-imported-modules '((guix build utils))
#~(begin
(use-modules (guix build utils))
(invoke #$(file-append python "/bin/python3")
#$(local-file "foo.py")))))
(program-file "foo" foo-gexp)
Deploy a python script with dependencies using a G-expression
If our python script depends on other python packages, that can be handled with ease too. Here is an example that pulls in python-colorama.
(use-modules (gnu packages python)
(gnu packages python-xyz)
(guix packages)
(guix gexp)
(guix utils))
(define foo-gexp
(with-imported-modules '((guix build utils))
#~(begin
(use-modules (guix build utils))
(set-path-environment-variable
"GUIX_PYTHONPATH"
(list #$(string-append "lib/python"
(version-major+minor (package-version python))
"/site-packages"))
(list #$python-colorama))
(invoke #$(file-append python "/bin/python3")
#$(local-file "foo-color.py")))))
(program-file "foo" foo-gexp)
Likewise, it is possible to deploy scripts written in Emacs Lisp, Rust, Ruby or indeed any other language of your heart's desire. This is the great unifying power of Guix and G-expressions that cuts across traditional language boundaries.
Have a nice day!
All content is licensed under a