https://guix.gnu.org/feeds/blog/continuous-integration.atomGNU Guix — Blog — Continuous integrationfeed author nameGNU Guixhttps://guix.gnu.org/themes/initial/img/icon.png2024-03-20T10:57:56Zhttps://guix.gnu.org/blog/2023/a-new-quality-assurance-tool-for-guix//A new Quality Assurance tool for GuixChristopher Baines2023-09-12T14:30:00Z2023-09-12T14:30:00Z Maintaining and expanding Guix's collection of packages can be
complicated. As a distribution with around 22,000 packages, spanning
across around 7 architectures and with support for cross-compilation,
it's quite common for problems to occur when making changes. Quality Assurance (QA) is a general term to describe the approach
taken to try and ensure something meets expectations. When applied to
software, the term testing is normally used. While Guix is software,
and has tests, much more than those tests are needed to maintain Guix
as a distribution. So what might quality relate to in the context of Guix as a
distribution? …<p>Maintaining and expanding Guix's collection of packages can be
complicated. As a distribution with around 22,000 packages, spanning
across around 7 architectures and with support for cross-compilation,
it's quite common for problems to occur when making changes.</p><p>Quality Assurance (QA) is a general term to describe the approach
taken to try and ensure something meets expectations. When applied to
software, the term testing is normally used. While Guix is software,
and has tests, much more than those tests are needed to maintain Guix
as a distribution.</p><p>So what might quality relate to in the context of Guix as a
distribution? This will differ from person to person, but these are
some common concerns:</p><ul><li>Packages successfully building (both now, and without any
<a href="https://issues.guix.gnu.org/56137"><em>time bombs</em></a> for the future)</li><li>The packaged software functioning correctly</li><li>Packages building on or for a specific architecture</li><li>Packages building reproducibly</li><li>Availability of translations for the package definitions</li></ul><h1>Tooling to help with Quality Assurance</h1><p>There's a range of tools to help maintain Guix. The <a href="https://guix.gnu.org/en/manual/en/html_node/Invoking-guix-lint.html">package
linters</a>
are a set of simple tools, they cover basic things from the naming of
packages to more complicated checkers that look for security issues
for example.</p><p>The <a href="https://guix.gnu.org/en/manual/en/html_node/Invoking-guix-weather.html"><code>guix weather</code></a>
tool looks at substitute availability information and can indicate how
many substitutes are available for the current Guix and system. The
<a href="https://guix.gnu.org/en/manual/en/html_node/Invoking-guix-challenge.html"><code>guix challenge</code></a>
tool is similar, but it highlights package reproducibility issues,
which is when the substitutes and local store items (if available)
differ.</p><p>For translations, <a href="https://guix.gnu.org/manual/en/html_node/Translating-Guix.html">Guix uses
Weblate</a>
which can provide information on how many translations are available.</p><h1>The QA front-page</h1><p>Then there's the relatively new <a href="https://qa.guix.gnu.org/">Quality Assurance (QA)
front-page</a>, the aim of which is to bring
together some of the existing Quality Assurance related information,
as well as new being a good place to do additional QA tasks.</p><p>The QA front-page
<a href="https://lists.gnu.org/archive/html/guix-devel/2022-09/msg00054.html">started</a>
as a service to coordinate automated testing for patches. When a patch
or patch series is submitted to guix-patches@gnu.org, it is
automatically applied to create a branch; then once the information is
available from the <a href="https://data.qa.guix.gnu.org/">Data Service</a> about
this branch, the QA front-page web interface lets you view which
packages were modified and submits builds for these changes to the
<a href="https://guix.gnu.org/en/blog/2021/building-derivations-how-complicated-can-it-be/">Build Coordinator</a>
behind <a href="https://guix.gnu.org/en/blog/2021/substitutes-now-also-available-from-bordeauxguixgnuorg/">bordeaux.guix.gnu.org</a>
to provide build information about the modified packages.</p><p><img src="/static/blog/img/qa-issue.png" alt="QA issue page" /></p><p>A very similar process applies for branches other than the master
branch, the QA front-page queries
<a href="https://issues.guix.gnu.org/">issues.guix.gnu.org</a> to find out which
branch is going to be merged next, then follows the same process for
patches.</p><p>For both patches and branches the QA front-page displays information
about the effects of the changes. When this information is available,
it can assist with reviewing the changes and help get patches merged
quicker. This is a work in progress though, and there's much more that
the QA front-page should be able to do as providing clearer
descriptions of the changes or any other problems that should be
addressed.
<img src="/static/blog/img/qa-package-changes.png" alt="QA package changes page" /></p><h1>How to get involved?</h1><p>There's plenty of ways to get involved or contribute to the QA
front-page.</p><p>If you submit patches to Guix, the QA front-page will attempt to apply
the patches and show what's changed. You can click through from
issues.guix.gnu.org to <a href="https://qa.guix.gnu.org/">qa.guix.gnu.org</a> via
the QA badge by the status of the issue.</p><p>From the QA front-page, you can also view the list of branches which
includes the requests for merging if they exist. Similar to the patch
series, for the branch the QA front-page can display information about
the package changes and substitute availability.</p><p>There's also plenty of ways to contribute to the QA front-page and
connected tools. You can find some ideas and information on how to
run the service in the <a href="https://qa.guix.gnu.org/README">README</a> and if
you have any questions or patches, please email <code>guix-devel@gnu.org</code>.</p><h1>Acknowledgments</h1><p>Thanks to Simon Tournier and Ludovic Courtès for providing feedback on
an earlier draft of this post.</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package manager
and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects user
freedom</a>.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system
distribution for i686, x86_64, ARMv7, AArch64 and POWER9 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package
management, per-user profiles, and garbage collection. When used as a
standalone GNU/Linux distribution, Guix offers a declarative,
stateless approach to operating system configuration management. Guix
is highly customizable and hackable through
<a href="https://www.gnu.org/software/guile">Guile</a> programming interfaces and
extensions to the <a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2023/from-development-environments-to-continuous-integrationthe-ultimate-guide-to-software-development-with-guix//From development environments to continuous integration—the ultimate guide to software development with GuixLudovic Courtès2023-06-05T16:30:00Z2023-06-05T16:30:00Z Guix is a handy tool for developers; guix shell ,
in particular, gives a standalone development environment for your
package, no matter what language(s) it’s written in. To benefit from
it, you have to initially write a package definition and have it either
in Guix proper, in a channel, or directly upstream as a guix.scm file.
This last option is appealing: all developers have to do to get set up
is clone the project's repository and run guix shell , with no
arguments—we looked at the rationale for guix shell in an earlier
article . …<p>Guix is a handy tool for developers; <a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-shell.html"><code>guix shell</code></a>,
in particular, gives a standalone development environment for your
package, no matter what language(s) it’s written in. To benefit from
it, you have to initially write a package definition and have it either
in Guix proper, in a channel, or directly upstream as a <code>guix.scm</code> file.
This last option is appealing: all developers have to do to get set up
is clone the project's repository and run <code>guix shell</code>, with no
arguments—we looked at the rationale for <code>guix shell</code> in <a href="https://guix.gnu.org/en/blog/2021/from-guix-environment-to-guix-shell/">an earlier
article</a>.</p><p>Development needs go beyond development environments though. How can
developers perform continuous integration of their code in Guix build
environments? How can they deliver their code straight to adventurous
users? This post describes a set of files developers can add
to their repository to set up Guix-based development
environments, continuous integration, and continuous delivery—all at
once.</p><blockquote><p><strong>Update</strong>: Check out <a href="https://guix.gnu.org/cookbook/en/html_node/Software-Development.html">the
Cookbook</a>
for an up-to-date and translated version of this tutorial!</p></blockquote><h1>Getting started</h1><p>How do we go about “Guixifying” a repository? The first step, as we’ve
seen, will be to add a <code>guix.scm</code> at the root of the repository in
question. We’ll take <a href="https://www.gnu.org/software/guile">Guile</a>
as an example in this post: it’s written in Scheme (mostly) and C, and
has a number of dependencies—a C compilation tool chain, C libraries,
Autoconf and its friends, LaTeX, and so on. The resulting <code>guix.scm</code>
looks like the usual <a href="https://guix.gnu.org/manual/devel/en/html_node/Defining-Packages.html">package
definition</a>,
just without the <code>define-public</code> bit:</p><pre><code class="language-scheme">;; The ‘guix.scm’ file for Guile, for use by ‘guix shell’.
(use-modules (guix)
(guix build-system gnu)
((guix licenses) #:prefix license:)
(gnu packages autotools)
(gnu packages base)
(gnu packages bash)
(gnu packages bdw-gc)
(gnu packages compression)
(gnu packages flex)
(gnu packages gdb)
(gnu packages gettext)
(gnu packages gperf)
(gnu packages libffi)
(gnu packages libunistring)
(gnu packages linux)
(gnu packages pkg-config)
(gnu packages readline)
(gnu packages tex)
(gnu packages texinfo)
(gnu packages version-control))
(package
(name "guile")
(version "3.0.99-git") ;funky version number
(source #f) ;no source
(build-system gnu-build-system)
(native-inputs
(append (list autoconf
automake
libtool
gnu-gettext
flex
texinfo
texlive-base ;for "make pdf"
texlive-epsf
gperf
git
gdb
strace
readline
lzip
pkg-config)
;; When cross-compiling, a native version of Guile itself is
;; needed.
(if (%current-target-system)
(list this-package)
'())))
(inputs
(list libffi bash-minimal))
(propagated-inputs
(list libunistring libgc))
(native-search-paths
(list (search-path-specification
(variable "GUILE_LOAD_PATH")
(files '("share/guile/site/3.0")))
(search-path-specification
(variable "GUILE_LOAD_COMPILED_PATH")
(files '("lib/guile/3.0/site-ccache")))))
(synopsis "Scheme implementation intended especially for extensions")
(description
"Guile is the GNU Ubiquitous Intelligent Language for Extensions,
and it's actually a full-blown Scheme implementation!")
(home-page "https://www.gnu.org/software/guile/")
(license license:lgpl3+))</code></pre><p>Quite a bit of boilerplate, but now someone who’d like to hack on Guile
just needs to run:</p><pre><code>guix shell</code></pre><p>That gives them a shell containing all the dependencies of Guile: those
listed above, but also <em>implicit dependencies</em> such as the GCC tool
chain, GNU Make, sed, grep, and so on. The chef’s recommendation:</p><pre><code>guix shell --container --link-profile</code></pre><p>That gives a shell in an isolated container, and all the dependencies
show up in <code>$HOME/.guix-profile</code>, which plays well with caches such as
<a href="https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.70/html_node/Cache-Files.html"><code>config.cache</code></a>
and absolute file names recorded in generated <code>Makefile</code>s and the likes.
The fact that the shell runs in a container brings peace of mind:
nothing but the current directory and Guile’s dependencies is visible
inside the container; nothing from the system can possibly interfere
with your development.</p><h1>Level 1: Building with Guix</h1><p>Now that we have a package definition, why not also take advantage of it
so we can build Guile with Guix? We had left the <code>source</code> field empty,
because <code>guix shell</code> above only cares about the <em>inputs</em> of our
package—so it can set up the development environment—not about the
package itself.</p><p>To build the package with Guix, we’ll need to fill out the <code>source</code>
field, along these lines:</p><pre><code class="language-scheme">(use-modules (guix)
(guix git-download) ;for ‘git-predicate’
…)
(define vcs-file?
;; Return true if the given file is under version control.
(or (git-predicate (current-source-directory))
(const #t))) ;not in a Git checkout
(package
(name "guile")
(version "3.0.99-git") ;funky version number
(source (local-file "." "guile-checkout"
#:recursive? #t
#:select? vcs-file?))
…)</code></pre><p>Here’s what we changed:</p><ol><li>We added <code>(guix git-download)</code> to our set of imported modules, so we
can use its <code>git-predicate</code> procedure.</li><li>We defined <code>vcs-file?</code> as a procedure that returns true when passed
a file that is under version control. For good measure, we add a
fallback case for when we’re not in a Git checkout: always return
true.</li><li>We set <code>source</code> to a
<a href="https://guix.gnu.org/manual/devel/en/html_node/G_002dExpressions.html#index-local_002dfile"><code>local-file</code></a>—a
recursive copy of the current directory (<code>"."</code>), limited to files
under version control (the <code>#:select?</code> bit).</li></ol><p>From there on, our <code>guix.scm</code> file serves a second purpose: it lets us
build the software with Guix. The whole point of building with Guix is
that it’s a “clean” build—you can be sure nothing from your working tree
or system interferes with the build result—and it lets you test a
variety of things. First, you can do a plain native build:</p><pre><code>guix build -f guix.scm</code></pre><p>But you can also build for another system (possibly after setting up
<a href="https://guix.gnu.org/manual/devel/en/html_node/Daemon-Offload-Setup.html">offloading</a>
or <a href="https://guix.gnu.org/manual/devel/en/html_node/Virtualization-Services.html#index-emulation">transparent
emulation</a>):</p><pre><code>guix build -f guix.scm -s aarch64-linux -s riscv64-linux</code></pre><p>… or cross-compile:</p><pre><code>guix build -f guix.scm --target=x86_64-w64-mingw32</code></pre><p>You can also use <a href="https://guix.gnu.org/manual/devel/en/html_node/Package-Transformation-Options.html">package transformation
options</a>
to test package variants:</p><pre><code># What if we built with Clang instead of GCC?
guix build -f guix.scm \
--with-c-toolchain=guile@3.0.99-git=clang-toolchain
# What about that under-tested configure flag?
guix build -f guix.scm \
--with-configure-flag=guile@3.0.99-git=--disable-networking</code></pre><p>Handy!</p><h1>Level 2: The repository as a channel</h1><p>We now have a Git repository containing (among other things) a package
definition. Can’t we turn it into a
<a href="https://guix.gnu.org/manual/devel/en/html_node/Channels.html"><em>channel</em></a>?
After all, channels are designed to ship package definitions to users,
and that’s exactly what we’re doing with our <code>guix.scm</code>.</p><p>Turns out we can indeed turn it into a channel, but with one caveat: we
must create a separate directory for the <code>.scm</code> file(s) of our channel
so that <code>guix pull</code> doesn’t load unrelated <code>.scm</code> files when
someone pulls the channel—and in Guile, there are lots of them! So
we’ll start like this, keeping a top-level <code>guix.scm</code> symlink for the
sake of <code>guix shell</code>:</p><pre><code>mkdir -p .guix/modules
mv guix.scm .guix/modules/guile-package.scm
ln -s .guix/modules/guile-package.scm guix.scm</code></pre><p>To make it usable as part of a channel, we
need to turn our <code>guix.scm</code> file into a
<a href="https://guix.gnu.org/manual/devel/en/html_node/Package-Modules.html">module</a>:
we do that by changing the <code>use-modules</code> form at the top to a
<code>define-module</code> form. We also need to actually <em>export</em> a package
variable, with <code>define-public</code>, while still returning the package value
at the end of the file so we can still use <code>guix shell</code> and <code>guix build -f guix.scm</code>. The end result looks like this (not repeating things that
haven’t changed):</p><pre><code class="language-scheme">(define-module (guile-package)
#:use-module (guix)
#:use-module (guix git-download) ;for ‘git-predicate’
…)
(define vcs-file?
;; Return true if the given file is under version control.
(or (git-predicate (dirname (dirname (current-source-directory))))
(const #t))) ;not in a Git checkout
(define-public guile
(package
(name "guile")
(version "3.0.99-git") ;funky version number
(source (local-file "../.." "guile-checkout"
#:recursive? #t
#:select? vcs-file?))
…))
;; Return the package object define above at the end of the module.
guile</code></pre><p>We need one last thing: a <a href="https://guix.gnu.org/manual/devel/en/html_node/Package-Modules-in-a-Sub_002ddirectory.html"><code>.guix-channel</code>
file</a>
so Guix knows where to look for package modules in our repository:</p><pre><code class="language-scheme">;; This file lets us present this repo as a Guix channel.
(channel
(version 0)
(directory ".guix/modules")) ;look for package modules under .guix/modules/</code></pre><p>To recap, we now have these files:</p><pre><code>.
├── .guix-channel
├── guix.scm → .guix/modules/guile-package.scm
└── .guix
└── modules
└── guile-package.scm</code></pre><p>And that’s it: we have a channel! (We could do better and support
<a href="https://guix.gnu.org/manual/devel/en/html_node/Specifying-Channel-Authorizations.html"><em>channel
authentication</em></a>
so users know they’re pulling genuine code. We’ll spare you the details
here but it’s worth considering!) Users can pull from this channel by
<a href="https://guix.gnu.org/manual/devel/en/html_node/Specifying-Additional-Channels.html">adding it to
<code>~/.config/guix/channels.scm</code></a>,
along these lines:</p><pre><code class="language-scheme">(append (list (channel
(name 'guile)
(url "https://git.savannah.gnu.org/git/guile.git")
(branch "main")))
%default-channels)</code></pre><p>After running <code>guix pull</code>, we can see the new package:</p><pre><code>$ guix describe
Generation 264 May 26 2023 16:00:35 (current)
guile 36fd2b4
repository URL: https://git.savannah.gnu.org/git/guile.git
branch: main
commit: 36fd2b4920ae926c79b936c29e739e71a6dff2bc
guix c5bc698
repository URL: https://git.savannah.gnu.org/git/guix.git
commit: c5bc698e8922d78ed85989985cc2ceb034de2f23
$ guix package -A ^guile$
guile 3.0.99-git out,debug guile-package.scm:51:4
guile 3.0.9 out,debug gnu/packages/guile.scm:317:2
guile 2.2.7 out,debug gnu/packages/guile.scm:258:2
guile 2.2.4 out,debug gnu/packages/guile.scm:304:2
guile 2.0.14 out,debug gnu/packages/guile.scm:148:2
guile 1.8.8 out gnu/packages/guile.scm:77:2
$ guix build guile@3.0.99-git
[…]
/gnu/store/axnzbl89yz7ld78bmx72vpqp802dwsar-guile-3.0.99-git-debug
/gnu/store/r34gsij7f0glg2fbakcmmk0zn4v62s5w-guile-3.0.99-git</code></pre><p>That’s how, as a developer, you get your software delivered directly into
the hands of users! No intermediaries, yet no loss of transparency and
provenance tracking.</p><p>With that in place, it also becomes trivial for anyone to create Docker
images, Deb/RPM packages, or a plain tarball with <a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-pack.html"><code>guix pack</code></a>:</p><pre><code># How about a Docker image of our Guile snapshot?
guix pack -f docker -S /bin=bin guile@3.0.99-git
# And a relocatable RPM?
guix pack -f rpm -R -S /bin=bin guile@3.0.99-git</code></pre><h1>Bonus: Package variants</h1><p>We now have an actual channel, but it contains only one package. While
we’re at it, we can <a href="https://guix.gnu.org/manual/devel/en/html_node/Defining-Package-Variants.html">define package
variants</a>
in our <code>guile-package.scm</code> file, variants that we want to be able to
test as Guile developers—similar to what we did above with
transformation options. We can add them like so:</p><pre><code class="language-scheme">;; This is the ‘.guix/modules/guile-package.scm’ file.
(define-module (guile-package)
…)
(define-public guile
…)
(define (package-with-configure-flags p flags)
"Return P with FLAGS as additional 'configure' flags."
(package/inherit p
(arguments
(substitute-keyword-arguments (package-arguments p)
((#:configure-flags original-flags #~(list))
#~(append #$original-flags #$flags))))))
(define-public guile-without-threads
(package
(inherit (package-with-configure-flags guile
#~(list "--without-threads")))
(name "guile-without-threads")))
(define-public guile-without-networking
(package
(inherit (package-with-configure-flags guile
#~(list "--disable-networking")))
(name "guile-without-networking")))
;; Return the package object defined above at the end of the module.
guile</code></pre><p>We can build these variants as regular packages once we’ve pulled the
channel. Alternatively, from a checkout of Guile, we can run a command like
this one from the top level:</p><pre><code>guix build -L $PWD/.guix/modules guile-without-threads</code></pre><h1>Level 3: Setting up continuous integration</h1><p>This channel becomes even more interesting once we set up <a href="https://en.wikipedia.org/wiki/Continuous_integration">continuous
integration</a> (CI).
There are several ways to do that.</p><p>You can use one of the mainstream continuous integration tools, such as
GitLab-CI. To do that, you need to make sure you run jobs in a Docker
image or virtual machine that has Guix installed. If we were to do that
in the case of Guile, we’d have a job that runs a shell command like
this one:</p><pre><code>guix build -L $PWD/.guix/modules guile@3.0.99-git</code></pre><p>Doing this works great and has the advantage of being easy to achieve on
your favorite CI platform.</p><p>That said, you’ll really get the most of it by using
<a href="https://guix.gnu.org/en/cuirass">Cuirass</a>, a CI tool designed for and
tightly integrated with Guix. Using it is more work than using a hosted
CI tool because you first need to set it up, but that setup phase is
greatly simplified if you use its Guix System
<a href="https://guix.gnu.org/manual/devel/en/html_node/Continuous-Integration.html">service</a>.
Going back to our example, we give Cuirass a spec file that goes like
this:</p><pre><code class="language-scheme">;; Cuirass spec file to build all the packages of the ‘guile’ channel.
(list (specification
(name "guile")
(build '(channels guile))
(channels
(append (list (channel
(name 'guile)
(url "https://git.savannah.gnu.org/git/guile.git")
(branch "main")))
%default-channels))))</code></pre><p>It differs from what you’d do with other CI tools in two important ways:</p><ul><li>Cuirass knows it’s tracking <em>two</em> channels, <code>guile</code> and <code>guix</code>.
Indeed, our own <code>guile</code> package depends on many packages provided by
the <code>guix</code> channel—GCC, the GNU libc, libffi, and so on. Changes to
packages from the <code>guix</code> channel can potentially influence our
<code>guile</code> build and this is something we’d like to see as soon as
possible as Guile developers.</li><li>Build results are not thrown away: they can be distributed as
<a href="https://guix.gnu.org/manual/devel/en/html_node/Substitutes.html"><em>substitutes</em></a>
so that users of our <code>guile</code> channel transparently get pre-built
binaries!</li></ul><p>From a developer’s viewpoint, the end result is this <a href="https://ci.guix.gnu.org/jobset/guile">status
page</a> listing <em>evaluations</em>: each
evaluation is a combination of commits of the <code>guix</code> and <code>guile</code>
channels providing a number of <em>jobs</em>—one job per package defined in
<code>guile-package.scm</code> times the number of target architectures.</p><p>As for substitutes, they come for free! As an example, since our
<code>guile</code> jobset is built on ci.guix.gnu.org, which runs <a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-publish.html"><code>guix publish</code></a>
in addition to Cuirass, one automatically gets substitutes for <code>guile</code>
builds from ci.guix.gnu.org; no additional work is needed for that.</p><h1>Bonus: Build manifest</h1><p>The Cuirass spec above is convenient: it builds every package in our
channel, which includes a few variants. However, this might be
insufficiently expressive in some cases: one might want specific
cross-compilation jobs, transformations, Docker images, RPM/Deb
packages, or even system tests.</p><p>To achieve that, you can write a
<a href="https://guix.gnu.org/manual/devel/en/html_node/Writing-Manifests.html"><em>manifest</em></a>.
The one we have for Guile has entries for the package variants we
defined above, as well as additional variants and cross builds:</p><pre><code class="language-scheme">;; This is ‘.guix/manifest.scm’.
(use-modules (guix)
(guix profiles)
(guile-package)) ;import our own package module
(define* (package->manifest-entry* package system
#:key target)
"Return a manifest entry for PACKAGE on SYSTEM, optionally cross-compiled to
TARGET."
(manifest-entry
(inherit (package->manifest-entry package))
(name (string-append (package-name package) "." system
(if target
(string-append "." target)
"")))
(item (with-parameters ((%current-system system)
(%current-target-system target))
package))))
(define native-builds
(manifest
(append (map (lambda (system)
(package->manifest-entry* guile system))
'("x86_64-linux" "i686-linux"
"aarch64-linux" "armhf-linux"
"powerpc64le-linux"))
(map (lambda (guile)
(package->manifest-entry* guile "x86_64-linux"))
(cons (package
(inherit (package-with-c-toolchain
guile
`(("clang-toolchain"
,(specification->package
"clang-toolchain")))))
(name "guile-clang"))
(list guile-without-threads
guile-without-networking
guile-debug
guile-strict-typing))))))
(define cross-builds
(manifest
(map (lambda (target)
(package->manifest-entry* guile "x86_64-linux"
#:target target))
'("i586-pc-gnu"
"aarch64-linux-gnu"
"riscv64-linux-gnu"
"i686-w64-mingw32"
"x86_64-linux-gnu"))))
(concatenate-manifests (list native-builds cross-builds))</code></pre><p>We won’t go into the details of this manifest; suffice to say that it
provides additional flexibility. We now need to tell Cuirass to build
this manifest, which is done with a spec slightly different from the
previous one:</p><pre><code class="language-scheme">;; Cuirass spec file to build all the packages of the ‘guile’ channel.
(list (specification
(name "guile")
(build '(manifest ".guix/manifest.scm"))
(channels
(append (list (channel
(name 'guile)
(url "https://git.savannah.gnu.org/git/guile.git")
(branch "main")))
%default-channels))))</code></pre><p>We changed the <code>(build …)</code> part of the spec to <code>'(manifest ".guix/manifest.scm")</code> so that it would pick our manifest, and that’s
it!</p><h1>Wrapping up</h1><p>We picked Guile as the running example in this post and you can see the
result here:</p><ul><li><a href="https://git.savannah.gnu.org/cgit/guile.git/tree/.guix-channel?id=cd57379b3df636198d8cd8e76c1bfbc523762e79"><code>.guix-channel</code></a>;</li><li><a href="https://git.savannah.gnu.org/cgit/guile.git/tree/.guix/modules/guile-package.scm?id=cd57379b3df636198d8cd8e76c1bfbc523762e79"><code>.guix/modules/guile-package.scm</code></a>
with the top-level <code>guix.scm</code> symlink;</li><li><a href="https://git.savannah.gnu.org/cgit/guile.git/tree/.guix/manifest.scm?id=cd57379b3df636198d8cd8e76c1bfbc523762e79"><code>.guix/manifest.scm</code></a>.</li></ul><p>These days, repositories are commonly peppered with dot files for
various tools: <code>.envrc</code>, <code>.gitlab-ci.yml</code>, <code>.github/workflows</code>,
<code>Dockerfile</code>, <code>.buildpacks</code>, <code>Aptfile</code>, <code>requirements.txt</code>, and whatnot.
It may sound like we’re proposing a bunch of <em>additional</em> files, but in
fact those files are expressive enough to <em>supersede</em> most or all of
those listed above.</p><p>With a couple of files, we get support for:</p><ul><li>development environments (<code>guix shell</code>);</li><li>pristine test builds, including for package variants and for
cross-compilation (<code>guix build</code>);</li><li>continuous integration (with Cuirass or with some other tool);</li><li>continuous delivery to users (<em>via</em> the channel and with pre-built
binaries);</li><li>generation of derivative build artifacts such as Docker images or
Deb/RPM packages (<code>guix pack</code>).</li></ul><p>At the Guix headquarters, we’re quite happy about the result. We’ve
been building a unified tool set for reproducible software deployment;
this is an illustration of how you as a developer can benefit
from it!</p><h1>Acknowledgments</h1><p>Thanks to Attila Lendvai, Brian Cully, and Ricardo Wurmus for providing
feedback on an earlier draft of this post.</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package manager and
an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects user
freedom</a>.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, AArch64 and POWER9 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2021/building-derivations-how-complicated-can-it-be//Building derivations, how complicated can it be?Christopher Baines2021-04-23T20:00:00Z2021-04-23T20:00:00Z Derivations are key to Guix, they're the low-level build instructions
used for things like packages, disk images, and most things than end
up in the store. Around a year ago, the established approach to build derivations
across multiple machines was daemon offloading . This
offloading approach is mostly static in terms of the machines involved
and uses SSH to communicate and move things between machines. The Guix Build Coordinator project set out to provide an alternative
approach, both to explore what's possible, but also to provide a
usable tool to address two specific use cases. The first use case…<p><a href="https://guix.gnu.org/manual/en/html_node/Derivations.html">Derivations</a> are key to Guix, they're the low-level build instructions
used for things like packages, disk images, and most things than end
up in the store.</p><p>Around a year ago, the established approach to build derivations
across multiple machines was <a href="https://guix.gnu.org/manual/en/html_node/Daemon-Offload-Setup.html">daemon offloading</a>. This
offloading approach is mostly static in terms of the machines involved
and uses SSH to communicate and move things between machines.</p><p>The Guix Build Coordinator project set out to provide an alternative
approach, both to explore what's possible, but also to provide a
usable tool to address two specific use cases.</p><p>The first use case was building things (mostly packages) for the
purpose of providing substitutes. At the time, the daemon offloading
approach used on ci.guix.gnu.org which is the default source of
substitutes. This approach was not scaling particularly well, so there
was room for improvement.</p><p>The second use case was more aspirational, support various quality
assurance tasks, like building packages changed by patches, regularly
testing fixed output derivations, or building the same derivations
across different machines to test for hardware specific differences.</p><p>While both these tasks have quite a lot in common, there's still quite
a lot of differences, this in part led to a lot of flexibility in the
design of the Guix Build Coordinator.</p><h1>Architecture</h1><p>Like offloading, the Guix Build Coordinator works in a centralised
manner. There's one coordinator process which manages state, and agent
processes run on the machines that perform builds. The coordinator
plans which builds to allocate to which agents, and agents ask the
coordinator for builds, which they then attempt.</p><p>Once agents complete a build, they send the log file and any outputs
back to the coordinator. This is shown in the diagram below. Note that
the Guix Build Coordinator doesn't actually take care of building the
individual derivations, that's still left to <code>guix-daemon</code>'s on the
machines involved.</p><p><img src="/static/blog/img/build-coordinator-architecture.svg" alt="Guix Build Coordinator sequence diagram" /></p><p>The builds to perform are worked through methodically, a build won't
start unless all the inputs are available. This behaviour replicates
what <code>guix-daemon</code> does, but across all the machines involved.</p><p>If agents can't setup to perform a build, they report this back to the
coordinator, which may then perform other builds to produce those
required inputs.</p><p>Currently HTTP is used when agents want to communicate to the
coordinator, although additional approaches could be implemented in
the future. Similarly, SQLite is used as the database, but from the
start there has been a plan to support PostgreSQL, but that's yet to
be implemented.</p><h1>Comparison to offloading</h1><p>There's lots more to the internal workings of the Guix Build
Coordinator, but how does this compare to the daemon offloading
builds?</p><p>Starting from the outside and working in, the API for the Guix Build
Coordinator is all asynchronous. When you submit a build, you get an
ID back, which you can use to track the progress of that build. This
is in contrast to the way the daemon is used, where you keep a
connection established while builds are progressing.</p><p>Offloading is tightly integrated with the daemon, which can be both
useful as offloading can transparently apply to anything that would
otherwise be built locally. However, this can also be a limitation
since the daemon is one of the harder components to modify.</p><p>With offloading, <code>guix-daemon</code> reaches out to another machine, copies
over all the inputs and the derivation, and then starts the
build. Rather than doing this, the Guix Build Coordinator agent pulls
in the inputs and derivation using substitutes.</p><p>This pull approach has a few advantages, firstly it removes the need
to keep a large store on the machine running the coordinator, and this
large store requirement of using offloading became a problem in terms
of scalability for the offloading approach. Another advantage is that
it makes deploying agents easier, as they don't need to be reachable
from the coordinator over the network, which can be an issue with NATs
or virtual machines.</p><p>When offloading builds, the outputs would be copied back to the store
on build success. Instead, the Guix Build Coordinator agent sends the
outputs back as nar files. The coordinator would then process these
nar files to make substitutes available. This helps distribute the
work in generating the nar files, which can be quite expensive.</p><p>These differences may be subtle, but the architecture makes a big
difference, it's much easier to store and serve nars at scale if this
doesn't require a large store managed by a single guix-daemon.</p><p>There's also quite a few things in common with the daemon offloading
approach. Builds are still methodically performed across multiple
machines, and load is taken in to account when starting new builds.</p><h1>A basic example</h1><p>Getting the Guix Build Coordinator up and running does require some
work, the following commands should get the coordinator and an agent
running on a single machine. First, you start the coordinator.</p><pre><code class="language-sh"> guix-build-coordinator</code></pre><p>Then in another terminal, you interact with the running coordinator to
create an agent in it's database.</p><pre><code class="language-sh"> guix-build-coordinator agent new</code></pre><p>Note the UUID of the generated agent.</p><pre><code class="language-sh"> guix-build-coordinator agent <AGENT ID> password new</code></pre><p>Note the generated password for the agent. With the UUID and password,
the agent can then be started.</p><pre><code class="language-sh"> guix-build-coordinator-agent --uuid=<AGENT ID> --password=<AGENT PASSWORD></code></pre><p>At this point, both processes should be running and the
guix-build-coordinator should be logging requests from the agent.</p><p>In a third terminal, also at the root of the repository, generate a
derivation, and then instruct the coordinator to have it built.</p><pre><code class="language-sh"> guix build --no-grafts -d hello</code></pre><p>Note the derivation that is output.</p><pre><code class="language-sh"> guix-build-coordinator build <DERIVATION FILE></code></pre><p>This will return a randomly generated UUID that represents the build.</p><p>While running from the command line is useful for development and
experimentation, there are <a href="https://guix.gnu.org/en/manual/en/html_node/Guix-Services.html#Guix-Build-Coordinator">services in Guix for running the
coordinator and agents</a>.</p><h1>Applications of the Guix Build Coordinator</h1><p>While I think the Guix Build Coordinator is a better choice than
daemon offloading in some circumstances, it doesn't currently replace
it.</p><p>At a high level, the Guix Build Coordinator is useful where there's a
need to build derivations and do something with the outputs or build
results, more than just having the outputs in the local store. This
could be serving substitutes, or testing derivations for example.</p><p>At small scales, the additional complexity of the coordinator is
probably unnecessary, but when it's useful to use multiple machines,
either because of the resources that provides, or because of a more
diverse range of hardware, then it makes much more sense to use the
Guix Build Coordinator to coordinate what's going on.</p><h1>Looking forward</h1><p>The Guix Build Coordinator isn't just an alternative to daemon
offloading, it's more a general toolkit for coordinating the building
of derivations.</p><p>Much of the functionality in the Guix Build Coordinator happens in
hooks. There are bits of code (hooks) that run when certain events
happen, like a build gets submitted, or a build finished
successfully. It's these hooks that are responsible for doing things
like processing nars to be served as substitutes, or submitting
retries for a failed build.</p><p>This hook to automatically retry building particular derivations is
particularly useful when trying to provide substitutes where you want
to lessen the impact of random failures, or for quality assurance
purposes, where you want more data to better identify problems.</p><p>There are also more features such as build and agent tags and build
priorities that can be really useful in some scenarios.</p><p>My hope is that the Guix Build Coordinator will enable a better
substitute experience for Guix users, as well as enabling a whole new
range of quality assurance tasks. It's already possible to see some
impact from the Guix Build Coordinator, but there's still much work to
do!</p><h2>Additional material</h2><ul><li><a href="https://git.cbaines.net/guix/build-coordinator/">Guix Build Coordinator Git repository</a></li><li><a href="https://lists.gnu.org/archive/html/guix-devel/2020-04/msg00323.html">2020/04/17 - Initial announcement - Prototype tool for building derivations</a></li><li><a href="https://issues.guix.gnu.org/43494">2020/09/19 - [PATCH 0/4] Add package and services for the Guix Build Coordinator</a></li><li><a href="https://lists.gnu.org/archive/html/guix-devel/2020-11/msg00417.html">2020/11/17 - Thoughts on building things for substitutes and the Guix Build Coordinator</a></li><li><a href="https://xana.lepiller.eu/guix-days-2020/guix-days-2020-christopher-baines-guix-build-coordinator.mp4">2020/11/22 - Guix Days 2020 - Progress so far on the Guix Build Coordinator</a></li><li><a href="https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00148.html">2021/02/09 - The Guix Build Coordinator in 2021</a></li><li><a href="https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00223.html">2021/02/14 - Getting the Guix Build Coordinator agent working on the Hurd</a></li><li><a href="https://lists.gnu.org/archive/html/guix-devel/2021-03/msg00074.html">2021/03/08 - Hurd substitute availability (27.5%) and next steps?</a></li></ul><h4>About GNU Guix</h4><p><a href="https://www.gnu.org/software/guix">GNU Guix</a> is a transactional package
manager and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects
user
freedom</a>.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2020/introduction-to-the-guix-data-service-the-missing-blog-post//Introduction to the Guix Data Service, the missing blog postChristopher Baines2020-11-08T20:30:00Z2020-11-08T20:30:00Z The Guix Data Service processes, stores and
provides data about Guix over time, at least that is what the
README says. It's been around since the start of
2019 , and while there have been plenty of long emails
to the guix-devel mailing list about it and a blog post about a
related Outreachy project , this is the first
blog post covering what it is and why it exists. Why? The initial motivation came from trying to automate aspects of
reviewing patches for Guix. If you have some patches for Guix, one
aspect of review…<p>The <a href="https://git.savannah.gnu.org/cgit/guix/data-service.git/">Guix Data Service</a> processes, stores and
provides data about Guix over time, at least that is what the
<a href="https://data.guix.gnu.org/README">README</a> says. It's been around since the <a href="https://lists.gnu.org/archive/html/guix-devel/2019-02/msg00089.html">start of
2019</a>, and while there have been plenty of long emails
to the guix-devel mailing list about it and a <a href="https://guix.gnu.org/en/blog/2020/improve-internationalization-support-for-the-guix-data-service/">blog post about a
related Outreachy project</a>, this is the first
blog post covering what it is and why it exists.</p><h1>Why?</h1><p>The initial motivation came from trying to automate aspects of
reviewing patches for Guix. If you have some patches for Guix, one
aspect of review might be to apply the patches and then build the
affected packages. How do you know what packages are affected though?</p><p>You could try and guess based on the content of the patches, and this
could work some of the time, but because Guix packages relate to one
another, changing one package may cause dependent packages to change.
Additionally, there are places in Guix where small changes could
affect a large number of packages, build systems for example. The
<code>guix refresh -l</code> command is really helpful when testing packages
locally, but it can in some cases miss some packages that are effected
by changes, as it only explores the package graph.</p><p>The approach taken to working out what packages are affected by a set
of patches, was to record information about all the packages in the
"base" revision of Guix, prior to applying the patches, and also
record information about the "target" revision generated from applying
the patches. With all that information about the two revisions, you
can then compare the data to determine what's changed. This goes
beyond finding out what packages are affected, and includes things
like looking at changes to lint warnings, channel news entries, and
more.</p><p><a href="https://data.guix.gnu.org/compare?base_commit=f503cfc9c51ea4ddd6cc9c027f1897e7866e411e&target_commit=f161bd2cd7af6a0a7027a2e4ed97912027d5033d"><img src="https://guix.gnu.org/static/blog/img/data-guix-gnu-org-compare.png" alt="Screenshot of the comparision between two commits" /></a></p><p>This approach of storing information about revisions has applications
beyond reviewing patches, which is another reason why this approach
was taken. While the Guix Data Service doesn't bring new knowledge to
the world, it can make information that is out there more accessible,
and that improved accessibility is a feature.</p><h1>Applications</h1><p>Say you want to know when the previous version of a package was
available, and what that version was. You could look through the Git
repository history, or inspect previous revisions to find out, but
because the Guix Data Service can store the available package names
and versions in a range of revisions, it can provide this information
more quickly and with less effort.</p><p><a href="https://data.guix.gnu.org/repository/1/branch/master/package/emacs"><img src="https://guix.gnu.org/static/blog/img/data-guix-gnu-org-emacs-versions.png" alt="Screenshot of a Guix Data Service package versions page for emacs" /></a></p><p>Now, questions about package versions is something a user of Guix
might have. However, so far I haven't seen the Guix Data Service as
something that users of Guix should necessarily use or be aware of.
Instead, I think it has a place to provide information to enable
things that users of Guix would directly use.</p><p>There are a few applications of data from the Guix Data Service in
varying states of development. I've been attempting to automate parts
of a <a href="https://git.cbaines.net/guix/weekly-news/">weekly news publication about Guix</a> through using
the Guix Data Service, I've also been writing a <a href="https://git.cbaines.net/guix/build-coordinator/">service for building
derivations</a>, which I've been using in
conjunction with the Guix Data Service to provide substitutes. As
part of an Outreachy internship on improving internationalisation
support in the Guix Data Service, Danjela worked on creating a package
search page for the Guix website, which wrapped the package search
functionality in the Guix Data Service.</p><p><a href="https://prototype-guix-weekly-news.cbaines.net/"><img src="https://guix.gnu.org/static/blog/img/prototype-guix-weekly-news.png" alt="Screenshot of the prototype weekly news site" /></a></p><p>While I'm cautious about having the Guix Data Service attempt to
address individual user needs, there are some applications where it
alone is sufficient. I've been using the Guix Data Service to gather
up data about which packages in Guix don't build reproducibly.
Hopefully the Guix Data Service is well positioned to help with
technical questions like this.</p><p><a href="https://data.guix-patches.cbaines.net/repository/2/branch/master/latest-processed-revision/package-reproducibility"><img src="https://guix.gnu.org/static/blog/img/data-guix-patches-package-reproducibility.png" alt="Screenshot of the Guix Data Service package reproducibility page" /></a></p><h1>Architecture</h1><p>The Guix Data Service is written in Guile, and uses PostgreSQL for the
database. There's plenty of SQL queries in the code, including some
quite long ones.</p><p>There are several scripts which act as entry points to different parts
of the Guile codebase:</p><ul><li><p><code>guix-data-service</code></p><ul><li>Provides the web interface</li></ul></li><li><p><code>guix-data-service-process-jobs</code></p><ul><li>Polls the database for new jobs, and forks <code>guix-data-service-process-job</code></li></ul></li><li><p><code>guix-data-service-process-job</code></p><ul><li>Processes an individual job, loading data for a single revision</li></ul></li><li><p><code>guix-data-service-process-branch-updated-email</code></p><ul><li><p>Processes an email to find out about new revisions</p></li></ul></li></ul><p>There's also other scripts which perform a range of functions, like
backing up the database, generating a minimal database which is
hopefully small in size and querying build/substitute servers for
information.</p><p>When running on a Guix system, there's a <a href="https://guix.gnu.org/manual/devel/en/html_node/Guix-Services.html#Guix-Data-Service">service to help with
deployment</a>.</p><h1>Getting information in</h1><p>Rather than polling the Git repository to find out about new
revisions, the methodology used so far has been to receive emails. In
the case of the main Guix Git repository, this can work as follows:</p><ul><li>New commits are pushed to the Guix Git repository on Savannah</li><li>A post-receive hook sends an email about the branch that's been
updated, plus emails about each commit</li><li>A dedicated email account is used to subscribe to guix-commits, and
this receives the emails</li><li><code>getmail</code> is running as a service on the machine running the Guix
Data Service, it receives the emails and calls
<code>guix-data-service-process-branch-updated-email</code> passing the
contents in on stdin</li><li>The script reads the email and inserts the relevant data in about
the branch that was updated, the time it was updated, and also
inserts a new job, representing the new revision to be processed</li></ul><p>Compared to polling the Git repository, this approach has a few
advantages:</p><ul><li>The time the email was sent is a good proxy to when the branch was
updated</li><li>Receiving emails promptly, having <code>getmail</code> use IDLE for example
helps with learning of changes quickly</li><li>The email account provides some reliability, so messages aren't
missed if the Guix Data Serivce is down, they'll just be processed
later</li><li>The mbox files for the guix-commits mailing list can be processed
to provide data for past revision, in years when the Guix Data
Service wasn't running for example</li></ul><p>When the <code>guix-data-service-process-job</code> script runs, it goes through
a long process to extract information about that revision of Guix, and
store it in the database.</p><p>The first part of this is to actually fetch and build the relevant
revision. The Guix Data Service uses <a href="https://guix.gnu.org/manual/devel/en/html_node/Channels.html">channels</a>
and <a href="https://guix.gnu.org/manual/devel/en/html_node/Inferiors.html">inferiors</a>, the same code used by <code>guix pull</code> and <code>guix time-machine</code> for communication with another revision
of Guix. It's through the inferior REPL that information from the
target revision is extracted.</p><p>In addition to receiving information about new revisions, the Guix
Data Service can accept POST requests to receive information about
builds. There's some support in Cuirass and the Guix Build
Coordinator to send these requests.</p><h1>Storing all that information</h1><p>Following on from one of the initial motivations for the Guix Data
Service, comparing two revisions to determine which packages have
changed, the schema for the database is organised to facilitate fast
comparisons between two arbitrary revisions. The compromise here is
the storage space taken up.</p><p>Similar to version control systems, an alternative schema would have
been to store the differences between revisions in a linked list or
tree. This would avoid storing lots of information that generally
doesn't often change between subsequent revisions, but at the expense
of making both determining the entire state of individual revisions
and comparing arbitrary revisions more complex and costly.</p><p>Even though all the information about each revision is associated with
that revision, there is some indirection, and deduplication involved.
For example, each revision is associated against entries in the
<code>package_derivations</code> table, which represents a package plus
derivation for a specific system and target. If this information
doesn't differ between two revisions, they'll just reference the same
entries in this table.</p><h1>Making information available</h1><p>Currently, the Guix Data Service provides a web interface. The HTML
pages and forms are designed to help potential users of the Guix Data
Service find and explore the available data. When using the Guix Data
Service, you'd probably want a more machine readable form for the
data, rather than HTML, and at the moment that's JSON. You should be
able to request JSON either through the HTTP Accept header, or by
using the <code>.json</code> extension on the URL path.</p><h1>Deployments</h1><p>There's not just one deployment of the Guix Data Service, currently I
know of two. There's <a href="https://data.guix.gnu.org/">data.guix.gnu.org</a>
which just tracks the master branch of Guix, and has data going back
to roughly the start of 2019 (with some gaps). There's also
<a href="https://data.guix-patches.cbaines.net/">data.guix-patches.cbaines.net</a>
which isn't limited to just the master branch, and has additional
branches constructed from patches that are submitted, but doesn't have
much historical data.</p><h1>Looking forward</h1><p>There's still lots of areas where the Guix Data Service can be
improved.</p><p>It would be convenient if getting data in to the Guix Data Service was
faster, currently there's quite a delay between the Guix Data Service
finding out about a new revision, and it completing processing it.</p><p>The processing could also be improved, there's some notable current
omissions like the package graph (inputs, propagated-inputs and
native-inputs) as well as package replacements (grafts). It would also
be interesting to see if the Guix Data Service could be generalised to
process other channels, instead or in addition to the main Guix
channel.</p><p>In the future, I'd like to make the data available in formats other
than JSON, like RDF. I'd also like for it to be possible to
watch/subscribe to particular things that the Guix Data Service knows
about, the Guix Data Service would then notify you via some means that
there's be a change. This could enable all sorts of applications to
respond to changes connected to Guix.</p><h2>Additional reading</h2><p>To provide some information, and to help get my own thoughts in order,
I sent out semi-regular emails about the Guix Data Service over the
last two years, I've linked to most of these below:</p><ul><li><a href="https://lists.gnu.org/archive/html/guix-devel/2020-06/msg00034.html">2020/06/03 - Build reproducibility metrics</a></li><li><a href="https://lists.gnu.org/archive/html/guix-devel/2020-05/msg00153.html">2020/05/07 - April update on data.guix.gnu.org (Guix Data Service)</a></li><li><a href="https://lists.gnu.org/archive/html/guix-devel/2020-03/msg00476.html">2020/03/30 - Patchwork + the Guix Data Service for assisting with patch review</a></li><li><a href="https://lists.gnu.org/archive/html/guix-devel/2020-03/msg00454.html">2020/03/29 - March update on data.guix.gnu.org (Guix Data Service)</a></li><li><a href="https://lists.gnu.org/archive/html/guix-devel/2020-02/msg00268.html">2020/02/17 - February update on data.guix.gnu.org and the Guix Data Service</a></li><li><a href="https://lists.gnu.org/archive/html/guix-devel/2020-01/msg00073.html">2020/01/05 - Another update on the Guix Data Service</a></li><li><a href="https://lists.gnu.org/archive/html/guix-devel/2019-09/msg00277.html">2019/09/30 - Anyone interested in getting involved with the Guix Data Service?</a></li><li><a href="https://lists.gnu.org/archive/html/guix-devel/2019-09/msg00104.html">2019/09/08 - Guix Data Service - September update</a></li><li><a href="https://lists.gnu.org/archive/html/guix-devel/2019-05/msg00332.html">2019/05/17 - More progress with the Guix Data Service</a></li><li><a href="https://lists.gnu.org/archive/html/guix-devel/2019-05/msg00127.html">2019/05/06 - Linting, and how to get the information in to the Guix Data Serivce</a></li><li><a href="https://lists.gnu.org/archive/html/guix-devel/2019-04/msg00094.html">2019/04/04 - Progress with the Guix Data Service</a></li><li><a href="https://lists.gnu.org/archive/html/guix-devel/2019-02/msg00089.html">2019/02/08 - Tracking and inspecting how Guix changes over time</a></li></ul><h4>About GNU Guix</h4><p><a href="https://www.gnu.org/software/guix">GNU Guix</a> is a transactional package
manager and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects
user
freedom</a>.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2018/gsoc-2018-report-cuirass-web-interface//GSoC 2018 report: Cuirass Web interfaceTatiana Sholokhova2018-08-13T17:00:00Z2018-08-13T17:00:00Z For the last three months I have been working with the Guix team as a
Google Summer of Code intern. The title of my project is "GNU Guix
(Cuirass): Adding a web interface similar to the Hydra web interface". Cuirass is a continuous integration system which monitors the Guix git
repository, schedules builds of Guix packages, and presents the build
status of all Guix packages. Before my project, Cuirass did not have
a web interface. The goal of the project was to implement an interface
for Cuirass which would allow a user to view the overall build
progress, details about evaluations, build failures,…<p>For the last three months I have been working with the Guix team as a
Google Summer of Code intern. The title of my project is "GNU Guix
(Cuirass): Adding a web interface similar to the Hydra web interface".</p><p>Cuirass is a continuous integration system which monitors the Guix git
repository, schedules builds of Guix packages, and presents the build
status of all Guix packages. Before my project, Cuirass did not have
a web interface. The goal of the project was to implement an interface
for Cuirass which would allow a user to view the overall build
progress, details about evaluations, build failures, etc. The web
interface of <a href="https://hydra.nixos.org/">Hydra</a> is a good example of
such a tool.</p><p>In this post, I present a final report on the project. The Cuirass
repository with the changes made during the project is located at
<a href="http://git.savannah.gnu.org/cgit/guix/guix-cuirass.git">http://git.savannah.gnu.org/cgit/guix/guix-cuirass.git</a>. A
working instance of the implemented interface is available at
<a href="https://berlin.guixsd.org/">https://berlin.guixsd.org/</a>. You can
find more examples and demonstrations of the achieved results below.</p><h1>About Cuirass</h1><p>Cuirass is designed to monitor a git repository containing Guix
package definitions and build binaries from these package definitions.
The state of planned builds is stored in a SQLite database. The key
concepts of the Cuirass internal state are:</p><ul><li><p><strong>Job specification</strong>. Specifications state what has actually to be
done by Cuirass. A specification <a href="https://www.gnu.org/software/guix/manual/en/html_node/Continuous-Integration.html">is
defined</a>
by a Scheme data structure (an association list) which includes a
job name, repository URL, as well as the branch and a procedure
<code>proc</code> that specifies how this is to be built.</p></li><li><p><strong>Evaluation</strong>. An evaluation is a high-level build action
related to a certain revision of a repository of a given
specification. For each specification, Cuirass continuously
produces new evaluations which build different versions of the
project represented by revisions of the corresponding repository.
Derivations and builds (see below) each belong to a specific
evaluation.</p></li><li><p><strong>Derivation</strong>. Derivations represent low-level build actions. They
store such information as name of a build script and its arguments,
input and output of a build action, target system type, and
necessary environment variables.</p></li><li><p><strong>Build</strong>. A build is a result of build actions that are prescribed
by a derivation. This could be a failed build or a directory
containing the files that were generated by compiling a package.</p></li></ul><p>Besides the core which executes build actions and records their
results in the database, Cuirass includes a web server which
previously only responded to a handful of API requests with JSON
containing information about the current status of builds.</p><h1>Web interface</h1><p>The Cuirass web interface implemented during the project is served by
the Cuirass web server whose functionality has been extended to
generating HTML responses and serving static files. General features
of the interface are listed below.</p><ul><li><p>The backend is written in Guile and implements request processing
procedures which parse request parameters and extract specific data
to be displayed from the database.</p></li><li><p>The frontend consists of HTML templates represented with <a href="https://www.gnu.org/software/guile/manual/en/html_node/SXML.html">Guile
SXML</a>
and the <a href="https://getbootstrap.com/">Bootstrap 4</a> CSS library.</p></li><li><p>The appearance is minimalistic. Every page includes only specific
content information and basic navigation tools.</p></li><li><p>The interface is lightweight and widely accessible. It does not use
JavaScript which makes it available to users who do not want to have
JavaScript running in the browser.</p></li></ul><h2>Structure</h2><p>Let's review the structure of the interface and take a look at the
information you can find in it. Note that the web-interface
screenshots presented below were obtained with synthetic data loaded
into Cuirass database.</p><h3>Main page</h3><p>The main page is accessible on the root request endpoint (<code>/</code>). The
main page displays a list of all the specifications stored in the
Cuirass database. Each entry of the list is a clickable link which
leads to a page about the evaluations of the corresponding
specification (see below).</p><p>Here is an example view of the main page.</p><p><img src="https://www.gnu.org/software/guix/static/blog/img/cuirass-wi-main.png" alt="Main page screenshot" /></p><h3>Evaluations list</h3><p>The evaluations list of a given specification with name <code><name></code> is
located at <code>/jobset/<name>/</code>. On this page, you can see a list of
evaluations of the given project starting from the most recent ones.
You can navigate to older evaluations using the pagination buttons at
the bottom of the page. In the table, you can find the following
information:</p><ul><li><p>The ID of the evaluation which is clickable and leads to a page with
information about all builds of the evaluation (see below).</p></li><li><p>List of commits corresponding to the evaluation.</p></li><li><p>Build summary of the evaluation: number of succeeded (green), failed
(red), and scheduled (grey) builds of this evaluation. You can open
the list of builds with a certain status by clicking on one of these
three links.</p></li></ul><p>Here is a possible view of the evaluations list page:</p><p><img src="https://www.gnu.org/software/guix/static/blog/img/cuirass-wi-evals.png" alt="Screenshoot of evaluations list" /></p><h3>Builds list</h3><p>The builds list of a evaluation with ID <code><id></code> is located at
<code>/eval/<id>/</code>. On this page, you can see a list of builds of the
given evaluation ordered by their stop time starting from the most
recent one. Similarly to the evaluation list, there are pagination
buttons located at the bottom of the page. For each build in the
list, there is information about the build status (succeeded, failed,
or scheduled), stop time, nixname (name of the derivation), system, and
also a link to the corresponding build log. As said above, it is
possible to filter builds with a certain status by clicking on the
status link in the evaluations list.</p><p><img src="https://www.gnu.org/software/guix/static/blog/img/cuirass-wi-builds.png" alt="Screenshot of builds list" /></p><h1>Summary</h1><p>Cuirass now has the web interface which makes it possible for users to
get an overview on the status of Guix package builds in a
user-friendly way. As the result of my GSoC internship, the core of
the web interface was developed. Now there are several possibilities
for future improvements and I would like to welcome everyone to
contribute.</p><p>It was a pleasure for me to work with the Guix team. I would like to
thank you all for this great experience! Special thanks to my GSoC
mentors: Ricardo Wurmus, Ludovic Courtès, and Gábor Boskovits, and
also to Clément Lassieur and Danny Milosavljevic for their guidance
and help throughout the project.</p>https://guix.gnu.org/blog/2016/guixsd-system-tests//GuixSD system testsLudovic Courtès2016-06-28T00:00:00+02002016-06-28T00:00:00+0200 From its inception, Guix has had a thorough test suite—something that’s not only reassuring, but also the thing that allows for fearless evolution of the code. That we didn’t have this safety net when hacking on the whole operating system, GuixSD, made it uncomfortable and more risky. We are now addressing the problem with the introduction of system tests , closing one of the major roadblocks towards 1.0. Before going into details, let me recap the sorts of testing that already occurred in Guix land. Unit tests Guix’s…<div><p>From its inception, Guix has had a thorough test suite—something that’s not only reassuring, but also the thing that allows for fearless evolution of the code. That we didn’t have this safety net when hacking on the whole operating system, GuixSD, made it uncomfortable and more risky. We are now addressing the problem with the introduction of <em>system tests</em>, closing one of the major roadblocks towards 1.0.<br /></p><p>Before going into details, let me recap the sorts of testing that already occurred in Guix land.<br /></p><h4>Unit tests</h4><p>Guix’s <a href="http://git.savannah.gnu.org/cgit/guix.git/tree/tests">test suite</a> currently contains almost 600 <em>unit tests</em>. Each one of these stresses one particular function or subset of the functionality of Guix. This covers core package management functionality such as package builds, utility modules such as monads or the public key infrastructure (PKI) used for authenticating binaries, maintenance tools such as <a href="https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-lint.html">lint</a> and the <a href="https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-import.html">importers</a>, as well as the command-line interface.<br /></p><p>Since Guix provides Scheme modules for use <a href="https://www.gnu.org/software/guix/manual/en/html_node/G_002dExpressions.html">both in the package management front-end and on the “build side”</a>, the latter is also tested. This includes part of the <a href="https://www.gnu.org/software/guix/manual/en/html_node/Build-Systems.html">build systems</a>, and helpers like our <a href="http://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/gremlin.scm#n264">ELF validation module</a>.<br /></p><h4>Package tests</h4><p>Then come the software packages that Guix ships. All of the packages in the distro are under <a href="https://hydra.gnu.org/jobset/gnu/master">continuous integration</a> on the 4 supported architectures (32-bit and 64-bit Intel compatible, as well as MIPS64 and ARMv7.) Our build farm serves the resulting binaries, which users can choose to download as <a href="https://www.gnu.org/software/guix/manual/en/html_node/Substitutes.html">substitutes for local builds</a>. Our build server, which currently runs an instance of <a href="https://nixos.org/hydra/">Hydra</a>, always shows the number of succeeding/failing builds on its dashboard. That way, breakage introduced by changes in the package collection is always rapidly detected. This is a direct benefit of the <a href="https://www.gnu.org/software/guix/manual/en/html_node/Introduction.html">functional packaging model</a>.<br /></p><p>Additionally, our policy is to always run each package’s test suite (typically “make check”) as part of its build process, unless there is a serious technical obstacle to doing that. That way, we can, and do catch integration issues, incompatibilities, and plain bugs before they hit users.<br /></p><h4>System tests</h4><p>So far, so good. Now, what about GuixSD itself? GuixSD did not have an automated test suite until now. What it did have, though, is the ability to instantiate an operating system in a virtual machine (VM) or in a container. You would write your <a href="https://www.gnu.org/software/guix/manual/en/html_node/Using-the-Configuration-System.html">operating system declaration</a> in a file, then run, say:<br /></p><div class="example"><pre>guix system vm my-config.scm </pre></div><p>This <a href="https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-system.html#Invoking-guix-system">gives you a script to launch a VM</a> running an instance of the OS declared in ‘my-config.scm’. Already pretty convenient! And indeed, even more so back in the days when we were eating a fair amount of dog food. In fact, that’s how we ate our <a href="/news/boot-to-guile.html">first</a><a href="/news/gnu-guix-04-released-happy-birthday-gnu.html">dog food dishes</a>, and the VM infrastructure was the fork and knife that made it more tolerable.<br /></p><p>So what could we test exactly? Roughly, we want to test that the instantiated system behaves according to the source ‘operating-system’ declaration: that user accounts are all there, that system services are running as expected, that all of the configuration is taken into account.<br /></p><p>To do that, we need to run the system under test in a VM, but we also need to instrument it. We use <a href="http://qemu.org">QEMU</a> to run our VMs, and QEMU along with the Linux virtio-serial module nicely supports communication between the guest operating system and the host, a strategy also used by <a href="https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/test-driver/Machine.pm">NixOS’ test driver</a>. Concretely, we define a <a href="http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/build/marionette.scm">“marionette”</a><a href="http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/tests.scm#n129">service</a>, which hooks a Guile <a href="https://www.gnu.org/software/guile/manual/en/html_node/The-REPL.html">read-eval-print loop</a> (REPL) inside the guest. This allows the host to evaluate arbitrary code in the guest VM.<br /></p><p>Now we can write build processes (<a href="https://www.gnu.org/software/guix/manual/en/html_node/Derivations.html">aka. “derivations”</a>) that will:<br /></p><ol><li>instantiate an instrumented variant of the operating system configuration we want to test in a VM image;
</li><li>spawn the VM, run a series of tests on the guest OS, and return the test results.
</li></ol><p>Thus, a system test to make sure the <a href="https://www.gnu.org/software/guile/manual/en/html_node/System-Identification.html#index-scm_005funame">‘uname’</a> system call returns something that matches the OS declaration looks like this:<br /></p><pre><code class="language-scheme">(define (run-test)
(define os
;; The declaration of the OS we want to instantiate and test.
;; Calling 'marionette-operating-system' instruments it.
(marionette-operating-system
(operating-system
(host-name "komputilo")
(timezone "Europe/Berlin")
(locale "en_US.UTF-8")
(bootloader (grub-configuration (device "/dev/sdX")))
(file-systems %base-file-systems))))
;; Compute the script to run OS in a VM.
(mlet %store-monad ((run (system-qemu-image/shared-store-script
os #:graphic? #f)))
(define test
;; The actual test. Here “#~” is like “quote”, allowing us
;; to describe code to run in the build environment; it’s a
;; “g-expression.”
#~(begin
(use-modules (gnu build marionette)
(srfi srfi-64)
(ice-9 match))
(define marionette
;; Spawn the VM that runs the declared OS.
(make-marionette (list #$run)))
(mkdir #$output)
(chdir #$output)
(test-begin "basic")
(test-assert "uname"
;; Call the ‘uname’ Scheme function in the guest.
;; In the host, make sure its result (a vector) matches
;; our OS declaration above.
(match (marionette-eval '(uname) marionette)
(#("Linux" host-name version _ architecture)
(and (string=? host-name
#$(operating-system-host-name os))
(string-prefix? #$(package-version
(operating-system-kernel os))
version)
(string-prefix? architecture %host-type)))))
(test-end)
(exit (= (test-runner-fail-count (test-runner-current)) 0))))
;; Turn the test into a buildable “derivation”.
(gexp->derivation "simple-test" test
#:modules '((gnu build marionette)))))
</code></pre><p>There are interesting things going on here. First, while this is all Scheme code, there are in fact three tiers or strata of code at play here: the code that produces the OS declaration and the derivation, the build code of that derivation—the test—embodied in a <a href="https://www.gnu.org/software/guix/manual/en/html_node/G_002dExpressions.html">g-expression</a>, and code sent from the host to the guest VM via ‘marionette-eval’.<br /></p><p>Using Scheme all the way means we can share code, use the right tools such as the <a href="http://srfi.schemers.org/srfi-64/srfi-64.html">SRFI-64 test framework</a> here, pass values from one tier to another, and so on. And of course, being a full-blown language rather than shell scripts or similar means we have a rich and semantically-clear interface at our fingertips: we can directly access the data structures that matter rather than grepping the output of high-level commands. As an example, we can directly query the <a href="https://www.gnu.org/software/guix/manual/en/html_node/Shepherd-Services.html">system service manager</a> right <a href="http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/services/herd.scm">from Scheme</a>, which is often useful in system tests.<br /></p><h4>Status</h4><p>Guix now includes the test infrastructure described above; running “make check-system” runs all the currently defined tests. Currently we have <a href="http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/tests/base.scm">tests for basic functionality</a>. This includes making sure that all the system services declared are available and running. We have tests for specific system services such as the <a href="https://www.gnu.org/software/mcron/">mcron</a> job scheduling daemon—inspired by your parents’ cron, but with a powerful Scheme interface—and <a href="http://avahi.org/">Avahi</a> and its <a href="https://www.gnu.org/software/guix/manual/en/html_node/Name-Service-Switch.html">name service switch</a> (NSS) integration.<br /></p><p>Last but not least, we have <a href="http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/tests/install.scm">tests</a> of the full <a href="https://www.gnu.org/software/guix/manual/en/html_node/System-Installation.html">GuixSD installation procedure</a>, which turned out to be <a href="https://lists.gnu.org/archive/html/guix-devel/2016-06/msg00815.html">more involved</a> than the other tests. This works by running the GuixSD installation image in a VM, using another VM image as the target installation media, and finally booting the newly-installed system.<br /></p><p>All the tests are automatically run on our build farm (see <a href="https://hydra.gnu.org/job/gnu/master/test.basic.x86_64-linux">here</a>, <a href="https://hydra.gnu.org/job/gnu/master/test.installed-os.x86_64-linux">here</a>, or <a href="https://hydra.gnu.org/job/gnu/master/test.mcron.i686-linux">there</a>), which provides quick feedback. One step closer to 1.0!<br /></p><h4>About GNU Guix</h4><p><a href="http://www.gnu.org/software/guix">GNU Guix</a> is a transactional package manager for the GNU system. The Guix System Distribution or GuixSD is an advanced distribution of the GNU system that relies on GNU Guix and <a href="http://www.gnu.org/distros/free-system-distribution-guidelines.html">respects the user's freedom</a>.<br /></p><p>In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. Guix uses low-level mechanisms from the Nix package manager, except that packages are defined as native <a href="http://www.gnu.org/software/guile">Guile</a> modules, using extensions to the <a href="http://schemers.org">Scheme</a> language. GuixSD offers a declarative approach to operating system configuration management, and is highly customizable and hackable.<br /></p><p>GuixSD can be used on an i686 or x86_64 machine. It is also possible to use Guix on top of an already installed GNU/Linux system, including on mips64el and armv7.<br /></p></div>