https://guix.gnu.org/feeds/blog/scheme-api.atomGNU Guix — Blog — Scheme APIfeed author nameGNU Guixhttps://guix.gnu.org/themes/initial/img/icon.png2024-03-20T10:57:58Zhttps://guix.gnu.org/blog/2023/write-package-definitions-in-a-breeze//Write package definitions in a breezeLudovic Courtès, Philippe Virouleau2023-11-24T14:30:00Z2023-11-24T14:30:00Z More than 28,000 packages are available in Guix today, not counting
third-party channels. That’s a lot—the 5th largest GNU/Linux
distro ! But it’s nothing if the one package you
care about is missing. So even you, dear reader, may one day find
yourself defining a package for your beloved deployment tool. This post
introduces a new tool poised to significantly lower the barrier to
writing new packages. Introducing Guix Packager Defining
packages
for Guix is not all that hard but, as always, it’s much harder the first
time you do it, especially when starting from a blank page…<p>More than 28,000 packages are available in Guix today, not counting
third-party channels. That’s a lot—the <a href="https://repology.org/">5th largest GNU/Linux
distro</a>! But it’s nothing if the one package you
care about is missing. So even you, dear reader, may one day find
yourself defining a package for your beloved deployment tool. This post
introduces a new tool poised to significantly lower the barrier to
writing new packages.</p><h1>Introducing Guix Packager</h1><p><a href="https://guix.gnu.org/manual/devel/en/html_node/Defining-Packages.html">Defining
packages</a>
for Guix is not all that hard but, as always, it’s much harder the first
time you do it, especially when starting from a blank page and/or not
being familiar with the programming environment of Guix. <a href="https://guix-hpc.gitlabpages.inria.fr/guix-packager/">Guix
Packager</a> is a new
web user interface to get you started—<a href="https://guix-hpc.gitlabpages.inria.fr/guix-packager/">try
it!</a>. It arrived
right in time as an aid to the <a href="https://hpc.guix.info/events/2023/workshop/program/#how-to-get-started-writing-guix-packages">packaging
tutorial</a>
given last week at the Workshop on Reproducible Software Environments.</p><p><img src="/static/blog/img/guix-packager.gif" alt="Screenshot showing the Guix Packager interface." /></p><p>The interface aims to be intuitive: fill in forms on the left
and it produces a correct, ready-to-use package definition on the right.
Importantly, it helps you avoid pitfalls that trip up many newcomers:</p><ul><li>When you add a dependency in one of the “Inputs” fields, it adds the
right variable name in the generated code <em>and</em> imports the right
<a href="https://guix.gnu.org/manual/devel/en/html_node/Package-Modules.html">package
module</a>.</li><li>Likewise, you can choose a license and be sure the <code>license</code> field
will refer to the right variable representing that license.</li><li>You can turn tests on and off, and add configure flags. These
translate to a valid <code>arguments</code> field of your package, letting you
discover the likes of <a href="https://guix.gnu.org/cookbook/en/html_node/A-Scheme-Crash-Course.html">keyword
arguments</a>
and
<a href="https://guix.gnu.org/manual/en/html_node/G_002dExpressions.html">G-expressions</a>
without having to first dive into the manual.</li></ul><p>Pretty cool, no?</p><h1>Implementation</h1><p>All the credit for this tool goes to co-worker and intrepid hacker
Philippe Virouleau. A unique combination of paren aversion and web
development superpowers—unique in the Guix community—led
Philippe to develop the whole thing in a glimpse (says Ludovic!).</p><p>The purpose was to provide a single view to be able to edit a package recipe,
therefore the application is a <em>single-page application</em> (SPA) written in
using the UI library Philippe is most comfortable with: <a href="https://react.dev/">React</a>,
and <a href="https://mui.com/material-ui/">MaterialUI</a> for styling the components.
It's built with <a href="https://www.typescriptlang.org/">TypeScript</a>, and the library
part actually defines all the types needed to manipulate Guix packages and their
components (such as build systems or package sources).
One of the more challenging parts was to be able to provide fast and helpful “search as you
type” results over the 28k+ packages. It required a combination of
MaterialUI's <a href="https://mui.com/material-ui/react-autocomplete/#virtualization">virtualized inputs</a>,
as well as caching the packages data in the browser's local storage,
when possible (packaging metadata itself is fetched from
<code>https://guix.gnu.org/packages.json</code>, a generic representation of the
current package set).</p><p>While the feature set provides a great starting point, there are still a few
things that may be worth implementing. For instance, only the GNU and
CMake build systems are supported so far; it would make sense to include
a few others (Python-related ones might be good candidates).</p><p>Running a local (development) version of the application can happen on
top of Guix, since—obviously—it's been developed with the <code>node</code>
version packaged in Guix, using the quite standard <code>packages.json</code> for
JavaScript dependencies installed through <code>npm</code>. <a href="https://gitlab.inria.fr/guix-hpc/guix-packager/">Contributions
welcome!</a></p><h1>Lowering the barrier to entry</h1><p>This neat tool complements a set of steps we’ve taken over time to make
packaging in Guix approachable. Indeed, while package definitions are
actually code written in the Scheme language, the <code>package</code> “language”
was designed <a href="https://arxiv.org/abs/1305.4584">from the get-go</a> to be
fully declarative—think JSON with parens instead of curly braces and
semicolons. More recently we <a href="https://guix.gnu.org/en/blog/2021/the-big-change/">simplified the way package inputs are
specified</a> with an
eye on making package definitions less intimidating.</p><p>The <a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-import.html"><code>guix import</code>
command</a>
also exists to make it easier to simplify packaging: it can generate a
package definition for anything available in other package
repositories such as PyPI, CRAN, Crates.io, and so forth. If your
preference goes to curly braces rather than parens, it can also <a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-import.html#index-JSON_002c-import">convert
a JSON package
description</a>
to Scheme code. Once you have your first <code>.scm</code> file, <code>guix build</code>
prints hints for common errors such missing module imports (those
<code>#:use-module</code> stanzas). We also put effort into providing <a href="https://guix.gnu.org/manual/devel/en/html_node/Defining-Packages.html">reference
documentation</a>,
a <a href="https://guix.gnu.org/en/videos/2020/packaging-part-two/">video
tutorial</a>, and
a <a href="https://guix.gnu.org/cookbook/en/html_node/Packaging-Tutorial.html">tutorial for more complex
packages</a>.</p><p>Do share your experience <a href="https://guix.gnu.org/contact">with us</a> and
until then, happy packaging!</p><h1>Acknowledgments</h1><p>Thanks to Felix Lechner and Timothy Sample for providing feedback on an
earlier draft of this post.</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package manager and
an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects user
freedom</a>.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, AArch64 and POWER9 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2023/parameterized-packages-for-gnu-guix//Parameterized Packages for GNU GuixSarthak Shah2023-06-09T16:00:00Z2023-06-09T16:00:00Z Hello Guix! I'm Sarthak and I'll be working on implementing Parameterized Packages
for GNU Guix as a Google Summer of Code
intern under the guidance of Pjotr Prins and Gábor Boskovits. What are Parameterized Packages? One of the many advantages of free software is the
availability of compile-time options for almost all packages.
Thanks to its dedication to building all packages from source,
Guix is one of the few GNU/Linux distributions that can take advantage
of these compile-time features; in fact, many advanced users such as
those using Guix on High-Performance Computing Systems
and new ISAs like…<p>Hello Guix!</p><p>I'm Sarthak and I'll be working on implementing Parameterized Packages
for GNU Guix as a <a href="https://summerofcode.withgoogle.com/">Google Summer of Code</a>
intern under the guidance of Pjotr Prins and Gábor Boskovits.</p><h1>What are Parameterized Packages?</h1><p>One of the many advantages of <a href="https://www.gnu.org/philosophy/free-sw.html">free software</a> is the
availability of compile-time options for almost all packages.
Thanks to its dedication to building all packages from source,
Guix is one of the few GNU/Linux distributions that can take advantage
of these compile-time features; in fact, many advanced users such as
those using Guix on <a href="https://hpc.guix.info">High-Performance Computing Systems</a>
and <a href="https://10years.guix.gnu.org/video/gnu-guix-and-the-risc-v-future/">new ISAs like RISC-V</a> have already been doing this
by utilizing a feature known as <a href="https://guix.gnu.org/manual/en/html_node/Package-Transformation-Options.html">Package Transformations</a>.</p><p><strong>Parameterized Packages</strong> are a new type of package transformations
that will be able to tweak an even wider array of compile-time
options, such as removing unused dependencies or building a package
with support for just a specific locale. These will have a wide
variety of applications, ranging from High-Performance Computing to
Embedded Systems and could also help tackle a few of Guix's issues
like large binary sizes and dense dependency graphs.</p><h1>What would these look like?</h1><p>The <em>syntax</em> for parameterized packages is still under heavy
deliberation, however the final syntax will have the following
features:</p><ul><li>Maintainers will be able to specify what combinations of
parameters a package supports, along with a <em>default configuration</em>
of parameters for a given package.</li><li>Users will be able to pass parameters they want enabled or disabled
through <code>--with-parameters</code> which will then get validated against the
valid combinations specified by maintainers before being run</li><li>For a given package and a given set of parameters, only those in
the package's parameter specification will be used</li><li>Users will be able to specify a global parameter transform that
will apply to all packages. Packages will be built with the
<em>default</em> configuration if the global transform creates an
invalid configuration.</li></ul><h1>Potential Problems with Parameterization</h1><h3>Combinatorial Explosion of Variants</h3><p>One of the biggest and most obvious issues with parameters is the
combinatorial explosion of package variants they will create. One way
to address this is to use tools to calculate and regulate allowed
complexity; one such tool could be a function that takes a
parameter combination specification and returns the number of variants
it could create.</p><h3>Increase in Maintenance Required</h3><p>Another concern is that this will lead to an increase in the workload
of maintainers, who have been generously volunteering their time and
have a lot of work as is. Hence, we will be treating parameters
the same way we have been treating other package transformations-
they are not expected to be stable, and should be used at the
user's discretion.</p><h3>Availability of Substitutes for Variants</h3><p>Lastly, parameterization could lead to an exponential increase in
the number of substitutes build farms will have to create, and
thus as such there are only plans on building substitutes for
default and very popular parameter combinations.</p><h3>Other topics under discussion</h3><p>Some of the other points of discussion with respect to parameters are</p><ul><li><strong>Scope</strong>: Parameterization has a very wide and overarching scope,
and it would be useful to have guidelines in place for when a
particular property should be considered for parameterization</li><li><strong>Syntax</strong>: There are many proposed syntax designs for
parameterization, and more are welcome! The final syntax will most
probably be an amalgamation of the best parts of all proposed designs.</li><li><strong>Substitutes</strong>: There is a lot of discussion on exactly what
parameter combinations should be considered for substitutes; while it
is obvious that it won't be possible to build <em>all</em> combinations, some
important combinations can and should be considered for having
substitutes built. We could perhaps have a separate category of
parameter combinations that would both receive substitutes and
support, and make these combinations discoverable through the
UI. Another suggestion is to have user-run channels for specific build
combinations, like for example there could be a RISC-V specific
channel supplying substitutes for the users running RISC-V.</li></ul><p>If you would like to join the discussion, check out <a href="https://lists.gnu.org/archive/html/guix-devel/2023-05/msg00156.html">this mailing list
discussion about this project</a>,
and also have a look at the <a href="https://lists.gnu.org/archive/html/guix-devel/2020-11/msg00312.html">original thread</a> about parameterization.</p><h1>Conclusion</h1><p>Parameters hold the potential to greatly increase Guix's flexibility,
but will also lead to greater complexity. In my opinion, Guix is
uniquely positioned to take full advantage of the customizability
provided by compile-time options while also enjoying relative
stability thanks to its transactional nature.</p><h4>About Me</h4><p>I'm a student studying Mathematics and ECE at BITS Pilani, and I love
computers and free software. I am the president of my university's
equivalent of a Free Software Advocacy Group, and I am also one of the
system administrators for my university's High-Performance Computing
System. As an advocate for free software and a Lisp user, I naturally
fell in love with GNU Guix when I discovered it. I have also used
Gentoo for some time in the past, which motivated me to try and bring
something similar to USE flags to Guix. You can find my blog at
<a href="https://blog.lispy.tech">blog.lispy.tech</a>, where I will be frequently
posting updates about the project.</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package manager
and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects user
freedom</a>.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system
distribution for i686, x86_64, ARMv7, AArch64 and POWER9 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package
management, per-user profiles, and garbage collection. When used as a
standalone GNU/Linux distribution, Guix offers a declarative,
stateless approach to operating system configuration management. Guix
is highly customizable and hackable through
<a href="https://www.gnu.org/software/guile">Guile</a> programming interfaces and
extensions to the <a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2023/dissecting-guix-part-3-g-expressions//Dissecting Guix, Part 3: G-Expressions(2023-04-19T12:30:00Z2023-04-19T12:30:00Z Welcome back to Dissecting Guix !
Last time, we discussed monads ,
the functional programming idiom used by Guix to thread a store connection
through a series of store-related operations. Today, we'll be talking about a concept rather more specific to Guix:
g-expressions . Being an implementation of the Scheme language, Guile is built
around s-expressions , which can
represent, as the saying goes, code as data , thanks to the simple structure of
Scheme forms. As Guix's package recipes are written in Scheme, it naturally needs some way to
represent code that is to be…<p>Welcome back to <a href="https://guix.gnu.org/en/blog/tags/dissecting-guix">Dissecting Guix</a>!
Last time, we discussed <a href="https://guix.gnu.org/en/blog/2023/dissecting-guix-part-2-the-store-monad">monads</a>,
the functional programming idiom used by Guix to thread a store connection
through a series of store-related operations.</p><p>Today, we'll be talking about a concept rather more specific to Guix:
<em>g-expressions</em>. Being an implementation of the Scheme language, Guile is built
around <a href="https://en.wikipedia.org/wiki/S-expression"><em>s-expressions</em></a>, which can
represent, as the saying goes, <em>code as data</em>, thanks to the simple structure of
Scheme forms.</p><p>As Guix's package recipes are written in Scheme, it naturally needs some way to
represent code that is to be run only when the package is built. Additionally,
there needs to be some way to reference dependencies and retrieve output paths;
otherwise, you wouldn't be able to, for instance, create a phase to install a
file in the output directory.</p><p>So, how do we implement this "deferred" code? Well, initially Guix used plain
old s-expressions for this purpose.</p><h1>Once Upon a Time</h1><p>Let's say we want to create a store item that's just a symlink to the
<code>bin/irssi</code> file of the <code>irssi</code> package. How would we do that with an
s-expression? Well, the s-expression itself, which we call the <em>builder</em>, is
fairly simple:</p><pre><code class="language-scheme">(define sexp-builder
`(let* ((out (assoc-ref %outputs "out"))
(irssi (assoc-ref %build-inputs "irssi"))
(bin/irssi (string-append irssi "/bin/irssi")))
(symlink bin/irssi out)))</code></pre><p>If you aren't familliar with the "quoting" syntax used to create s-expressions,
I strongly recommend that you read the excellent Scheme Primer; specifically,
section 7, <a href="https://spritely.institute/static/papers/scheme-primer.html#scheme-lists-and-cons"><em>Lists and
"cons"</em></a>
and section 11, <a href="https://spritely.institute/static/papers/scheme-primer.html#scheme-extensibility"><em>On the extensibility of Scheme (and Lisps in
general)</em></a></p><p>The <code>%outputs</code> and <code>%build-inputs</code> variables are bound within builder scripts to
<em>association lists</em>, which are lists of pairs that act like key/value stores,
for instance:</p><pre><code class="language-scheme">'(("foo" . "bar")
("floob" . "blarb")
("fvoolag" . "bvarlag"))</code></pre><p>To retrieve values from association lists, which are often referred to as
<em>alists</em>, we use the <code>assoc-ref</code> procedure:</p><pre><code class="language-scheme">(assoc-ref '(("boing" . "bouncy")
("floing" . "flouncy"))
"boing")
⇒ "bouncy"</code></pre><p><code>%outputs</code>, as the name might suggest, maps derivation output names to the paths
of their respective store items, the default output being <code>out</code>, and
<code>%build-inputs</code> maps inputs labels to their store items.</p><p>The builder is the easy part; we now need to turn it into a derivation and tell
it what <code>"irssi"</code> actually refers to. For this, we use the
<code>build-expression->derivation</code> procedure from <code>(guix derivations)</code>:</p><pre><code class="language-scheme">(use-modules (guix derivations)
(guix packages)
(guix store)
(gnu packages guile)
(gnu packages irc))
(with-store store
(let ((guile-3.0-drv (package-derivation store guile-3.0))
(irssi-drv (package-derivation store irssi)))
(build-expression->derivation store "irssi-symlink" sexp-builder
#:guile-for-build guile-3.0-drv
#:inputs `(("irssi" ,irssi-drv)))))
⇒ #<derivation /gnu/store/…-irssi-symlink.drv => /gnu/store/…-irssi-symlink …></code></pre><p>There are several things to note here:</p><ul><li>The inputs <em>must</em> all be derivations, so we need to first convert the packages
using <code>package-derivation</code>.</li><li>We need to explicitly set <code>#:guile-for-build</code>; there's no default value.</li><li>The <code>build-expression->derivation</code> and <code>package-derivation</code> procedures are
<em>not</em> monadic, so we need to explicitly pass them the store connection.</li></ul><p>The shortcomings of using s-expressions in this way are numerous: we have to
convert everything to a derivation before using it, and <em>inputs are not an
inherent aspect of the builder</em>. G-expressions were designed to overcome these
issues.</p><h1>Premortem Examination</h1><p>A g-expression is fundamentally a record of type <code><gexp></code>, which is, naturally,
defined in <code>(guix gexp)</code>. The two most important fields of this record type,
out of a total of five, are <code>proc</code> and <code>references</code>; the former is a procedure
that returns the equivalent s-expression, the latter a list containing
everything from the "outside world" that's used by the g-expression.</p><p>When we want to turn the g-expression into something that we can actually run as
code, we combine these two fields by first building any g-expression inputs that
can become derivations (leaving alone those that cannot), and then passing the
built <code>references</code> as the arguments of <code>proc</code>.</p><p>Here's an example g-expression that is essentially equivalent to our
<code>sexp-builder</code>:</p><pre><code class="language-scheme">(use-modules (guix gexp))
(define gexp-builder
#~(symlink #$(file-append irssi "/bin/irssi")
#$output))</code></pre><p><code>gexp-builder</code> is far more concise than <code>sexp-builder</code>; let's examine the syntax
and the <code><gexp></code> object we've created. To make a g-expression, we use the <code>#~</code>
syntax, equivalent to the <code>gexp</code> macro, rather than the <code>quasiquote</code> backtick
used to create s-expressions.</p><p>When we want to embed values from outside as references, we use <code>#$</code>, or
<code>ungexp</code>, which is, in appearance if not function, equivalent to <code>unquote</code>
(<code>,</code>). <code>ungexp</code> can accept any of four reference types:</p><ul><li>S-expressions (strings, lists, etc), which will be embedded literally.</li><li>Other g-expressions, embedded literally.</li><li>Expressions returning any sort of object that can be lowered into a
derivation, such as <code><package></code>, embedding that object's <code>out</code> store item; if
the expression is specifically a symbol bound to a buildable object, you can
optionally follow it with a colon and an alternative output name, so
<code>package:lib</code> is permitted, but <code>(get-package):lib</code> isn't.</li><li>The symbol <code>output</code>, embedding an output path. Like symbols bound to
buildable objects, this can be followed by a colon and the output name that
should be used rather than the default <code>out</code>.</li></ul><p>All these reference types will be represented by <code><gexp-input></code> records in the
<code>references</code> field, except for the last kind, which will become <code><gexp-output></code>
records. To give an example of each type of reference (with the return value
output formatted for easier reading):</p><pre><code class="language-scheme">(use-modules (gnu packages glib))
#~(list #$"foobar" ;s-expression
#$#~(string-append "foo" "bar") ;g-expression
#$(file-append irssi "/bin/irssi") ;buildable object (expression)
#$glib:bin ;buildable object (symbol)
#$output:out) ;output
⇒ #<gexp (list #<gexp-input "foobar":out>
#<gexp-input #<gexp (string-append "foo" "bar") …>:out>
#<gexp-input #<file-append #<package irssi@1.4.3 …> "/bin/irssi">:out>
#<gexp-input #<package glib@2.70.2 …>:bin>
#<gexp-output out>) …></code></pre><p>Note the use of <code>file-append</code> in both the previous example and <code>gexp-builder</code>;
this procedure produces a <code><file-append></code> object that builds its first argument
and is embedded as the concatenation of the first argument's output path and the
second argument, which should be a string. For instance,
<code>(file-append irssi "/bin/irssi")</code> builds <code>irssi</code> and expands to
<code>/gnu/store/…-irssi/bin/irssi</code>, rather than the <code>/gnu/store/…-irssi</code> that the
package alone would be embedded as.</p><p>So, now that we have a g-expression, how do we turn it into a derivation? This
process is known as <em>lowering</em>; it entails the use of the aptly-named
<code>lower-gexp</code> monadic procedure to combine <code>proc</code> and <code>references</code> and produce a
<code><lowered-gexp></code> record, which acts as a sort of intermediate representation
between g-expressions and derivations. We can piece apart this lowered form to
get a sense of what the final derivation's builder script would look like:</p><pre><code class="language-scheme">(define lowered-gexp-builder
(with-store store
(run-with-store store
(lower-gexp gexp-builder))))
(lowered-gexp-sexp lowered-gexp-builder)
⇒ (symlink
"/gnu/store/…-irssi-1.4.3/bin/irssi"
((@ (guile) getenv) "out"))</code></pre><p>And there you have it: a s-expression compiled from a g-expression, ready to be
written into a builder script file in the store. So, how exactly do you turn
this into said derivation?</p><p>Well, it turns out that there isn't an interface for turning lowered
g-expressions into derivations, only one for turning regular g-expressions into
derivations that first uses <code>lower-gexp</code>, then implements the aforementioned
conversion internally, rather than outsourcing it to some other procedure, so
that's what we'll use.</p><p>Unsurprisingly, that procedure is called <code>gexp->derivation</code>, and unlike its
s-expression equivalent, it's monadic. (<code>build-expression->derivation</code> and
other deprecated procedures were in Guix since before the monads system
existed.)</p><pre><code class="language-scheme">(with-store store
(run-with-store store
(gexp->derivation "irssi-symlink" gexp-builder)))
⇒ #<derivation /gnu/store/…-irssi-symlink.drv => /gnu/store/…-irssi-symlink …></code></pre><p>Finally, we have a g-expression-based equivalent to the derivation we earlier
created with <code>build-expression->derivation</code>! Here's the code we used for the
s-expression version in full:</p><pre><code class="language-scheme">(define sexp-builder
`(let* ((out (assoc-ref %outputs "out"))
(irssi (assoc-ref %build-inputs "irssi"))
(bin/irssi (string-append irssi "/bin/irssi")))
(symlink bin/irssi out)))
(with-store store
(let ((guile-3.0-drv (package-derivation store guile-3.0))
(irssi-drv (package-derivation store irssi)))
(build-expression->derivation store "irssi-symlink" sexp-builder
#:guile-for-build guile-3.0-drv
#:inputs `(("irssi" ,irssi-drv)))))</code></pre><p>And here's the g-expression equivalent:</p><pre><code class="language-scheme">(define gexp-builder
#~(symlink #$(file-append irssi "/bin/irssi")
#$output))
(with-store store
(run-with-store store
(gexp->derivation "irssi-symlink" gexp-builder)))</code></pre><p>That's a lot of complexity abstracted away! For more complex packages and
services, especially, g-expressions are a lifesaver; you can refer to the output
paths of inputs just as easily as you would a string constant. You do, however,
have to watch out for situations where <code>ungexp-native</code>, written as <code>#+</code>, would
be preferable over regular <code>ungexp</code>, and that's something we'll discuss later.</p><p>A brief digression before we continue: if you'd like to look inside a <code><gexp></code>
record, but you'd rather not build anything, you can use the
<code>gexp->approximate-sexp</code> procedure, which replaces all references with dummy
values:</p><pre><code class="language-scheme">(gexp->approximate-sexp gexp-builder)
⇒ (symlink (*approximate*) (*approximate*))</code></pre><h1>The Lowerable-Object Hardware Shop</h1><p>We've seen two examples already of records we can turn into derivations, which
are generally referred to as <em>lowerable objects</em> or <em>file-like objects</em>:</p><ul><li><code><package></code>, a Guix package.</li><li><code><file-append></code>, which wraps another lowerable object and appends a string to
the embedded output path when <code>ungexp</code>ed.</li></ul><p>There are many more available to us. Recall from the previous post,
<a href="https://guix.gnu.org/en/blog/2023/dissecting-guix-part-2-the-store-monad"><em>The Store Monad</em></a>,
that Guix provides the two monadic procedures <code>text-file</code> and <code>interned-file</code>,
which can be used, respectively, to put arbitrary text or files from the
filesystem in the store, returning the path to the created item.</p><p>This doesn't work so well with g-expressions, though; you'd have to wrap each
<code>ungexp</code>ed use of either of them with
<code>(with-store store (run-with-store store …))</code>, which would be quite tedious.
Thankfully, <code>(guix gexp)</code> provides the <code>plain-file</code> and <code>local-file</code> procedures,
which return equivalent lowerable objects. This code example builds a directory
containing symlinks to files greeting the world:</p><pre><code class="language-scheme">(use-modules (guix monads)
(ice-9 ftw)
(ice-9 textual-ports))
(define (build-derivation monadic-drv)
(with-store store
(run-with-store store
(mlet* %store-monad ((drv monadic-drv))
(mbegin %store-monad
;; BUILT-DERIVATIONS is the monadic version of BUILD-DERIVATIONS.
(built-derivations (list drv))
(return (derivation-output-path
(assoc-ref (derivation-outputs drv) "out"))))))))
(define world-greeting-output
(build-derivation
(gexp->derivation "world-greeting"
#~(begin
(mkdir #$output)
(symlink #$(plain-file "hi-world"
"Hi, world!")
(string-append #$output "/hi"))
(symlink #$(plain-file "hello-world"
"Hello, world!")
(string-append #$output "/hello"))
(symlink #$(plain-file "greetings-world"
"Greetings, world!")
(string-append #$output "/greetings"))))))
;; We turn the list into multiple values using (APPLY VALUES …).
(apply values
(map (lambda (file-path)
(let* ((path (string-append world-greeting-output "/" file-path))
(contents (call-with-input-file path get-string-all)))
(list path contents)))
;; SCANDIR from (ICE-9 FTW) returns the list of all files in a
;; directory (including ``.'' and ``..'', so we remove them with the
;; second argument, SELECT?, which specifies a predicate).
(scandir world-greeting-output
(lambda (path)
(not (or (string=? path ".")
(string=? path "..")))))))
⇒ ("/gnu/store/…-world-greeting/greetings" "Greetings, world!")
⇒ ("/gnu/store/…-world-greeting/hello" "Hello, world!")
⇒ ("/gnu/store/…-world-greeting/hi" "Hi, world!")</code></pre><p>Note that we define a procedure for building the output; we will need to build
more derivations in a very similar fashion later, so it helps to have this to
reuse instead of copying the code in <code>world-greeting-output</code>.</p><p>There are many other useful lowerable objects available as part of the gexp
library. These include <code>computed-file</code>, which accepts a gexp that builds
the output file, <code>program-file</code>, which creates an executable Scheme script in
the store using a g-expression, and <code>mixed-text-file</code>, which allows you to,
well, mix text and lowerable objects; it creates a file from the concatenation
of a sequence of strings and file-likes. The
<a href="https://guix.gnu.org/manual/en/html_node/G_002dExpressions.html">G-Expressions</a>
manual page has more details.</p><p>So, you may be wondering, at this point: there's so many lowerable objects
included with the g-expression library, surely there must be a way to define
more? Naturally, there is; this is Scheme, after all! We simply need to
acquaint ourselves with the <code>define-gexp-compiler</code> macro.</p><p>The most basic usage of <code>define-gexp-compiler</code> essentially creates a procedure
that takes as arguments a record to lower, the host system, and the target
system, and returns a derivation or store item as a monadic value in
<code>%store-monad</code>.</p><p>Let's try implementing a lowerable object representing a file that greets the
world. First, we'll define the record type:</p><pre><code class="language-scheme">(use-modules (srfi srfi-9))
(define-record-type <greeting-file>
(greeting-file greeting)
greeting?
(greeting greeting-file-greeting))</code></pre><p>Now we use <code>define-gexp-compiler</code> like so; note how we can use <code>lower-object</code>
to compile down any sort of lowerable object into the equivalent store item or
derivation; essentially, <code>lower-object</code> is just the procedure for applying the
right gexp-compiler to an object:</p><pre><code class="language-scheme">(use-modules (ice-9 i18n))
(define-gexp-compiler (greeting-file-compiler
(greeting-file <greeting-file>)
system target)
(lower-object
(let ((greeting (greeting-file-greeting greeting-file)))
(plain-file (string-append greeting "-greeting")
(string-append (string-locale-titlecase greeting) ", world!")))))</code></pre><p>Let's try it out now. Here's how we could rewrite our greetings directory
example from before using <code><greeting-file></code>:</p><pre><code class="language-scheme">(define world-greeting-2-output
(build-derivation
(gexp->derivation "world-greeting-2"
#~(begin
(mkdir #$output)
(symlink #$(greeting-file "hi")
(string-append #$output "/hi"))
(symlink #$(greeting-file "hello")
(string-append #$output "/hello"))
(symlink #$(greeting-file "greetings")
(string-append #$output "/greetings"))))))
(apply values
(map (lambda (file-path)
(let* ((path (string-append world-greeting-2-output
"/" file-path))
(contents (call-with-input-file path get-string-all)))
(list path contents)))
(scandir world-greeting-2-output
(lambda (path)
(not (or (string=? path ".")
(string=? path "..")))))))
⇒ ("/gnu/store/…-world-greeting-2/greetings" "Greetings, world!")
⇒ ("/gnu/store/…-world-greeting-2/hello" "Hello, world!")
⇒ ("/gnu/store/…-world-greeting-2/hi" "Hi, world!")</code></pre><p>Now, this is probably not worth a whole new gexp-compiler. How about something
a bit more complex? Sharp-eyed readers who are trying all this in the REPL may
have noticed the following output when they used <code>define-gexp-compiler</code>
(formatted for ease of reading):</p><pre><code class="language-scheme">⇒ #<<gexp-compiler>
type: #<record-type <greeting-file>>
lower: #<procedure … (greeting-file system target)>
expand: #<procedure default-expander (thing obj output)>></code></pre><p>Now, the purpose of <code>type</code> and <code>lower</code> is self-explanatory, but what's this
<code>expand</code> procedure here? Well, if you recall <code>file-append</code>, you may realise
that the text produced by a gexp-compiler for embedding into a g-expression
doesn't necessarily have to be the exact output path of the produced derivation.</p><p>There turns out to be another way to write a <code>define-gexp-compiler</code> form that
allows you to specify <em>both</em> the lowering procedure, which produces the
derivation or store item, and the expanding procedure, which produces the text.</p><p>Let's try making another new lowerable object; this one will let us build a
Guile package and expand to the path to its module directory. Here's our
record:</p><pre><code class="language-scheme">(define-record-type <module-directory>
(module-directory package)
module-directory?
(package module-directory-package))</code></pre><p>Here's how we define both a compiler and expander for our new record:</p><pre><code class="language-scheme">(use-modules (gnu packages guile)
(guix utils))
(define lookup-expander (@@ (guix gexp) lookup-expander))
(define-gexp-compiler module-directory-compiler <module-directory>
compiler => (lambda (obj system target)
(let ((package (module-directory-package obj)))
(lower-object package system #:target target)))
expander => (lambda (obj drv output)
(let* ((package (module-directory-package obj))
(expander (or (lookup-expander package)
(lookup-expander drv)))
(out (expander package drv output))
(guile (or (lookup-package-input package "guile")
guile-3.0))
(version (version-major+minor
(package-version guile))))
(string-append out "/share/guile/site/" version))))</code></pre><p>Let's try this out now:</p><pre><code class="language-scheme">(use-modules (gnu packages guile-xyz))
(define module-directory-output/guile-webutils
(build-derivation
(gexp->derivation "module-directory-output"
#~(symlink #$(module-directory guile-webutils) #$output))))
(readlink module-directory-output/guile-webutils)
⇒ "/gnu/store/…-guile-webutils-0.1-1.d309d65/share/guile/site/3.0"
(scandir module-directory-output/guile-webutils)
⇒ ("." ".." "webutils")
(define module-directory-output/guile2.2-webutils
(build-derivation
(gexp->derivation "module-directory-output"
#~(symlink #$(module-directory guile2.2-webutils) #$output))))
(readlink module-directory-output/guile2.2-webutils)
⇒ "/gnu/store/…-guile-webutils-0.1-1.d309d65/share/guile/site/2.2"
(scandir module-directory-output/guile2.2-webutils)
⇒ ("." ".." "webutils")</code></pre><p>Who knows why you'd want to do this, but it certainly works! We've looked at
why we need g-expressions, how they work, and how to extend them, and we've now
only got two more advanced features to cover: cross-build support, and modules.</p><h1>Importing External Modules</h1><p>Let's try using one of the helpful procedures from the <code>(guix build utils)</code>
module in a g-expression.</p><pre><code class="language-scheme">(define simple-directory-output
(build-derivation
(gexp->derivation "simple-directory"
#~(begin
(use-modules (guix build utils))
(mkdir-p (string-append #$output "/a/rather/simple/directory"))))))</code></pre><p>Looks fine, right? We've even got a <code>use-modules</code> in th--</p><pre><code class="language-Scheme">ERROR:
1. &store-protocol-error:
message: "build of `/gnu/store/…-simple-directory.drv' failed"
status: 100</code></pre><p>OUTRAGEOUS. Fortunately, there's an explanation to be found in the Guix build
log directory, <code>/var/log/guix/drvs</code>; locate the file using the first two
characters of the store hash as the subdirectory, and the rest as the file name,
and remember to use <code>zcat</code> or <code>zless</code>, as the logs are gzipped:</p><pre><code class="language-scheme">Backtrace:
9 (primitive-load "/gnu/store/…")
In ice-9/eval.scm:
721:20 8 (primitive-eval (begin (use-modules (guix build #)) (?)))
In ice-9/psyntax.scm:
1230:36 7 (expand-top-sequence ((begin (use-modules (guix ?)) #)) ?)
1090:25 6 (parse _ (("placeholder" placeholder)) ((top) #(# # ?)) ?)
1222:19 5 (parse _ (("placeholder" placeholder)) ((top) #(# # ?)) ?)
259:10 4 (parse _ (("placeholder" placeholder)) (()) _ c&e (eval) ?)
In ice-9/boot-9.scm:
3927:20 3 (process-use-modules _)
222:17 2 (map1 (((guix build utils))))
3928:31 1 (_ ((guix build utils)))
3329:6 0 (resolve-interface (guix build utils) #:select _ #:hide ?)
ice-9/boot-9.scm:3329:6: In procedure resolve-interface:
no code for module (guix build utils)</code></pre><p>It turns out <code>use-modules</code> can't actually find <code>(guix build utils)</code> at all.
There's no typo; it's just that to ensure the build is isolated, Guix builds
<code>module-import</code> and <code>module-importe-compiled</code> directories, and sets the
<em>Guile module path</em> within the build environment to contain said directories,
along with those containing the Guile standard library modules.</p><p>So, what to do? Turns out one of the fields in <code><gexp></code> is <code>modules</code>, which,
funnily enough, contains the names of the modules which will be used to build
the aforementioned directories. To add to this field, we use the
<code>with-imported-modules</code> macro. (<code>gexp->derivation</code> <em>does</em> provide a <code>modules</code>
parameter, but <code>with-imported-modules</code> lets you add the required modules
directly to the g-expression value, rather than later on.)</p><pre><code class="language-scheme">(define simple-directory-output
(build-derivation
(gexp->derivation "simple-directory"
(with-imported-modules '((guix build utils))
#~(begin
(use-modules (guix build utils))
(mkdir-p (string-append #$output "/a/rather/simple/directory")))))))
simple-directory-output
⇒ "/gnu/store/…-simple-directory"</code></pre><p>It works, yay. It's worth noting that while passing just the list of modules to
<code>with-imported-modules</code> works in this case, this is only because
<code>(guix build utils)</code> has no dependencies on other Guix modules. Were we to try
adding, say, <code>(guix build emacs-build-system)</code>, we'd need to use the
<code>source-module-closure</code> procedure to add its dependencies to the list:</p><pre><code class="language-scheme">(use-modules (guix modules))
(source-module-closure '((guix build emacs-build-system)))
⇒ ((guix build emacs-build-system)
(guix build gnu-build-system)
(guix build utils)
(guix build gremlin)
(guix elf)
(guix build emacs-utils))</code></pre><p>Here's another scenario: what if we want to use a module not from Guix or Guile
but a third-party library? In this example, we'll use <a href="https://github.com/aconchillo/guile-json">guile-json
</a>, a library for converting between
S-expressions and <a href="https://json.org">JavaScript Object Notation</a>.</p><p>We can't just <code>with-imported-modules</code> its modules, since it's not part of Guix,
so <code><gexp></code> provides another field for this purpose: <code>extensions</code>. Each of
these extensions is a lowerable object that produces a Guile package directory;
so usually a package. Let's try it out using the <code>guile-json-4</code> package to
produce a JSON file from a Scheme value within a g-expression.</p><pre><code class="language-scheme">(define helpful-guide-output
(build-derivation
(gexp->derivation "json-file"
(with-extensions (list guile-json-4)
#~(begin
(use-modules (json))
(mkdir #$output)
(call-with-output-file (string-append #$output "/helpful-guide.json")
(lambda (port)
(scm->json '((truth . "Guix is the best!")
(lies . "Guix isn't the best!"))
port))))))))
(call-with-input-file
(string-append helpful-guide-output "/helpful-guide.json")
get-string-all)
⇒ "{\"truth\":\"Guix is the best!\",\"lies\":\"Guix isn't the best!\"}"</code></pre><p>Amen to that, <code>helpful-guide.json</code>. Before we continue on to cross-compilation,
there's one last feature of <code>with-imported-modules</code> you should note. We can
add modules to a g-expression by name, but we can also create entirely new ones
using lowerable objects, such as in this pattern, which is used in several
places in the Guix source code to make an appropriately-configured
<code>(guix config)</code> module available:</p><pre><code class="language-scheme">(with-imported-modules `(((guix config) => ,(make-config.scm))
…)
…)</code></pre><p>In case you're wondering, <code>make-config.scm</code> is found in <code>(guix self)</code> and
returns a lowerable object that compiles to a version of the <code>(guix config)</code>
module, which contains constants usually substituted into the source code at
compile time.</p><h1>Native <code>ungexp</code></h1><p>There is another piece of syntax we can use with g-expressions, and it's called
<code>ungexp-native</code>. This helps us distinguish between native inputs and regular
host-built inputs in cross-compilation situations. We'll cover
cross-compilation in detail at a later date, but the gist of it is that it
allows you to compile a derivation for one architecture X, the target, using a
machine of architecture Y, the host, and Guix has excellent support for it.</p><p>If we cross-compile a g-expression G that <em>non-natively</em> <code>ungexp</code>s L1, a
lowerable object, from architecture Y to architecture X, both G and L1 will be
compiled for architecture X. However, if G <em>natively</em> <code>ungexp</code>s L1, G will be
compiled for X and L1 for Y.</p><p>Essentially, we use <code>ungexp-native</code> in situations where there would be no
difference between compiling on different architectures (for instance, if <code>L1</code>
were a <code>plain-file</code>), or where using L1 built for X would actually <em>break</em> G
(for instance, if <code>L1</code> corresponds to a compiled executable that needs to be run
during the build; the executable would fail to run on Y if it was built for X.)</p><p>The <code>ungexp-native</code> macro naturally has a corresponding reader syntax, <code>#+</code>, and
there's also <code>ungexp-native-splicing</code>, which is written as <code>#+@</code>. These two
pieces of syntax are used in the same way as their regular counterparts.</p><h1>Conclusion</h1><p>What have we learned in this post? To summarise:</p><ul><li>G-expressions are essentially abstractions on top of s-expressions used in
Guix to stage code, often for execution within a build environment or a
Shepherd service script.</li><li>Much like you can <code>unquote</code> external values within a <code>quasiquote</code> form, you
can <code>ungexp</code> external values to access them within a <code>gexp</code> form. The key
difference is that you may use not only s-expressions with <code>ungexp</code>, but other
g-expressions and lowerable objects too.</li><li>When a lowerable object is used with <code>ungexp</code>, the g-expression ultimately
receives the path to the object's store item (or whatever string the lowerable
object's expander produces), rather than the object itself.</li><li>A lowerable object is any record that has a "g-expression compiler" defined
for it using the <code>define-gexp-compiler</code> macro. G-expression compilers always
contain a <code>compiler</code> procedure, which converts an appropriate record into a
derivation, and sometimes an <code>expander</code> procedure, which produces the string
that is to be expanded to within g-expressions when the object is <code>ungexp</code>ed.</li><li>G-expressions record the list of modules available in their environment, which
you may expand using <code>with-imported-modules</code> to add Guix modules, and
<code>with-extensions</code> to add modules from third-party Guile packages.</li><li><code>ungexp-native</code> may be used within g-expressions to compile lowerable objects
for the host rather than the target system in cross-compilation scenarios.</li></ul><p>Mastering g-expressions is essential to understanding Guix's inner workings, so
the aim of this blog post is to be as thorough as possible. However, if you
still find yourself with questions, please don't hesitate to stop by at the IRC
channel <code>#guix:libera.chat</code> and mailing list <code>help-guix@gnu.org</code>; we'll be glad
to assist you!</p><p>Also note that due to the centrality of g-expressions to Guix, there exist a
plethora of alternative resources on this topic; here are some which you may
find useful:</p><ul><li>Arun Isaac's
<a href="https://www.systemreboot.net/post/deploy-scripts-using-g-expressions">post</a>
on using g-expressions with <code>guix deploy</code>.</li><li>Marius Bakke's
<a href="https://gexp.no/blog/guix-drops-part-3-g-expressions.html">"Guix Drops" post</a>
which explains g-expressions in a more "top-down" way.</li><li>This 2020
<a href="https://archive.fosdem.org/2020/schedule/event/gexpressionsguile/">FOSDEM talk</a>
by Christopher Marusich on the uses of g-expressions.</li><li>And, of course, the one and only original
<a href="https://hal.inria.fr/hal-01580582v1">g-expression paper</a> by Ludovic Courtès,
the original author of Guix.</li></ul><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package manager and
an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects user
freedom</a>.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, AArch64 and POWER9 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2023/building-toolchains-with-guix//Building Toolchains with GuixMitchell Schmeisser2023-03-15T16:00:00Z2023-03-15T16:00:00Z In order to deploy embedded software using Guix we first need to teach Guix
how to cross-compile it. Since Guix builds everything from source, this
means we must teach Guix how to build our cross-compilation toolchain. The Zephyr Project uses its own fork of GCC with custom configs for
the architectures supported by the project. In this article, we
describe the cross-compilation toolchain we defined for Zephyr; it is
implemented as a Guix
channel . About Zephyr Zephyr is a real-time operating system from the Linux Foundation.
It aims to provide a common environment which…<p>In order to deploy embedded software using Guix we first need to teach Guix
how to cross-compile it. Since Guix builds everything from source, this
means we must teach Guix how to build our cross-compilation toolchain.</p><p>The <a href="https://zephyrproject.org">Zephyr Project</a> uses its own fork of GCC with custom configs for
the architectures supported by the project. In this article, we
describe the cross-compilation toolchain we defined for Zephyr; it is
implemented as a <a href="https://github.com/paperclip4465/guix-zephyr">Guix
channel</a>.</p><h1>About Zephyr</h1><p>Zephyr is a <em>real-time operating system</em> from the Linux Foundation.
It aims to provide a common environment which can target even the most
resource constrained devices.</p><p>Zephyr introduces a module system which allows third parties to share code
in a uniform way. Zephyr uses CMake to perform <em>physical component composition</em>
of these modules. It searches the filesystem and generates scripts which
the toolchain will use to successfully combine those components into a
firmware image.</p><p>The fact that Zephyr provides this mechanism is one reason I chose to
target it in the first place.</p><p>This separation of modules in an embedded context is a really great thing.
It brings many of the advantages that it brings to the Linux world such as
code re-use, smaller binaries, more efficient cache/RAM usage, etc.
It also allows us to work as independent groups and compose
contributions from many teams.</p><p>It also brings all of the complexity. Suddenly most of the problems
that plague traditional deployment now apply to our embedded
system. The fact that the libraries are statically linked at compile
time instead of dynamically at runtime is simply an implementation detail.
I say most because everything is statically linked so there is no runtime
component discovery that needs to be accounted for.</p><h1>Anatomy of a Toolchain</h1><p>Toolchains are responsible for taking high level descriptions of programs
and lowering them down to a series of equivalent machine instructions.
This process involves more than just a compiler. The compiler uses the
<a href="https://www.gnu.org/software/binutils">GNU Binutils</a>
to manipulate its internal representation down to a given architecture.
It also needs the use of the C standard library as well as a few other libraries
needed for some compiler optimizations.</p><p>The C library provides the interface to the underlying kernel. System
calls like <code>write</code> and <code>read</code> are provided by <a href="https://www.gnu.org/software/libc">GNU C Library
(glibc)</a> on most distributions.</p><p>In embedded systems, smaller implementations like <a href="https://sourceware.org/newlib/">RedHat's
newlib</a> and
newlib-nano are used.</p><h1>Bootstrapping a Toolchain</h1><p>In order to compile GCC we need a C library that's been compiled for
our target architecture. How can we cross compile our C library if we
need our C library to build a cross compiler? The solution is to build
a simpler compiler that doesn't require the C library to function.
It will not be capable of as many optimizations and it will be very slow,
however it will be able to build the C libraries as well as the complete version
of GCC.</p><p>In order to build the simpler compiler we need to compile the Binutils to
work with our target architecture.
Binutils can be bootstrapped with our host GCC and have no target dependencies.
More information is available in <a href="https://crosstool-ng.github.io/docs/toolchain-construction/">this
article</a>.</p><p>Doesn't sound so bad right? It isn't... in theory.
However internet forums since time immemorial have been
littered with the laments of those who came before.
From incorrect versions of ISL to the wrong C library being linked
or the host linker being used, etc.
The one commonality between all of these issues is the environment.
Building GCC is difficult because isolating build environments is hard.</p><p>In fact as of <code>v0.14.2</code>, the Zephyr “software development kit” (SDK) repository took down the build
instructions and posted a sign that read
"Building this is too complicated, don't worry about it."
(I'm paraphrasing, but
<a href="https://github.com/zephyrproject-rtos/sdk-ng/tree/v0.14.2#build-process">not by much</a>.)</p><p>We will neatly sidestep all of these problems and not
risk destroying or polluting our host system with garbage
by using Guix to manage our environments for us.</p><p>Our toolchain only requires the first pass compiler because
newlib(-nano) is statically linked and introduced to the toolchain
by normal package composition.</p><h1>Defining the Packages</h1><p>All of the base packages are defined in
<a href="https://github.com/paperclip4465/guix-zephyr/blob/master/zephyr/packages/zephyr.scm"><code>zephyr/packages/zephyr.scm</code></a>.
Zephyr modules (coming soon!) are defined in
<a href="https://github.com/paperclip4465/guix-zephyr/blob/master/zephyr/packages/zephyr-xyz.scm"><code>zephyr/packages/zephyr-xyz.scm</code></a>,
following the pattern of other module systems implemented by Guix.</p><h2>Binutils</h2><p>First thing we need to build is the <code>arm-zephyr-eabi</code> binutils.
This is very easy in Guix.</p><pre><code class="language-scheme">(define-public arm-zephyr-eabi-binutils
(let ((xbinutils (cross-binutils "arm-zephyr-eabi")))
(package
(inherit xbinutils)
(name "arm-zephyr-eabi-binutils")
(version "2.38")
(source (origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/zephyrproject-rtos/binutils-gdb")
(commit "6a1be1a6a571957fea8b130e4ca2dcc65e753469")))
(file-name (git-file-name name version))
(sha256 (base32 "0ylnl48jj5jk3jrmvfx5zf8byvwg7g7my7jwwyqw3a95qcyh0isr"))))
(arguments
`(#:tests? #f
,@(substitute-keyword-arguments (package-arguments xbinutils)
((#:configure-flags flags)
`(cons "--program-prefix=arm-zephyr-eabi-" ,flags)))))
(native-inputs
(modify-inputs (package-native-inputs xbinutils)
(prepend texinfo bison flex gmp dejagnu)))
(home-page "https://zephyrproject.org")
(synopsis "Binutils for the Zephyr RTOS"))))</code></pre><p>The function
<a href="https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/cross-base.scm?id=2397f4768091210b0a705ef750f2f38d6946fb89#n84"><code>cross-binutils</code></a>
returns a package which has been configured for the given GNU triplet.
We simply inherit that package and replace the source. The Zephyr build
system expects the binutils to be prefixed with <code>arm-zephyr-eabi-</code> which
is accomplished by adding another flag to the <code>#:configure-flags</code>
argument.</p><p>We can test our package definition using the <code>-L</code> flag with <code>guix build</code>
to add our packages.</p><pre><code class="language-sh">$ guix build -L guix-zephyr zephyr-binutils
/gnu/store/...-zephyr-binutils-2.38</code></pre><p>This directory contains the results of <code>make install</code>.</p><h2>GCC sans libc</h2><p>This one is a bit more involved. Don't be afraid!
This version of GCC wants ISL version 0.15. It's easy enough
to make that happen. Inherit the current version of ISL and swap
out the source and update the version. For most packages the build process doesn't
change that much between versions.</p><pre><code class="language-scheme">(define-public isl-0.15
(package
(inherit isl)
(version "0.15")
(source (origin
(method url-fetch)
(uri (list (string-append "mirror://sourceforge/libisl/isl-"
version ".tar.gz")))
(sha256
(base32
"11vrpznpdh7w8jp4wm4i8zqhzq2h7nix71xfdddp8xnzhz26gyq2"))))))
</code></pre><p>Like the binutils, there is a <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/cross-base.scm?id=2397f4768091210b0a705ef750f2f38d6946fb89#n242"><code>cross-gcc</code>
function</a>
for creating cross-GCC packages. This one accepts keywords specifying
which binutils and libc to use. If libc isn't given (like here), gcc is
configured with many options disabled to facilitate being built without
libc. Therefore we need to add the extra options we want (I got them
from the SDK configuration scripts in the <a href="https://github.com/zephyrproject-rtos/sdk-ng"><code>sdk-ng</code>
Git repository</a> as well as the
commits to use for each of the tools).</p><pre><code class="language-scheme">(define-public gcc-arm-zephyr-eabi-12
(let ((xgcc (cross-gcc "arm-zephyr-eabi"
#:xbinutils zephyr-binutils)))
(package
(inherit xgcc)
(version "12.1.0")
(source (origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/zephyrproject-rtos/gcc")
(commit "0218469df050c33479a1d5be3e5239ac0eb351bf")))
(file-name (git-file-name (package-name xgcc) version))
(sha256
(base32
"1s409qmidlvzaw1ns6jaanigh3azcxisjplzwn7j2n3s33b76zjk"))
(patches (search-patches
"gcc-12-cross-environment-variables.patch"
"gcc-cross-gxx-include-dir.patch"))))
(native-inputs (modify-inputs (package-native-inputs xgcc)
;; Get rid of stock ISL
(delete "isl")
;; Add additional dependencies that xgcc doesn't have
;; including our special ISL
(prepend flex
isl-0.15)))
(arguments
(substitute-keyword-arguments (package-arguments xgcc)
((#:phases phases)
`(modify-phases ,phases
(add-after 'unpack 'fix-genmultilib
(lambda _
(patch-shebang "gcc/genmultilib")))
(add-after 'set-paths 'augment-CPLUS_INCLUDE_PATH
(lambda* (#:key inputs #:allow-other-keys)
(let ((gcc (assoc-ref inputs "gcc")))
;; Remove the default compiler from CPLUS_INCLUDE_PATH to
;; prevent header conflict with the GCC from native-inputs.
(setenv "CPLUS_INCLUDE_PATH"
(string-join (delete (string-append gcc
"/include/c++")
(string-split (getenv
"CPLUS_INCLUDE_PATH")
#\:)) ":"))
(format #t
"environment variable `CPLUS_INCLUDE_PATH' changed to `a`%"
(getenv "CPLUS_INCLUDE_PATH")))))))
((#:configure-flags flags)
;; The configure flags are largely identical to the flags used by the
;; "GCC ARM embedded" project.
`(append (list
"--enable-multilib"
"--with-newlib"
"--with-multilib-list=rmprofile"
"--with-host-libstdcxx=-static-libgcc -Wl,-Bstatic,-lstdc++,-Bdynamic -lm"
"--enable-plugins"
"--disable-decimal-float"
"--disable-libffi"
"--disable-libgomp"
"--disable-libmudflap"
"--disable-libquadmath"
"--disable-libssp"
"--disable-libstdcxx-pch"
"--disable-nls"
"--disable-shared"
"--disable-threads"
"--disable-tls"
"--with-gnu-ld"
"--with-gnu-as"
"--enable-initfini-array")
(delete "--disable-multilib"
,flags)))))
(native-search-paths
(list (search-path-specification
(variable "CROSS_C_INCLUDE_PATH")
(files '("arm-zephyr-eabi/include")))
(search-path-specification
(variable "CROSS_CPLUS_INCLUDE_PATH")
(files '("arm-zephyr-eabi/include" "arm-zephyr-eabi/c++"
"arm-zephyr-eabi/c++/arm-zephyr-eabi")))
(search-path-specification
(variable "CROSS_LIBRARY_PATH")
(files '("arm-zephyr-eabi/lib")))))
(home-page "https://zephyrproject.org")
(synopsis "GCC for the Zephyr RTOS"))))</code></pre><p>This GCC can be built like so.</p><pre><code class="language-sh">$ guix build -L guix-zephyr gcc-cross-sans-libc-arm-zephyr-eabi
/gnu/store/...-gcc-cross-sans-libc-arm-zephyr-eabi-12.1.0-lib
/gnu/store/...-gcc-cross-sans-libc-arm-zephyr-eabi-12.1.0
</code></pre><p>Great! We now have our stage-1 compiler.</p><h2>Newlib(-nano)</h2><p>The newlib package package is quite straight forward (relatively).
It is mostly adding in the relevent configuration flags and patching
the files the <code>patch-shebangs</code> phase missed.</p><pre><code class="language-scheme">(define-public zephyr-newlib
(package
(name "zephyr-newlib")
(version "3.3")
(source (origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/zephyrproject-rtos/newlib-cygwin")
(commit "4e150303bcc1e44f4d90f3489a4417433980d5ff")))
(sha256
(base32 "08qwjpj5jhpc3p7a5mbl7n6z7rav5yqlydqanm6nny42qpa8kxij"))))
(build-system gnu-build-system)
(arguments
`(#:out-of-source? #t
#:configure-flags '("--target=arm-zephyr-eabi"
"--enable-newlib-io-long-long"
"--enable-newlib-io-float"
"--enable-newlib-io-c99-formats"
"--enable-newlib-retargetable-locking"
"--enable-newlib-lite-exit"
"--enable-newlib-multithread"
"--enable-newlib-register-fini"
"--enable-newlib-extra-sections"
"--disable-newlib-wide-orient"
"--disable-newlib-fseek-optimization"
"--disable-newlib-supplied-syscalls"
"--disable-newlib-target-optspace"
"--disable-nls")
#:phases
(modify-phases %standard-phases
(add-after 'unpack 'fix-references-to-/bin/sh
(lambda _
(substitute# '("libgloss/arm/cpu-init/Makefile.in"
"libgloss/arm/Makefile.in"
"libgloss/libnosys/Makefile.in"
"libgloss/Makefile.in")
(("/bin/sh") (which "sh")))
#t)))))
(native-inputs
`(("xbinutils" ,zephyr-binutils)
("xgcc" ,gcc-arm-zephyr-eabi-12)
("texinfo" ,texinfo)))
(home-page "https://www.sourceware.org/newlib/")
(synopsis "C library for use on embedded systems")
(description "Newlib is a C library intended for use on embedded
systems. It is a conglomeration of several library parts that are easily
usable on embedded products.")
(license (license:non-copyleft
"https://www.sourceware.org/newlib/COPYING.NEWLIB"))))</code></pre><p>And the build.</p><pre><code class="language-sh :exports both">$ guix build -L guix-zephyr zephyr-newlib
/gnu/store/...-zephyr-newlib-3.3</code></pre><h2>Complete Toolchain</h2><p><em>Mostly</em> complete. libstdc++ does not build because
<code>arm-zephyr-eabi</code> is not <code>arm-none-eabi</code> so a dynamic link check is
performed/failed. I cannot figure out how crosstool-ng handles this.</p><p>Now that we've got the individual tools it's time to create our complete toolchain.
For this we need to do some package transformations.
Because these transformations are going to have to be done for every combination of
binutils/gcc/newlib it is best to create a function which we can reuse for every version
of the SDK.</p><pre><code class="language-scheme">(define (arm-zephyr-eabi-toolchain xgcc newlib version)
"Produce a cross-compiler zephyr toolchain package with the compiler XGCC and the C\n library variant NEWLIB."
(let ((newlib-with-xgcc
(package
(inherit newlib)
(native-inputs
(modify-inputs (package-native-inputs newlib)
(replace "xgcc" xgcc))))))
(package
(name (string-append "arm-zephyr-eabi"
(if (string=? (package-name newlib-with-xgcc)
"newlib-nano")
"-nano"
"")
"-toolchain"))
(version version)
(source #f)
(build-system trivial-build-system)
(arguments
'(#:modules ((guix build union)
(guix build utils))
#:builder (begin
(use-modules (ice-9 match)
(guix build union)
(guix build utils))
(let ((out (assoc-ref %outputs "out")))
(mkdir-p out)
(match %build-inputs
(((names . directories) ...)
(union-build (string-append out "/arm-zephyr-eabi")
directories)))))))
(inputs `(("binutils" ,zephyr-binutils)
("gcc" ,xgcc)
("newlib" ,newlib-with-xgcc)))
(synopsis "Complete GCC tool chain for ARM zephyrRTOS development")
(description
"This package provides a complete GCC tool chain for ARM
bare metal development with zephyr rtos. This includes the GCC arm-zephyr-eabi cross compiler
and newlib (or newlib-nano) as the C library. The supported programming
language is C.")
(home-page (package-home-page xgcc))
(license (package-license xgcc)))))</code></pre><p>This function creates a special package which consists of the toolchain
in a special directory hierarchy, i.e <code>arm-zephyr-eabi/</code>.
Our complete toolchain definition looks like this.</p><pre><code class="language-scheme">(define-public arm-zephyr-eabi-toolchain-0.15.0
(arm-zephyr-eabi-toolchain gcc-arm-zephyr-eabi-12 zephyr-newlib
"0.15.0"))</code></pre><p>To build:</p><pre><code class="language-sh">$ guix build -L guix-zephyr arm-zephyr-eabi-toolchain
/gnu/store/...-arm-zephyr-eabi-toolchain-0.15.0</code></pre><blockquote><p>Note: Guix now includes a <a href="https://guix.gnu.org/manual/en/html_node/Platforms.html">mechanism to describe
<em>platforms</em></a>
at a high level, and which the <a href="https://guix.gnu.org/manual/en/html_node/Additional-Build-Options.html"><code>--system</code> and <code>--target</code> build
options</a>
build upon. It is not used here but could be a way to better
integrate Zephyr support in the future.</p></blockquote><h1>Integrating with Zephyr Build System</h1><p>Zephyr uses CMake as its build system. It contains numerous CMake files in both the so-called <code>ZEPHYR_BASE</code>,
the zephyr source code repository, as well as a handful in the SDK which help select the correct toolchain
for a given board.</p><p>There are standard locations the build system will look for the SDK. We are not using any of them.
Our SDK lives in the store, immutable forever.
According to <a href="https://docs.zephyrproject.org/3.1.0/develop/west/without-west.html">the Zephyr documentation</a>, the variable <code>ZEPHYR_SDK_INSTALL_DIR</code> needs to point to our custom spot.</p><p>We also need to grab the CMake files from the
<a href="https://github.com/zephyrproject-rtos/sdk-ng">repository</a>
and create a file, <code>sdk_version</code>, which
contains the version string <code>ZEPHYR_BASE</code> uses to find a compatible SDK.</p><p>Along with the SDK proper we need to include a number of
python packages required by the build system.</p><pre><code class="language-scheme">(define-public zephyr-sdk
(package
(name "zephyr-sdk")
(version "0.15.0")
(home-page "https://zephyrproject.org")
(source (origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/zephyrproject-rtos/sdk-ng")
(commit "v0.15.0")))
(file-name (git-file-name name version))
(sha256
(base32
"04gsvh20y820dkv5lrwppbj7w3wdqvd8hcanm8hl4wi907lwlmwi"))))
(build-system trivial-build-system)
(arguments
`(#:modules ((guix build union)
(guix build utils))
#:builder (begin
(use-modules (guix build union)
(ice-9 match)
(guix build utils))
(let ((out (assoc-ref %outputs "out"))
(cmake-scripts (string-append (assoc-ref
%build-inputs
"source")
"/cmake"))
(sdk-out (string-append out "/zephyr-sdk-0.15.0")))
(mkdir-p out)
(match (assoc-remove! %build-inputs "source")
(((names . directories) ...)
(union-build sdk-out directories)))
(copy-recursively cmake-scripts
(string-append sdk-out "/cmake"))
(with-directory-excursion sdk-out
(call-with-output-file "sdk_version"
(lambda (p)
(format p "0.15.0"))))))))
(propagated-inputs (list arm-zephyr-eabi-toolchain-0.15.0
zephyr-binutils
dtc
python-3
python-pyelftools
python-pykwalify
python-pyyaml
python-packaging))
(native-search-paths
(list (search-path-specification
(variable "ZEPHYR_SDK_INSTALL_DIR")
(separator #f)
(files '("")))))
(synopsis "Zephyr SDK")
(description
"zephyr-sdk contains bundles a complete gcc toolchain as well
as host tools like dtc, openocd, qemu, and required python packages.")
(license license:apsl2)))</code></pre><h1>Testing</h1><p>In order to test we will need an environment with the SDK installed.
We can take advantage of <code>guix shell</code> to avoid installing test packages into
our home environment. This way if it causes problems we can just exit the shell
and try again.</p><pre><code class="language-sh">guix shell -L guix-zephyr zephyr-sdk cmake ninja git</code></pre><p><code>ZEPHYR_BASE</code> can be cloned into a temporary workspace to test our toolchain functionality.
(For now. Eventually we will need to create a package for <code>zephyr-base</code> that
our Guix <code>zephyr-build-system</code> can use.)</p><pre><code class="language-sh">mkdir /tmp/zephyr-project
cd /tmp/zephyr-project
git clone https://github.com/zephyrproject-rtos/zephyr
export ZEPHYR_BASE=/tmp/zephyr-project/zephyr</code></pre><p>In order to build for the test board (k64f in this case) we need to get a hold of the vendor
Hardware Abstraction Layers and CMSIS.
(These will also need to become Guix packages to allow the build system to compose modules).</p><pre><code class="language-sh">git clone https://github.com/zephyrproject-rtos/hal_nxp && \
git clone https://github.com/zephyrproject-rtos/cmsis</code></pre><p>To inform the build system about this module we pass it in with <code>-DZEPHYR_MODULES=</code> which is
a semicolon separated list of paths containing a module.yml file.</p><p>To build the hello world sample we use the following incantation.</p><pre><code class="language-sh">cmake -Bbuild $ZEPHYR_BASE/samples/hello_world \
-GNinja \
-DBOARD=frdm_k64f \
-DBUILD_VERSION=3.1.0 \
-DZEPHYR_MODULES="/tmp/zephyr-project/hal_nxp;/tmp/zephyr-project/cmsis" \
&& ninja -Cbuild</code></pre><p>If everything is set up correctly we will end up with a <code>./build</code>
directory with all our build artifacts. The SDK is correctly installed!</p><h1>Conclusion</h1><p>A customized cross toolchain is one of the most difficult pieces of
software to build. Using Guix, we do not need to be afraid of the
complexity! We can fiddle with settings, swap out components, and do
the most brain dead things to our environments without a care in the
world. Just exit the environment and it's like it never happened at
all.</p><p>It highlights one of my favorite aspects of Guix, every package is a
working reference design for you to modify and learn from.</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package manager and
an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects user
freedom</a>.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, AArch64 and POWER9 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2023/dissecting-guix-part-2-the-store-monad//Dissecting Guix, Part 2: The Store Monad(2023-02-20T21:30:00Z2023-02-20T21:30:00Z Hello again! In the last post ,
we briefly mentioned the with-store and run-with-store macros. Today, we'll
be looking at those in further detail, along with the related monad library and
the %store-monad ! Typically, we use monads to chain operations together, and the %store-monad is
no different; it's used to combine operations that work on the Guix store (for
instance, creating derivations, building derivations, or adding data files to
the store). However, monads are a little hard to explain, and from a distance, they seem to
be quite incomprehensible. …<p>Hello again!</p><p>In <a href="https://guix.gnu.org/en/blog/2023/dissecting-guix-part-1-derivations/">the last post</a>,
we briefly mentioned the <code>with-store</code> and <code>run-with-store</code> macros. Today, we'll
be looking at those in further detail, along with the related monad library and
the <a href="https://guix.gnu.org/manual/devel/en/html_node/The-Store-Monad.html"><code>%store-monad</code></a>!</p><p>Typically, we use monads to chain operations together, and the <code>%store-monad</code> is
no different; it's used to combine operations that work on the Guix store (for
instance, creating derivations, building derivations, or adding data files to
the store).</p><p>However, monads are a little hard to explain, and from a distance, they seem to
be quite incomprehensible. So, I want you to erase them from your mind for now.
We'll come back to them later. And be aware that if you can't seem to get your
head around them, it's okay; you can understand most of the architecture of Guix
without understanding monads.</p><h1>Yes, No, Maybe So</h1><p>Let's instead implement another M of functional programming, <em><code>maybe</code></em> values,
representing a value that may or may not exist. For instance, there could be a
procedure that attempts to pop a stack, returning the result if there is one, or
<code>nothing</code> if the stack has no elements.</p><p><code>maybe</code> is a very common feature of statically-typed functional languages, and
you'll see it all over the place in Haskell and OCaml code. However, Guile is
dynamically typed, so we usually use ad-hoc <code>#f</code> values as the "null value"
instead of a proper "nothing" or "none".</p><p>Just for fun, though, we'll implement a proper <code>maybe</code> in Guile. Fire up that
REPL once again, and let's import a bunch of modules that we'll need:</p><pre><code class="language-scheme">(use-modules (ice-9 match)
(srfi srfi-9))</code></pre><p>We'll implement <code>maybe</code> as a record with two fields, <code>is?</code> and <code>value</code>. If the
value contains something, <code>is?</code> will be <code>#t</code> and <code>value</code> will contain the thing
in question, and if it's empty, <code>is?</code>'ll be <code>#f</code>.</p><pre><code class="language-scheme">(define-record-type <maybe>
(make-maybe is? value)
maybe?
(is? maybe-is?)
(value maybe-value))</code></pre><p>Now we'll define constructors for the two possible states:</p><pre><code class="language-scheme">(define (something value)
(make-maybe #t value))
(define (nothing)
(make-maybe #f #f)) ;the value here doesn't matter; we'll just use #f</code></pre><p>And make some silly functions that return optional values:</p><pre><code class="language-scheme">(define (remove-a str)
(if (eq? (string-ref str 0) #\a)
(something (substring str 1))
(nothing)))
(define (remove-b str)
(if (eq? (string-ref str 0) #\b)
(something (substring str 1))
(nothing)))
(remove-a "ahh")
⇒ #<<maybe> is?: #t value: "hh">
(remove-a "ooh")
⇒ #<<maybe> is?: #f value: #f>
(remove-b "bad")
⇒ #<<maybe> is?: #t value: "ad"></code></pre><p>But what if we want to compose the results of these functions?</p><h1>Keeping Your Composure</h1><p>As you might have guessed, this is not fun. Cosplaying as a compiler backend
typically isn't.</p><pre><code class="language-scheme">(let ((t1 (remove-a "abcd")))
(if (maybe-is? t1)
(remove-b (maybe-value t1))
(nothing)))
⇒ #<<maybe> is?: #t value: "cd">
(let ((t1 (remove-a "bbcd")))
(if (maybe-is? t1)
(remove-b (maybe-value t1))
(nothing)))
⇒ #<<maybe> is?: #f value: #f></code></pre><p>I can almost hear the heckling. Even worse, composing three:</p><pre><code class="language-scheme">(let* ((t1 (remove-a "abad"))
(t2 (if (maybe-is? t1)
(remove-b (maybe-value t1))
(nothing))))
(if (maybe-is? t2)
(remove-a (maybe-value t2))
(nothing)))
⇒ #<<maybe> is?: #t value: "d"></code></pre><p>So, how do we go about making this more bearable? Well, one way could be to
make <code>remove-a</code> and <code>remove-b</code> accept <code>maybe</code>s:</p><pre><code class="language-scheme">(define (remove-a ?str)
(match ?str
(($ <maybe> #t str)
(if (eq? (string-ref str 0) #\a)
(something (substring str 1))
(nothing)))
(_ (nothing))))
(define (remove-b ?str)
(match ?str
(($ <maybe> #t str)
(if (eq? (string-ref str 0) #\b)
(something (substring str 1))
(nothing)))
(_ (nothing))))</code></pre><p>Not at all pretty, but it works!</p><pre><code class="language-scheme">(remove-b (remove-a (something "abc")))
⇒ #<<maybe> is?: #t value: "c"></code></pre><p>Still, our procedures now require quite a bit of boilerplate. Might there be a
better way?</p><h1>The Ties That <code>>>=</code> Us</h1><p>First of all, we'll revert to our original definitions of <code>remove-a</code> and
<code>remove-b</code>, that is to say, the ones that take a regular value and return a
<code>maybe</code>.</p><pre><code class="language-scheme">(define (remove-a str)
(if (eq? (string-ref str 0) #\a)
(something (substring str 1))
(nothing)))
(define (remove-b str)
(if (eq? (string-ref str 0) #\b)
(something (substring str 1))
(nothing)))</code></pre><p>What if we tried introducing higher-order procedures (procedures that accept
other procedures as arguments) into the equation? Because we're functional
programmers and we have an unhealthy obsession with that sort of thing.</p><pre><code class="language-scheme">(define (maybe-chain maybe proc)
(if (maybe-is? maybe)
(proc (maybe-value maybe))
(nothing)))
(maybe-chain (something "abc")
remove-a)
⇒ #<<maybe> is?: #t value: "bc">
(maybe-chain (nothing)
remove-a)
⇒ #<<maybe> is?: #f value: #f></code></pre><p>It lives! To make it easier to compose procedures like this, we'll define a
macro that allows us to perform any number of sequenced operations with only one
composition form:</p><pre><code class="language-scheme">(define-syntax maybe-chain*
(syntax-rules ()
((_ maybe proc)
(maybe-chain maybe proc))
((_ maybe proc rest ...)
(maybe-chain* (maybe-chain maybe proc)
rest ...))))
(maybe-chain* (something "abad")
remove-a
remove-b
remove-a)
⇒ #<<maybe> is?: #t value: "d"></code></pre><p>Congratulations, you've just implemented the <code>bind</code> operation, commonly written
as <code>>>=</code>, for our <code>maybe</code> type. And it turns out that a monad is just any
container-like value for which <code>>>=</code> (along with another procedure called
<code>return</code>, which wraps a given value in the simplest possible form of a monad)
has been implemented.</p><p>A more formal definition would be that a monad is a mathematical object composed
of three parts: a type, a <code>bind</code> function, and a <code>return</code> function. So, how do
monads relate to Guix?</p><h1>New Wheel, Old Wheel</h1><p>Now that we've reinvented the wheel, we'd better learn to use the original
wheel. Guix provides a generic, high-level monads library, along with the two
generic monads <code>%identity-monad</code> and <code>%state-monad</code>, and the Guix-specific
<code>%store-monad</code>. Since <code>maybe</code> is not one of them, let's integrate our version
into the Guix monad system!</p><p>First we'll import the module that provides the aforementioned library:</p><pre><code class="language-scheme">(use-modules (guix monads))</code></pre><p>To define a monad's behaviour in Guix, we simply use the <code>define-monad</code> macro,
and provide two procedures: <code>bind</code>, and <code>return</code>.</p><pre><code class="language-scheme">(define-monad %maybe-monad
(bind maybe-chain)
(return something))</code></pre><p><code>bind</code> is just the procedure that we use to compose monadic procedure calls
together, and <code>return</code> is the procedure that wraps values in the most basic form
of the monad. A properly implemented <code>bind</code> and <code>return</code> must follow the
so-called <em>monad laws</em>:</p><ol><li><code>(bind (return x) proc)</code> must be equivalent to <code>(proc x)</code>.</li><li><code>(bind monad return)</code> must be equivalent to just <code>monad</code>.</li><li><code>(bind (bind monad proc-1) proc-2)</code> must be equivalent to
<code>(bind monad (lambda (x) (bind (proc-1 x) proc-2)))</code>.</li></ol><p>Let's verify that our <code>maybe-chain</code> and <code>something</code> procedures adhere to the
monad laws:</p><pre><code class="language-scheme">(define (mlaws-proc-1 x)
(something (+ x 1)))
(define (mlaws-proc-2 x)
(something (+ x 2)))
;; First law: the left identity.
(equal? (maybe-chain (something 0)
mlaws-proc-1)
(mlaws-proc-1 0))
⇒ #t
;; Second law: the right identity.
(equal? (maybe-chain (something 0)
something)
(something 0))
⇒ #t
;; Third law: associativity.
(equal? (maybe-chain (maybe-chain (something 0)
mlaws-proc-1)
mlaws-proc-2)
(maybe-chain (something 0)
(lambda (x)
(maybe-chain (mlaws-proc-1 x)
mlaws-proc-2))))
⇒ #t</code></pre><p>Now that we know they're valid, we can use the <code>with-monad</code> macro to tell Guix
to use these specific implementations of <code>bind</code> and <code>return</code>, and the <code>>>=</code>
macro to thread monads through procedure calls!</p><pre><code class="language-scheme">(with-monad %maybe-monad
(>>= (something "aabbc")
remove-a
remove-a
remove-b
remove-b))
⇒ #<<maybe> is?: #t value: "c"></code></pre><p>We can also now use <code>return</code>:</p><pre><code class="language-scheme">(with-monad %maybe-monad
(return 32))
⇒ #<<maybe> is?: #t value: 32></code></pre><p>But Guix provides many higher-level interfaces than <code>>>=</code> and <code>return</code>, as we
will see. There's <code>mbegin</code>, which evaluates monadic expressions without binding
them to symbols, returning the last one. This, however, isn't particularly
useful with our <code>%maybe-monad</code>, as it's only really usable if the monadic
operations within have side effects, just like the non-monadic <code>begin</code>.</p><p>There's also <code>mlet</code> and <code>mlet*</code>, which <em>do</em> bind the results of monadic
expressions to symbols, and are essentially equivalent to a chain of
<code>(>>= MEXPR (lambda (BINDING) ...))</code>:</p><pre><code class="language-scheme">;; This is equivalent...
(mlet* %maybe-monad ((str -> "abad") ;non-monadic binding uses the -> symbol
(str1 (remove-a str))
(str2 (remove-b str1)))
(remove-a str2))
⇒ #<<maybe> is?: #t value: "d">
;; ...to this:
(with-monad %maybe-monad
(>>= (return "abad")
(lambda (str)
(remove-a str))
(lambda (str1)
(remove-b str1))
(lambda (str2)
(remove-a str2))))</code></pre><p>Various abstractions over these two exist too, such as <code>mwhen</code> (a <code>when</code> plus an
<code>mbegin</code>), <code>munless</code> (an <code>unless</code> plus an <code>mbegin</code>), and <code>mparameterize</code>
(dynamically-scoped value rebinding, like <code>parameterize</code>, in a monadic context).
<code>lift</code> takes a procedure and a monad and creates a new procedure that returns
a monadic value.</p><p>There are also interfaces for manipulating lists wrapped in monads; <code>listm</code>
creates such a list, <code>sequence</code> turns a list of monads into a list wrapped in a
monad, and the <code>anym</code>, <code>mapm</code>, and <code>foldm</code> procedures are like their non-monadic
equivalents, except that they return lists wrapped in monads.</p><p>This is all well and good, you may be thinking, but why does Guix need a monad
library, anyway? The answer is technically that it doesn't. But building on
the monad API makes a lot of things much easier, and to learn why, we're going
to look at one of Guix's built-in monads.</p><h1>In a State</h1><p>Guix implements a monad called <code>%state-monad</code>, and it works with single-argument
procedures returning two values. Behold:</p><pre><code class="language-scheme">(with-monad %state-monad
(return 33))
⇒ #<procedure 21dc9a0 at <unknown port>:1106:22 (state)></code></pre><p>The <code>run-with-state</code> value turns this procedure into an actually useful value,
or, rather, two values:</p><pre><code class="language-scheme">(run-with-state (with-monad %state-monad (return 33))
(list "foo" "bar" "baz"))
⇒ 33
⇒ ("foo" "bar" "baz")</code></pre><p>What can this actually do for us, though? Well, it gets interesting if we do
some <code>>>=</code>ing:</p><pre><code class="language-scheme">(define state-seq
(mlet* %state-monad ((number (return 33)))
(state-push number)))
result
⇒ #<procedure 7fcb6f466960 at <unknown port>:1484:24 (state)>
(run-with-state state-seq (list 32))
⇒ (32)
⇒ (33 32)
(run-with-state state-seq (list 30 99))
⇒ (30 99)
⇒ (33 30 99)</code></pre><p>What is <code>state-push</code>? It's a monadic procedure for <code>%state-monad</code> that takes
whatever's currently in the first value (the primary value) and pushes it onto
the second value (the state value), which is assumed to be a list, returning the
old state value as the primary value and the new list as the state value.</p><p>So, when we do <code>(run-with-state result (list 32))</code>, we're passing <code>(list 32)</code> as
the initial state value, and then the <code>>>=</code> form passes that and <code>33</code> to
<code>state-push</code>. What <code>%state-monad</code> allows us to do is thread together some
procedures that require some kind of state, while essentially pretending the
state value is stored globally, like you might do in, say, C, and then retrieve
both the final state and the result at the end!</p><p>If you're a bit confused, don't worry. We'll write some of our own
<code>%state-monad</code>-based monadic procedures and hopefully all will become clear.
Consider, for instance, the
<a href="https://en.wikipedia.org/wiki/Fibonacci_number">Fibonacci sequence</a>, in which
each value is computed by adding the previous two. We could use the
<code>%state-monad</code> to compute Fibonacci numbers by storing the previous number as
the primary value and the number before that as the state value:</p><pre><code class="language-scheme">(define (fibonacci-thing value)
(lambda (state)
(values (+ value state)
value)))</code></pre><p>Now we can feed our Fibonacci-generating procedure the first value using
<code>run-with-state</code> and the second using <code>return</code>:</p><pre><code class="language-scheme">(run-with-state
(mlet* %state-monad ((starting (return 1))
(n1 (fibonacci-thing starting))
(n2 (fibonacci-thing n1)))
(fibonacci-thing n2))
0)
⇒ 3
⇒ 2
(run-with-state
(mlet* %state-monad ((starting (return 1))
(n1 (fibonacci-thing starting))
(n2 (fibonacci-thing n1))
(n3 (fibonacci-thing n2))
(n4 (fibonacci-thing n3))
(n5 (fibonacci-thing n4)))
(fibonacci-thing n5))
0)
⇒ 13
⇒ 8</code></pre><p>This is all very nifty, and possibly useful in general, but what does this have
to do with Guix? Well, many Guix store-based operations are meant to be used
in concert with yet another monad, called the <code>%store-monad</code>. But if we look at
<code>(guix store)</code>, where <code>%store-monad</code> is defined...</p><pre><code class="language-scheme">(define-alias %store-monad %state-monad)
(define-alias store-return state-return)
(define-alias store-bind state-bind)</code></pre><p>It was all a shallow façade! All the "store monad" is is a special case of the
state monad, where a value representing the store is passed as the state value.</p><h1>Lies, Damned Lies, and Abstractions</h1><p>We mentioned that, technically, we didn't need monads for Guix. Indeed, many
(now deprecated) procedures take a store value as the argument, such as
<code>build-expression->derivation</code>. However, monads are far more elegant and
simplify store code by quite a bit.</p><p><code>build-expression->derivation</code>, being deprecated, should never of course be
used. For one thing, it uses the "quoted build expression" style, rather than
G-expressions (we'll discuss gexps another time). The best way to create a
derivation from some basic build code is to use the new-fangled
<code>gexp->derivation</code> procedure:</p><pre><code class="language-scheme">(use-modules (guix gexp)
(gnu packages irc))
(define symlink-irssi
(gexp->derivation "link-to-irssi"
#~(symlink #$(file-append irssi "/bin/irssi") #$output)))
⇒ #<procedure 7fddcc7b81e0 at guix/gexp.scm:1180:2 (state)></code></pre><p>You don't have to understand the <code>#~(...)</code> form yet, only everything surrounding
it. We can see that this <code>gexp->derivation</code> returns a procedure taking the
initial state (store), just like our <code>%state-monad</code> procedures did, and like we
used <code>run-with-state</code> to pass the initial state to a <code>%state-monad</code> monadic
value, we use our old friend <code>run-with-store</code> when we have a <code>%store-monad</code>
monadic value!</p><pre><code class="language-scheme">(define symlink-irssi-drv
(with-store store
(run-with-store store
symlink-irssi)))
⇒ #<derivation /gnu/store/q7kwwl4z6psifnv4di1p1kpvlx06fmyq-link-to-irssi.drv => /gnu/store/6a94niigx4ii0ldjdy33wx9anhifr25x-link-to-irssi 7fddb7ef52d0></code></pre><p>Let's just check this derivation is as expected by reading the code from the
builder script.</p><pre><code class="language-scheme">(define symlink-irssi-builder
(list-ref (derivation-builder-arguments symlink-irssi-drv) 1))
(call-with-input-file symlink-irssi-builder
(lambda (port)
(read port)))
⇒ (symlink
"/gnu/store/hrlmypx1lrdjlxpkqy88bfrzg5p0bn6d-irssi-1.4.3/bin/irssi"
((@ (guile) getenv) "out"))</code></pre><p>And indeed, it symlinks the <code>irssi</code> binary to the output path. Some other,
higher-level, monadic procedures include <code>interned-file</code>, which copies a file
from outside the store into it, and <code>text-file</code>, which copies some text into it.
Generally, these procedures aren't used, as there are higher-level procedures
that perform similar functions (which we will discuss later), but for the sake
of this blog post, here's an example:</p><pre><code class="language-scheme">(with-store store
(run-with-store store
(text-file "unmatched-paren"
"( <paren@disroot.org>")))
⇒ "/gnu/store/v6smacxvdk4yvaa3s3wmd54lixn1dp3y-unmatched-paren"</code></pre><h1>Conclusion</h1><p>What have we learned about monads? The key points we can take away are:</p><ol><li>Monads are a way of composing together procedures and values that are wrapped
in containers that give them extra context, like <code>maybe</code> values.</li><li>Guix provides a high-level monad library that compensates for Guile's lack of
static typing or an interface-like system.</li><li>The <code>(guix monads)</code> module provides the state monad, which allows you to
thread state through procedures, allowing you to essentially pretend it's a
global variable that's modified by each procedure.</li><li>Guix uses the store monad frequently to thread a store connection through
procedures that need it.</li><li>The store monad is really just the state monad in disguise, where the state
value is used to thread the store object through monadic procedures.</li></ol><p>If you've read this post in its entirety but still don't yet quite get it, don't
worry. Try to modify and tinker about with the examples, and ask any questions
on the IRC channel <code>#guix:libera.chat</code> and mailing list at <code>help-guix@gnu.org</code>,
and hopefully it will all click eventually!</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package manager and
an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects user
freedom</a>.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, AArch64 and POWER9 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2023/dissecting-guix-part-1-derivations//Dissecting Guix, Part 1: Derivations(2023-01-04T12:30:00Z2023-01-04T12:30:00Z To a new user, Guix's functional architecture can seem quite alien, and possibly
offputting. With a combination of extensive #guix -querying, determined
manual-reading, and plenty of source-perusing, they may eventually figure out
how everything fits together by themselves, but this can be frustrating and
often takes a fairly long time. However, once you peel back the layers, the "Nix way" is actually rather
elegant, if perhaps not as simple as the mutable, imperative style implemented
by the likes of dpkg and
pacman .
This series of blog posts will cover basic Guix concepts, taking a "ground-up"
approach…<p>To a new user, Guix's functional architecture can seem quite alien, and possibly
offputting. With a combination of extensive <code>#guix</code>-querying, determined
manual-reading, and plenty of source-perusing, they may eventually figure out
how everything fits together by themselves, but this can be frustrating and
often takes a fairly long time.</p><p>However, once you peel back the layers, the "Nix way" is actually rather
elegant, if perhaps not as simple as the mutable, imperative style implemented
by the likes of <a href="https://wiki.debian.org/dpkg"><code>dpkg</code></a> and
<a href="https://wiki.archlinux.org/title/pacman"><code>pacman</code></a>.
This series of blog posts will cover basic Guix concepts, taking a "ground-up"
approach by dealing with lower-level concepts first, and hopefully make those
months of information-gathering unnecessary.</p><p>Before we dig in to Guix-specific concepts, we'll need to learn about one
inherited from <a href="https://nixos.org">Nix</a>, the original functional package manager
and the inspiration for Guix; the idea of a
<a href="https://guix.gnu.org/manual/devel/en/html_node/Derivations.html"><em>derivation</em></a>
and its corresponding <em>store items</em>.</p><p>These concepts were originally described by Eelco Dolstra, the original author
of Nix, in their <a href="https://edolstra.github.io/pubs/phd-thesis.pdf">PhD thesis</a>;
see <em>§ 2.1 The Nix store</em> and <em>§ 2.4 Store Derivations</em>.</p><h1>Store Items</h1><p>As you probably know, everything that Guix builds is stored in the <em>store</em>,
which is almost always the <code>/gnu/store</code> directory. It's the job of the
<a href="https://guix.gnu.org/manual/en/html_node/Invoking-guix_002ddaemon.html"><code>guix-daemon</code></a>
to manage the store and build things. If you run
<a href="https://guix.gnu.org/manual/en/html_node/Invoking-guix-build.html"><code>guix build PKG</code></a>,
<code>PKG</code> will be built or downloaded from a substitute server if available, and a
path to an item in the store will be displayed.</p><pre><code>$ guix build irssi
/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3</code></pre><p>This item contains the final result of building <a href="https://irssi.org"><code>irssi</code></a>.
Let's peek inside:</p><pre><code>$ ls $(guix build irssi)
bin/ etc/ include/ lib/ share/
$ ls $(guix build irssi)/bin
irssi*</code></pre><p><code>irssi</code> is quite a simple package. What about a more complex one, like
<a href="https://docs.gtk.org/glib"><code>glib</code></a>?</p><pre><code>$ guix build glib
/gnu/store/bx8qq76idlmjrlqf1faslsq6zjc6f426-glib-2.73.3-bin
/gnu/store/j65bhqwr7qq7l77nj0ahmk1f1ilnjr3a-glib-2.73.3-debug
/gnu/store/3pn4ll6qakgfvfpc4mw89qrrbsgj3jf3-glib-2.73.3-doc
/gnu/store/dvsk6x7d26nmwsqhnzws4iirb6dhhr1d-glib-2.73.3
/gnu/store/4c8ycz501n2d0xdi4blahvnbjhd5hpa8-glib-2.73.3-static</code></pre><p><code>glib</code> produces five <code>/gnu/store</code> items, because it's possible for a package to
produce multiple <a href="https://guix.gnu.org/manual/en/html_node/Packages-with-Multiple-Outputs.html">outputs</a>.
Each output can be referred to separately, by suffixing a package's name with
<code>:OUTPUT</code> where supported. For example, this
<a href="https://guix.gnu.org/manual/en/html_node/Invoking-guix-package.html"><code>guix install</code></a>
invocation will add <code>glib</code>'s <code>bin</code> output to your profile:</p><pre><code>$ guix install glib:bin</code></pre><p>The default output is <code>out</code>, so when you pass <code>glib</code> by itself to that command,
it will actually install <code>glib:out</code> to the profile.</p><p><code>guix build</code> also provides the <code>--source</code> flag, which produces the store item
corresponding to the given package's downloaded source code.</p><pre><code>$ guix build --source irssi
/gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz
$ guix build --source glib
/gnu/store/d22wzjq3xm3q8hwnhbgk2xd3ph7lb6ay-glib-2.73.3.tar.xz</code></pre><p>But how does Guix know how to build these store outputs in the first place?
That's where derivations come in.</p><h1><code>.drv</code> Files</h1><p>You've probably seen these being printed by the Guix program now and again.
Derivations, represented in the daemon's eyes by <code>.drv</code> files, contain
instructions for building store items. We can retrieve the paths of these
<code>.drv</code> files with the <code>guix build --derivations</code> command:</p><pre><code>$ guix build --derivations irssi
/gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv</code></pre><p><code>guix build</code> can actually also accept derivation paths as an argument, in lieu
of a package, like so:</p><pre><code>$ guix build /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv
/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3</code></pre><p>Let's look inside this derivation file.</p><pre><code>Derive([("out","/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3","","")],[("/gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv",["out"]),("/gnu/store/baqpbl4wck7nkxrbyc9nlhma7kq5dyfl-guile-2.0.14.drv",["out"]),("/gnu/store/bfirgq65ndhf63nn4q6vlkbha9zd931q-openssl-1.1.1l.drv",["out"]),("/gnu/store/gjwpqzvfhz13shix6a6cs2hjc18pj7wy-module-import-compiled.drv",["out"]),("/gnu/store/ij8651x4yh53hhcn6qw2644nhh2s8kcn-glib-2.70.2.drv",["out"]),("/gnu/store/jg2vv6yc2yqzi3qzs82dxvqmi5k21lhy-irssi-1.4.3.drv",["out"]),("/gnu/store/qggpjl9g6ic3cq09qrwkm0dfsdjf7pyr-glibc-utf8-locales-2.33.drv",["out"]),("/gnu/store/zafabw13yyhz93jwrcz7axak1kn1f2cx-openssl-1.1.1s.drv",["out"])],["/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import","/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"],"x86_64-linux","/gnu/store/hnr4r2d0h0xarx52i6jq9gvsrlc3q81a-guile-2.0.14/bin/guile",["--no-auto-compile","-L","/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import","-C","/gnu/store/6rkkvvb7pl1l9ng8vvywvwf357vhm3va-module-import-compiled","/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"],[("allowSubstitutes","0"),("guix properties","((type . graft) (graft (count . 2)))"),("out","/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3"),("preferLocalBuild","1")])</code></pre><p>It's... not exactly human-readable. We could try to format it and break it
down, but it'd still be pretty hard to understand, since <code>.drv</code> files contain no
labels for the fields or any other human-readable indicators. Instead, we're
going to explore derivations in a Guile REPL.</p><h1>Exploring Guix Interactively</h1><p>Before we continue, we'll want to start a REPL, so that we can try out the Guix
Guile API interactively. To run a REPL in the terminal, simply
<a href="https://guix.gnu.org/manual/devel/en/html_node/Using-Guix-Interactively.html">call <code>guix repl</code></a>.</p><p>If you're using Emacs, you can instead install
<a href="https://nongnu.org/geiser">Geiser</a>, which provides a comfortable Emacs UI for
various Lisp REPLs, invoke <code>guix repl --listen=tcp:37146 &</code>, and type
<code>M-x geiser-connect RET RET RET</code> to connect to the running Guile instance.</p><p>Your <code>.guile</code> file may contain code for enabling colours and readline bindings
that Geiser will choke on. The default Guix System <code>.guile</code> contains code to
suppress these features when <code>INSIDE_EMACS</code> is set, so you'll need to run
<code>guix repl</code> like this:</p><pre><code class="language-sh">INSIDE_EMACS=1 guix repl --listen=tcp:37146 &</code></pre><p>There are a few Guix modules we'll need. Run this Scheme code to import them:</p><pre><code class="language-scheme">(use-modules (guix)
(guix derivations)
(guix gexp)
(guix packages)
(guix store)
(gnu packages glib)
(gnu packages irc))</code></pre><p>We now have access to the store, G-expression, package, and derivation APIs,
along with the <code>irssi</code> and <code>glib</code> <code><package></code> objects.</p><h1>Creating a <code><derivation></code></h1><p>The Guix API for derivations revolves around the <code><derivation></code> record, which is
the Scheme representation of that whole block of text surrounded by
<code>Derive(...)</code>. If we look in <code>guix/derivations.scm</code>, we can see that it's
defined like this:</p><pre><code class="language-scheme">(define-immutable-record-type <derivation>
(make-derivation outputs inputs sources system builder args env-vars
file-name)
derivation?
(outputs derivation-outputs) ; list of name/<derivation-output> pairs
(inputs derivation-inputs) ; list of <derivation-input>
(sources derivation-sources) ; list of store paths
(system derivation-system) ; string
(builder derivation-builder) ; store path
(args derivation-builder-arguments) ; list of strings
(env-vars derivation-builder-environment-vars) ; list of name/value pairs
(file-name derivation-file-name)) ; the .drv file name</code></pre><p>With the exception of <code>file-name</code>, each of those fields corresponds to a field
in the <code>Derive(...)</code> form. Before we can examine them, though, we need to
figure out how to <em>lower</em> that <code>irssi</code> <code><package></code> object into a derivation.</p><p><code>guix repl</code> provides the <code>,lower</code> command to create derivations quickly,
as shown in this sample REPL session:</p><pre><code>scheme@(guile-user)> ,use (guix)
scheme@(guile-user)> ,use (gnu packages irc)
scheme@(guile-user)> irssi
$1 = #<package irssi@1.4.3 gnu/packages/irc.scm:153 7f3ff98e0c60>
scheme@(guile-user)> ,lower irssi
$2 = #<derivation /gnu/store/drjfddvlblpr635jazrg9kn5azd9hsbj-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7f3ff7782d70>
;; Below we use the $N variable automatically bound by the REPL.
scheme@(guile-user)> (derivation-system $2)
$3 = "x86_64-linux"</code></pre><p>Since <code>,lower</code> is a REPL command, however, we can't use it in proper Scheme
code. It's quite useful for exploring specific derivations interactively, but
since the purpose of this blog post is to explain how things work inside, we're
going to use the pure-Scheme approach here.</p><p>The procedure we need to use to turn a high-level object like <code><package></code> into a
derivation is called <code>lower-object</code>; more on that in a future post. However,
this doesn't initially produce a derivation:</p><pre><code class="language-scheme">(pk (lower-object irssi))
;;; (#<procedure 7fe17c7af540 at guix/store.scm:1994:2 (state)>)</code></pre><p><code>pk</code> is an abbreviation for the procedure <code>peek</code>, which takes the given object,
writes a representation of it to the output, and returns it. It's especially
handy when you want to view an intermediate value in a complex expression.</p><p>The returned object is a monadic value (more on those in the next post on
monads) that needs to be evaluated in the context of a store connection. We do
this by first using <code>with-store</code> to connect to the store and bind the connection
to a name, then wrapping the <code>lower-object</code> call with <code>run-with-store</code>:</p><pre><code class="language-scheme">(define irssi-drv
(pk (with-store %store
(run-with-store %store
(lower-object irssi)))))
;;; (#<derivation /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7fe1902b6140>)
(define glib-drv
(pk (with-store %store
(run-with-store %store
(lower-object glib)))))
;;; (#<derivation /gnu/store/81qqs7xah2ln39znrji4r6xj85zi15bi-glib-2.70.2.drv => /gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin /gnu/store/22mkp8cr6rxg6w8br9q8dbymf51b44m8-glib-2.70.2-debug /gnu/store/a6qb5arvir4vm1zlkp4chnl7d8qzzd7x-glib-2.70.2 /gnu/store/y4ak268dcdwkc6lmqfk9g1dgk2jr9i34-glib-2.70.2-static 7fe17ca13b90>)</code></pre><p>And we have liftoff! Now we've got two <code><derivation></code> records to play with.</p><h1>Exploring <code><derivation></code></h1><h2><code><derivation-output></code></h2><p>The first "argument" in the <code>.drv</code> file is <code>outputs</code>, which tells the Guix
daemon about the outputs that this build can produce:</p><pre><code class="language-scheme">(define irssi-outputs
(pk (derivation-outputs irssi-drv)))
;;; ((("out" . #<<derivation-output> path: "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3" hash-algo: #f hash: #f recursive?: #f>)))
(pk (assoc-ref irssi-outputs "out"))
(define glib-outputs
(pk (derivation-outputs glib-drv)))
;;; ((("bin" . #<<derivation-output> path: "/gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin" hash-algo: #f hash: #f recursive?: #f>) ("debug" . #<<derivation-output> path: "/gnu/store/22mkp8cr6rxg6w8br9q8dbymf51b44m8-glib-2.70.2-debug" hash-algo: #f hash: #f recursive?: #f>) ("out" . #<<derivation-output> path: "/gnu/store/a6qb5arvir4vm1zlkp4chnl7d8qzzd7x-glib-2.70.2" hash-algo: #f hash: #f recursive?: #f>) ("static" . #<<derivation-output> path: "/gnu/store/y4ak268dcdwkc6lmqfk9g1dgk2jr9i34-glib-2.70.2-static" hash-algo: #f hash: #f recursive?: #f>)))
(pk (assoc-ref glib-outputs "bin"))
;;; (#<<derivation-output> path: "/gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin" hash-algo: #f hash: #f recursive?: #f>)</code></pre><p>It's a simple association list mapping output names to <code><derivation-output></code>
records, and it's equivalent to the first "argument" in the <code>.drv</code> file:</p><pre><code>[ ("out", "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3", "", "")
]</code></pre><p>The <code>hash-algo</code> and <code>hash</code> fields are for storing the content hash and the
algorithm used with that hash for what we term a <em>fixed-output derivation</em>,
which is essentially a derivation where we know what the hash of the content
will be in advance. For instance, <code>origin</code>s produce fixed-output derivations:</p><pre><code class="language-scheme">(define irssi-src-drv
(pk (with-store %store
(run-with-store %store
(lower-object (package-source irssi))))))
;;; (#<derivation /gnu/store/mcz3vzq7lwwaqjb8dy7cd69lvmi6d241-irssi-1.4.3.tar.xz.drv => /gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz 7fe17b3c8d70>)
(define irssi-src-outputs
(pk (derivation-outputs irssi-src-drv)))
;;; ((("out" . #<<derivation-output> path: "/gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz" hash-algo: sha256 hash: #vu8(185 63 113 82 35 163 34 230 127 66 182 26 8 165 18 174 41 227 75 212 165 61 127 34 55 102 102 10 170 90 4 52) recursive?: #f>)))
(pk (assoc-ref irssi-src-outputs "out"))
;;; (#<<derivation-output> path: "/gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz" hash-algo: sha256 hash: #vu8(185 63 113 82 35 163 34 230 127 66 182 26 8 165 18 174 41 227 75 212 165 61 127 34 55 102 102 10 170 90 4 52) recursive?: #f>)</code></pre><p>Note how the <code>hash</code> and <code>hash-algo</code> now have values.</p><p>Perceptive readers may note that the <code><derivation-output></code> has four fields,
whereas the tuple in the <code>.drv</code> file only has three (minus the label). The
serialisation of <code>recursive?</code> is done by adding the prefix <code>r:</code> to the
<code>hash-algo</code> field, though its actual purpose is difficult to explain, and is out
of scope for this post.</p><h2><code><derivation-input></code></h2><p>The next field is <code>inputs</code>, which corresponds to the second field in the <code>.drv</code>
file format:</p><pre><code>[ ("/gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv", ["out"]),
("/gnu/store/baqpbl4wck7nkxrbyc9nlhma7kq5dyfl-guile-2.0.14.drv", ["out"]),
("/gnu/store/bfirgq65ndhf63nn4q6vlkbha9zd931q-openssl-1.1.1l.drv", ["out"]),
("/gnu/store/gjwpqzvfhz13shix6a6cs2hjc18pj7wy-module-import-compiled.drv", ["out"]),
("/gnu/store/ij8651x4yh53hhcn6qw2644nhh2s8kcn-glib-2.70.2.drv", ["out"]),
("/gnu/store/jg2vv6yc2yqzi3qzs82dxvqmi5k21lhy-irssi-1.4.3.drv", ["out"]),
("/gnu/store/qggpjl9g6ic3cq09qrwkm0dfsdjf7pyr-glibc-utf8-locales-2.33.drv", ["out"]),
("/gnu/store/zafabw13yyhz93jwrcz7axak1kn1f2cx-openssl-1.1.1s.drv", ["out"])
]</code></pre><p>Here, each tuple specifies a derivation that needs to be built before this
derivation can be built, and the outputs of the derivation that the build
process of this derivation uses. Let's grab us the Scheme equivalent:</p><pre><code class="language-scheme">(define irssi-inputs
(pk (derivation-inputs irssi-drv)))
;;; [a fairly large amount of output]
(pk (car irssi-inputs))
;;; (#<<derivation-input> drv: #<derivation /gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv => /gnu/store/2jj2mxn6wfrcw7i85nywk71mmqbnhzps-glib-2.70.2 7fe1902b6640> sub-derivations: ("out")>)</code></pre><p>Unlike <code>derivation-outputs</code>, <code>derivation-inputs</code> maps 1:1 to the <code>.drv</code>
form; the <code>drv</code> field is a <code><derivation></code> to be built, and the
<code>sub-derivations</code> field is a list of outputs.</p><h2>Builder Configuration</h2><p>The other fields are simpler; none of them involve new records. The third is
<code>derivation-sources</code>, which contains a list of all store items used in the build
which aren't themselves built using derivations, whereas <code>derivation-inputs</code>
contains the dependencies which are.</p><p>This list usually just contains the path to the Guile <em>build script</em> that
realises the store items when run, which we'll examine in a later post, and
the path to a directory containing extra modules to add to the build script's
<code>%load-path</code>, called <code>/gnu/store/...-module-import</code>.</p><p>The next field is <code>derivation-system</code>, which specifies the system type (such as
<code>x86_64-linux</code>) we're building for. Then we have <code>derivation-builder</code>, pointing
to the <code>guile</code> executable that runs the build script; and the second-to-last is
<code>derivation-builder-arguments</code>, which is a list of arguments to pass to
<code>derivation-builder</code>. Note how we use <code>-L</code> and <code>-C</code> to extend the Guile
<code>%load-path</code> and <code>%load-compiled-path</code> to include the <code>module-import</code> and
<code>module-import-compiled</code> directories:</p><pre><code class="language-scheme">(pk (derivation-system irssi-drv))
;;; ("x86_64-linux")
(pk (derivation-builder irrsi-drv))
;;; ("/gnu/store/hnr4r2d0h0xarx52i6jq9gvsrlc3q81a-guile-2.0.14/bin/guile")
(pk (derivation-builder-arguments irrsi-drv))
;;; (("--no-auto-compile" "-L" "/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import" "-C" "/gnu/store/6rkkvvb7pl1l9ng8vvywvwf357vhm3va-module-import-compiled" "/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"))</code></pre><p>The final field contains a list of environment variables to set before we start
the build process:</p><pre><code class="language-scheme">(pk (derivation-builder-environment-vars irssi-drv))
;;; ((("allowSubstitutes" . "0") ("guix properties" . "((type . graft) (graft (count . 2)))") ("out" . "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3") ("preferLocalBuild" . "1")))</code></pre><p>The last record field, <code>derivation-file-name</code> contains the path to the <code>.drv</code>
file, and so isn't represented in a serialised derivation.</p><h1>Utilising <code><derivation></code></h1><p>Speaking of serialisation, to convert between the <code>.drv</code> text format and the
Scheme <code><derivation></code> record, you can use <code>write-derivation</code>, <code>read-derivation</code>,
and <code>read-derivation-from-file</code>:</p><pre><code class="language-scheme">(define manual-drv
(with-store %store
(derivation %store "manual"
"/bin/sh" '())))
(write-derivation manual-drv (current-output-port))
;;; -| Derive([("out","/gnu/store/kh7fais2zab22fd8ar0ywa4767y6xyak-example","","")],[],[],"x86_64-linux","/bin/sh",[],[("out","/gnu/store/kh7fais2zab22fd8ar0ywa4767y6xyak-example")])
(pk (read-derivation-from-file (derivation-file-name irssi-drv)))
;;; (#<derivation /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7fb3798788c0>)
(call-with-input-file (derivation-file-name irssi-drv)
read-derivation)
;;; (#<derivation /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7fb37ad19e10>)</code></pre><p>You can realise <code><derivation></code>s as store items using the <code>build-derivations</code>
procedure:</p><pre><code class="language-scheme">(use-modules (ice-9 ftw))
(define irssi-drv-out
(pk (derivation-output-path
(assoc-ref (derivation-outputs irssi-drv) "out"))))
;;; ("/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3")
(pk (scandir irssi-drv-out))
;;; (#f)
(pk (with-store %store (build-derivations %store (list irssi-drv))))
;;; (#t)
(pk (scandir irssi-drv-out))
;;; (("." ".." "bin" "etc" "include" "lib" "share"))</code></pre><h1>Conclusion</h1><p>Derivations are one of Guix's most important concepts, but are fairly easy to
understand once you get past the obtuse <code>.drv</code> file format. They provide the
Guix daemon with the initial instructions that it uses to build store items
like packages, origins, and other file-likes such as <code>computed-file</code> and
<code>local-file</code>, which will be discussed in a future post!</p><p>To recap, a derivation contains the following fields:</p><ol><li><code>derivation-outputs</code>, describing the various output paths that the derivation
builds</li><li><code>derivation-inputs</code>, describing the other derivations that need to be built
before this one is</li><li><code>derivation-sources</code>, listing the non-derivation store items that the
derivation depends on</li><li><code>derivation-system</code>, specifying the system type a derivation will be compiled
for</li><li><code>derivation-builder</code>, the executable to run the build script with</li><li><code>derivation-builder-arguments</code>, arguments to pass to the builder</li><li><code>derivation-builder-environment-vars</code>, variables to set in the builder's
environment</li></ol><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package manager and
an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects user
freedom</a>.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, AArch64 and POWER9 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2022/keeping-ones-home-tidy//Keeping one’s home tidyLudovic Courtès2022-03-21T15:30:00Z2022-03-21T15:30:00Z How much effort to recreate your work environment when you switch to a
new machine? What would it take to roll back to your previous
environment once you’ve noticed a program no longer behaves as expected?
What about sharing your environment with friends of yours? These are
some of the things that Guix
Home ,
which landed in Guix as a “technology preview” in September 2021, aims
to make effortless, reliable, and fun. In a nutshell, Guix Home brings the fully declarative configuration of
Guix
System
to home directories. With Guix System, users and administrators provide
a configuration file that defines the…<p>How much effort to recreate your work environment when you switch to a
new machine? What would it take to roll back to your previous
environment once you’ve noticed a program no longer behaves as expected?
What about sharing your environment with friends of yours? These are
some of the things that <a href="https://guix.gnu.org/manual/devel/en/html_node/Home-Configuration.html">Guix
Home</a>,
which landed in Guix as a “technology preview” in September 2021, aims
to make effortless, reliable, and fun.</p><p>In a nutshell, Guix Home brings the <a href="https://guix.gnu.org/manual/devel/en/html_node/Using-the-Configuration-System.html">fully declarative configuration of
Guix
System</a>
to home directories. With Guix System, users and administrators provide
a configuration file that defines the operating system configuration;
with Guix Home, users provide a configuration file that defines the
configuration of their work environment in their home directory—their
<em>home environment</em>. That configuration is meant to <em>stand alone</em>, to
describe all the relevant aspects of your work environment. But what
exactly goes in a home environment?</p><h1>“Dot files” don’t live in a vacuum</h1><p>Among seasoned Unix-style users, we often equate “home environment” with
“dot files”—configuration files in our home directory, from <code>~/.bashrc</code>
and <code>~/.ssh/config</code> to <code>~/.emacs</code> and everything under <code>~/.config</code>.
These files are precious and many store them under version control, to
keep track of changes made to their configuration. That’s a good idea,
but is that all it takes to describe the home environment? To roll
back to a previous version?</p><p>Of course not. Dot files don’t exist in a vacuum; at the very least,
your home environment is not just a set of dot files, but also a set of
installed packages. They work together: if you upgrade a package, the
corresponding dot file might need to be adjusted; if a package is
missing, its dot file is not of any use. Sometimes a home environment
contains additional things: daemons (programs that run in the
background), or periodically executed jobs.</p><p>Guix Home goes beyond dot files: it lets you declare and
instantiate all these aspects that make up your home environment.</p><h1>Genesis</h1><p>Guix Home was initially developed by Andrew Tropin as part of the <a href="https://git.sr.ht/~abcdw/rde">rde
project</a>; it was integrated in Guix proper
six months ago. I am writing this as an adopter and contributor, but
there were a number of earlier adopters and earlier contributors. In
fact, despite being still very much under development, the tool has
already attracted a number of excited users eager to find a way to keep
their home tidy!</p><p>The idea of writing down a declaration of your home environment that you
can reproduce anytime is a natural followup to everything Guix does—you
could already declare a package set in a
<a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-package.html#index-profile-manifest">manifest</a>
or even a <a href="https://guix.gnu.org/manual/devel/en/html_node/Using-the-Configuration-System.html">complete operating
system</a>.
It had been floating around, in Nix land with <a href="https://github.com/nix-community/home-manager">Home
Manager</a> and in Guix land
with the now-defunct <a href="https://framagit.org/tyreunom/guix-home-manager/">Guix Home
Manager</a> by Julien
Lepiller. The latter was similar to today’s Guix Home, but went one
step further by making your home directory read-only—yes, <em>read-only</em>!
The main advantage is that it would ensure statelessness—you’d be sure
that absolutely <em>all</em> your home configuration is under Guix Home
Manager’s control; sub-directories containing mutable data would have to
be explicitly declared. The downside is that it raised the barrier to
entry: you’d have to either switch entirely, or not use it at all. Guix
Home takes a more pragmatic approach and happily coexists with
configuration managed “the old way”.</p><h1>Getting started</h1><p>To get started, you need a Home configuration file. There’s
<a href="https://guix.gnu.org/manual/devel/en/html_node/Declaring-the-Home-Environment.html">documentation</a>,
but as always, starting from a blank page is a bit intimidating. So
instead of starting from a blank page, you can let <code>guix home import</code>
generate an initial config for you:</p><pre><code>guix home import ~/src/guix-config</code></pre><p>This will create the <code>~/src/guix-config</code> directory and populate it with a
bunch of files among which <code>home-configuration.scm</code> along these lines:</p><pre><code class="language-scheme">(use-modules (gnu home)
(gnu packages)
(gnu services)
(guix gexp)
(gnu home services shells))
(home-environment
(packages
(map (compose list specification->package+output)
(list "emacs-geiser-guile"
"emacs-geiser"
"pinentry-emacs"
"emacs-exwm"
"gnome-maps"
"pipe-viewer"
"emacs"
"pavucontrol"
"git"
"xterm"
"qemu"
"openssh")))
(services
(list (service home-bash-service-type
(home-bash-configuration
(aliases
'(("grep" . "grep --color=auto")
("ll" . "ls -l")
("ls" . "ls -p --color=auto")
("qemu" . "qemu-system-x86_64 -enable-kvm -m 512")
("rm" . "rm --one-file-system")))
(bashrc
(list (local-file "/home/charlie/src/guix-config/.bashrc"
"bashrc")))
(bash-profile
(list (local-file
"/home/charlie/src/guix-config/.bash_profile"
"bash_profile"))))))))</code></pre><p><code>guix home import</code> automatically added the packages of <code>~/.guix-profile</code>
to the <code>packages</code> field. Because I’m using
<a href="https://www.gnu.org/software/bash">Bash</a>, it also added an instance of
<a href="https://guix.gnu.org/manual/devel/en/html_node/Shells-Home-Services.html"><code>home-bash-service-type</code></a>
with aliases extracted from my <code>~/.bashrc</code>; it also made copies of
<code>~/.bashrc</code> and <code>~/.bash_profile</code> and refers to them.</p><p>Now that I have an initial configuration, I can first test it in an
<em>isolated container</em>:</p><pre><code>guix home container ~/src/guix-config/home-configuration.scm</code></pre><p>This command gives an interactive shell in a container where my home
environment, as declared in <code>home-configuration.scm</code>, is deployed.
There I can see my home directory as it would look like if I deploy my
home environment “for real”: I can see my <code>~/.bashrc</code> and co., I can
check that all the packages declared are in <code>$PATH</code> and visible in
<code>~/.guix-home</code>, and so on. And all this is safe: my actual home
directory has been left unchanged!</p><p>Once satisfied with my configuration, I can instantiate it:</p><pre><code>guix home reconfigure ~/src/guix-config/home-configuration.scm</code></pre><p>At that point, my actual home directory corresponds to that
configuration. Some of my dot files are now provided by Guix Home, and
thus they’re symbolic links (“symlinks”) to their read-only copy in
<code>/gnu/store</code>:</p><pre><code>$ ls -l ~/.bashrc ~/.bash_profile
lrwxrwxrwx 1 charlie users 56 Mar 7 15:46 /home/charlie/.bash_profile -> /gnu/store/lpdydssyyxx9n0xvp2jmv7yqgyr2pcg3-bash_profile
lrwxrwxrwx 1 charlie users 50 Mar 7 15:46 /home/charlie/.bashrc -> /gnu/store/kxc0j4i05sib04vf92nr8xxkb8isdfn7-bashrc</code></pre><p>But don’t worry: before creating those symlinks, <code>guix home reconfigure</code>
created backups of existing files under
<code>~/TIMESTAMP-guix-home-legacy-configs-backup</code>, where <code>TIMESTAMP</code> is a
Unix-style timestamp.</p><p>And voilà, I have my first Guix Home generation!</p><pre><code>$ guix home describe
Generation 1 Mar 07 2022 15:46:20 (current)
file name: /var/guix/profiles/per-user/charlie/guix-home-1-link
canonical file name: /gnu/store/qr1c5jpfrj815ncv6yr2lfdgs8nq8kkn-home
channels:
guix:
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: 3ac1366648f933f7244c2d0b9926f7ba5d92a113
configuration file: /gnu/store/xfgasfms9rhhigyj7i8za77zpqx6zbhn-configuration.scm</code></pre><p><code>guix home describe</code> shows provenance tracking we know and love from
Guix System: all the info we need to redeploy the same home environment
elsewhere, or at a different point in time. It’s also information <code>guix home reconfigure</code> relies on to make sure you never accidentally
<em>downgrade</em> you home environment to an older Guix revision.</p><h1>Going further</h1><p>Alright, at this point, you might be thinking that it’s a lot of fuss
but the “only” benefit over dot files under version control is that
<code>guix home</code> also takes care of installing packages. Guix Home really
shines once you use higher-level services, and when you start composing
services together.</p><p>To the example above, in the <code>services</code> field, we can add a service
declaration that runs
<a href="https://guix.gnu.org/manual/devel/en/html_node/Desktop-Home-Services.html#index-home_002dredshift_002dservice_002dtype">Redshift</a>,
a program that adjusts the display color temperature according to the
time of day:</p><pre><code class="language-scheme">(service home-redshift-service-type
(home-redshift-configuration
(location-provider 'manual)
(latitude 35.81) ;northern hemisphere
(longitude -0.80))) ;west of Greenwich</code></pre><p>The effect is that, as soon as we log in, under Xorg, Redshift will be
started in the background as a <a href="https://guix.gnu.org/manual/devel/en/html_node/Shepherd-Home-Service.html">Shepherd
service</a>.
The Home-generated <code>~/.profile</code> takes care of spawning <code>shepherd</code>, which
in turn spawns the <code>redshift</code> service:</p><pre><code>$ herd status
Started:
+ root
+ redshift</code></pre><p>We gained another thing here: a consistent, unified configuration
language. Instead of learning Redshift’s configuration file format, we
define a <code>home-redshift-configuration</code> record, right in Scheme. Under
the hood, that configuration is converted into Redshift’s file format;
any error is caught at configuration time, when running <code>guix home reconfigure</code>, and we can be sure that Redshift is passed a valid
configuration file.</p><p>We can similarly define a <a href="https://guix.gnu.org/manual/devel/en/html_node/Mcron-Home-Service.html#index-home_002dmcron_002dservice_002dtype">periodic mcron
job</a>,
for example one that updates a
<a href="https://www.gnu.org/software/idutils">GNU Idutils</a> search database
(that’s a pretty convenient and speedy way to look for code or
documents!):</p><pre><code class="language-scheme">(simple-service 'idutils home-mcron-service-type
;; Every day at 12:15 and 19:15.
(list #~(job '(next-minute-from (next-hour '(12 19)) '(15))
(string-append #$idutils "/bin/mkid \
-o $HOME/.idutils/src.db $HOME/src"))))</code></pre><p>Again, <code>guix home</code> creates a Shepherd service that start mcron with a
configuration file containing definitions for periodic jobs, which we
can inspect <em>via</em> <code>herd</code>:</p><pre><code>$ herd schedule mcron | head -5
Sun Mar 20 19:15:00 2022 +0000
/gnu/store/2d026nan309qkci968k8gpa8fcv9q4mv-idutils-4.6/bin/mkid -o $HOME/.idutils/src $HOME/src
Mon Mar 21 12:15:00 2022 +0000
/gnu/store/2d026nan309qkci968k8gpa8fcv9q4mv-idutils-4.6/bin/mkid -o $HOME/.idutils/src $HOME/src</code></pre><h1>Services, composed</h1><p>If you already use Guix System, all the above certainly looks familiar:
Guix Home builds upon the <a href="https://guix.gnu.org/manual/devel/en/html_node/Defining-Services.html">service
framework</a>
that powers Guix System; Home services are defined in the <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/home/services"><code>(gnu home services …)</code> module
tree</a>.</p><p>That framework lets us define relations among
“services”, in a broad sense, and how services <em>extend</em> each other—in
the example above, <code>redshift</code> and <code>mcron</code> both extend <code>shepherd</code> by
giving it a daemon to take care of. We can see those relations at play
by running:</p><pre><code>guix home extension-graph home-configuration.scm</code></pre><p>… which, for the configuration described above, gives a graph that looks
like this:</p><p><img src="/static/blog/img/home-extension-graph.svg" alt="Extension graph for home services." /></p><p>We see <code>redshift</code>, <code>mcron</code>, and <code>shepherd</code>, but we also see lower-level
services that <code>guix home</code> instantiates for us, such as the <code>profile</code>
service which takes care of deploying packages listed in the <code>packages</code>
field under <code>~/.guix-home/profile</code>. Each arrow denotes a <em>service
extension</em>. You can <a href="https://guix.gnu.org/manual/devel/en/html_node/Service-Composition.html">read
more</a>
(and <a href="https://archive.fosdem.org/2017/schedule/event/composingsystemservicesinguixsd/">view
more!</a>)
about service composition. To satisfy our math and
functional-programming geek audience, we should mention that service
types and their extension operation form a
<a href="https://en.wikipedia.org/wiki/Monoid">monoid</a>.</p><h1>What’s next?</h1><p>Let’s be clear: Guix Home is pretty new and chances are that <code>guix home search</code>—the command to search for services by keyword—won’t give you the
service you’re looking for. There’s also a bunch of open questions
left, such as how to reuse services initially defined for Guix System in
cases where they could be equally useful in Guix
Home—<a href="https://guix.gnu.org/manual/devel/en/html_node/Networking-Services.html#index-syncthing_002dservice_002dtype">Syncthing</a>,
for example.</p><p>But while it’s still a “technology preview”, it’s already a tool that
tinkerers can play with and benefit from. Patches adding new services
have already been proposed; maybe your favorite service is next?
Consider
<a href="https://guix.gnu.org/manual/devel/en/html_node/Contributing.html">contributing</a>.</p><p>With a new release and ten-year anniversary coming up, we’re happy to
celebrate with a tool that extends the reach of declarative and
reproducible deployment!</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package manager and
an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects user
freedom</a>.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, AArch64 and POWER9 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2021/the-big-change//The Big ChangeLudovic Courtès2021-12-15T15:00:00Z2021-12-15T15:00:00Z Making cross-cutting changes over a large code base is difficult, but
it's occasionally necessary if we are to keep the code base tidy and
malleable. With almost 800K source lines of code, Guix can reasonably
be called a large code base. One might argue that almost 80% of this
code is package
definitions ,
which “doesn't count”. Reality is that it does count, not only
because those package definitions are regular Scheme code, but also
they are directly affected by the big change Guix has just undergone. This post looks at what’s probably the biggest change Guix…<p>Making cross-cutting changes over a large code base is difficult, but
it's occasionally necessary if we are to keep the code base tidy and
malleable. With almost 800K source lines of code, Guix can reasonably
be called a large code base. One might argue that almost 80% of this
code is <a href="https://guix.gnu.org/manual/devel/en/html_node/Defining-Packages.html">package
definitions</a>,
which “doesn't count”. Reality is that it <em>does</em> count, not only
because those package definitions <em>are</em> regular Scheme code, but also
they are directly affected by the big change Guix has just undergone.</p><p>This post looks at what’s probably the biggest change Guix has seen
since it started nine years ago and that anyone writing packages will
immediately notice: simplified <a href="https://guix.gnu.org/manual/en/html_node/package-Reference.html#index-package">package
inputs</a>.
Yes, we just changed how each of the 20K packages plus those in
third-party channels can declare their dependencies. Before describing
the change, how we implemented it, and how packagers can adapt, let’s
first take a look at the previous situation and earlier improvements
that made this big change possible.</p><h1>Packages and inputs</h1><p>Packages in Guix are
<a href="https://guix.gnu.org/manual/devel/en/html_node/Defining-Packages.html">defined</a>
using a <em>domain-specific language</em> embedded in the Scheme programming
language—an EDSL, for the programming language geeks among us. This is
a departure from other designs such as that of Nix, which comes with a
dedicated language, and it gives packagers and users access to a rich
programming interface while
retaining the purely declarative style of package definitions.</p><p>This package EDSL is one of the oldest bits of Guix and was described
<a href="https://arxiv.org/abs/1305.4584">in a 2013 paper</a>. Although embedded
in Scheme, the package “language” was designed to be understandable by
people who’ve never practiced any kind of Lisp before—you could think of
it as a parenthesized syntax for JSON or XML. It’s reasonable to say
that it’s been successful at that: of the 600+ people who contributed to
Guix over the years, most had never written Scheme or Lisp before. The
example given in that paper remains a valid package definition:</p><pre><code class="language-scheme">(define hello
(package
(name "hello")
(version "2.8")
(source (origin
(method url-fetch)
(uri (string-append "mirror://gnu/hello/hello-"
version ".tar.gz"))
(sha256 (base32 "0wqd8..."))))
(build-system gnu-build-system)
(arguments
'(#:configure-flags
`("--disable-color"
,(string-append "--with-gawk="
(assoc-ref %build-inputs "gawk")))))
(inputs `(("gawk" ,gawk)))
(synopsis "GNU Hello")
(description "An illustration of GNU's engineering practices.")
(home-page "http://www.gnu.org/software/hello/")
(license gpl3+)))</code></pre><p>Of particular interest here is the <code>inputs</code> field, which lists
build-time dependencies. Here there’s only one, GNU Awk; it has an
associated <em>label</em>, <code>"gawk"</code>. The <code>,gawk</code> bit lets us insert the value
of the <code>gawk</code> variable, which is another package. We can list more
dependencies like so:</p><pre><code class="language-scheme">(inputs `(("gawk" ,gawk)
("gettext" ,gnu-gettext)
("pkg-config" ,pkg-config)))</code></pre><p>Quite a bit of boilerplate. Unless you’re into Lisp, this probably
looks weird to you—manageable, but weird. What’s the deal with this
backquote and those commas? The backquote is shorthand for <code>quasiquote</code>
and commas are shorthand for <code>unquote</code>; it’s a facility that Lisp and
Scheme provide <a href="https://www.gnu.org/software/guile/manual/html_node/Expression-Syntax.html">to construct
lists</a>.</p><p>Lispers couldn’t live without quasiquote, it’s wonderful. Still,
exposing newcomers to this syntax has always been uncomfortable; in
tutorials you’d end up saying “yeah don’t worry, just write it this
way”. Our goal though is to empower users by giving them abstractions
they can comprehend, hopefully providing a smooth path towards
<a href="https://www.gnu.org/software/guile/manual/html_node/The-Emacs-Thesis.html">programming without
noticing</a>.
This seemingly opaque backquote-string-unquote construct works against
that goal.</p><p>Then, you ask, why did Guix adopt this unpleasant syntax for inputs in
the first place? Input syntax had to satisfy one requirement: that it’d
be possible for “build-side code” to refer to a specific input. And
what’s <em>build-side code</em>? It’s code that appears in the package
definition that is <em>staged</em> for later execution—code that’s only
evaluated when the package is actually built. The bit that follows
<code>#:configure-flags</code> in the example above is build-side code: it’s an
expression evaluated if and when the package gets built. That
<code>#:configure-flags</code> expression refers to the gawk package to construct a
flag like <code>"--with-gawk=/gnu/store/…-gawk-5.0.1"</code>; it does so by
referring to the special <code>%build-inputs</code> variable, which happens to
contain an <a href="https://www.gnu.org/software/guile/manual/html_node/Association-Lists.html">association
list</a>
that maps input labels to file names. The <code>"gawk"</code> label in <code>inputs</code> is
here to allow build-side code to get at an
input’s file name.</p><p>Still here? The paragraphs above are a demonstration of the shortcoming
of this approach. That we have to explain so much suggests we’re
lacking an abstraction that would make the whole pattern clearer.</p><h1>G-expressions and self-referential records</h1><p>The missing abstraction came to Guix a couple of years later:
<a href="https://guix.gnu.org/manual/devel/en/html_node/G_002dExpressions.html"><em>G-expressions</em></a>.
Without going into the details, which were covered elsewhere, notably
<a href="https://hal.inria.fr/hal-01580582/en">in a research article</a>,
g-expressions, or “gexps”, are traditional <a href="https://en.wikipedia.org/wiki/S-Expression">Lisp
s-expressions</a> (“sexps”) on
steroids. A gexp can contain a package record or any other “file-like
object” and, when that gexp is serialized for eventual execution, the
package is replaced by its <code>/gnu/store/…</code> file name.</p><p>Gexps have been used since 2014–2015 in all of Guix System and they’ve
been great to work with, but package definitions were stuck with
old-style sexps. One reason is that a big chunk of the code that deals
with packages and build systems had to be ported to the gexp-related
programming interfaces; a first attempt had been made long ago but
performance was not on par with that of the old code, so postponing
until that was addressed seemed wiser. The second reason was that using
gexps in package definitions could be so convenient that packagers might
unwillingly find themselves creating inconsistent packages.</p><p>We can now rewrite our <code>hello</code> package such that configure flags are
expressed using a gexp:</p><pre><code class="language-scheme">(define hello
(package
(name "hello")
;; …
(arguments
(list #:configure-flags
#~`("--disable-color"
,(string-append "--with-gawk=" #$gawk))))
(inputs `(("gawk" ,gawk)))))</code></pre><p>The reference inserted here with <code>#$gawk</code> (<code>#$</code> is synonymous for
<code>ungexp</code>, the gexp equivalent of traditional <code>unquote</code>) refers to the
global <code>gawk</code> variable. This is more concise and semantically clearer
than the <code>(assoc-ref %build-inputs "gawk")</code> snippet we had before.</p><p>Now suppose you define a package variant <a href="https://guix.gnu.org/manual/devel/en/html_node/Defining-Package-Variants.html">using this common
idiom</a>:</p><pre><code class="language-scheme">(define hello-variant
(package
(inherit hello)
(name "hello-variant")
(inputs `(("gawk" ,gawk-4.0)))))</code></pre><p>Here the intent is to create a package that depends on a different
version of GNU Awk—the hypothetical <code>gawk-4.0</code> instead of <code>gawk</code>.
However, the <code>#:configure-flags</code> gexp of this variant still refers to
the <code>gawk</code> variable, contrary to what the <code>inputs</code> field prescribes; in
effect, this variant depends on the <em>two</em> Awk versions.</p><p>To address this discrepancy, we needed a new <em>linguistic device</em>, to put
it in a fancy way. It <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=abd4d6b33dba4de228e90ad15a8efb456fcf7b6e">arrived in
2019</a>
in the form of <em>self-referential records</em>. Within a field such as the
<code>arguments</code> field above, it’s now possible to refer to <code>this-package</code> to
get at the value of the package being defined. (If you’ve done
object-oriented programming before, you might be thinking that we just
rediscovered the <code>this</code> or <code>self</code> pointer, and there’s some truth in
it. :-)) With a bit of syntactic sugar, we can rewrite the example above
so that it refers to the Awk package that appears in its own <code>inputs</code>
field:</p><pre><code class="language-scheme">(define hello
(package
(name "hello")
;; …
(arguments
(list #:configure-flags
#~(list (string-append "--with-gawk="
#$(this-package-input "gawk")))))
(inputs `(("gawk" ,gawk)))))</code></pre><p>With this in place, we can take advantage of gexps in package
definitions while still supporting the common idiom to define package
variants, wheee!</p><p>That was a long digression from our input label theme but, as you can
see, all this interacts fairly tightly.</p><h1>Getting rid of input labels</h1><p>Now that we have gexps and self-referential records, it looks like we
can finally get rid of input labels: they’re no longer strictly
necessary because we can insert packages in gexps instead of looking
them up by label in build-side code. We “just” need to devise a
backward compatibility plan…</p><p>Input labels are pervasive; they’re visible in three contexts:</p><ol><li>in the <code>inputs</code> fields of package definitions;</li><li>on the build side with the <code>inputs</code> keyword parameter of <a href="https://guix.gnu.org/manual/devel/en/html_node/Build-Phases.html">build
phases</a>;</li><li>in the Scheme programming interface since <code>package-inputs</code> and
related functions are expected to return a list of labeled inputs.</li></ol><p>We’re brave but not completely crazy, so we chose to focus on #1 for
now—it’s also the most visible of all three—, with an plan to
incrementally address #2, leaving #3 for later.</p><p>To allow for label-less inputs, we augmented the record interface with
<a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=5291fd7a4205394b863a8705b32fbb447321dc60"><em>field
sanitizers</em></a>.
This feature allows us to define a procedure that inspects and
transforms the value specified for the <code>inputs</code>, <code>native-inputs</code>, and
<code>propagated-inputs</code>. Currently that procedure <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=8524349f78c37439698f29d43049c2b21df6370f">reintroduces input
labels</a>
when they’re missing. In a sense, we’re just changing the surface
syntax but under the hood everything is unchanged. With this change,
our example above can now be written like this:</p><pre><code class="language-scheme">(define hello
(package
(name "hello")
;; …
(inputs (list gawk))))</code></pre><p>Much nicer, no? The effect is even more pleasant for packages with a
number of inputs:</p><pre><code class="language-scheme">(define-public guile-sqlite3
(package
(name "guile-sqlite3")
;; …
(build-system gnu-build-system)
(native-inputs (list autoconf automake guile-3.0 pkg-config))
(inputs (list guile-3.0 sqlite))))</code></pre><p>That’s enough to spark joy to anyone who’s taught the previous syntax.
Currently this is transformed into something like:</p><pre><code class="language-scheme">(define-public guile-sqlite3
(package
;; …
(native-inputs
(map add-input-label
(list autoconf automake guile-3.0 pkg-config)))))</code></pre><p>… where the <code>add-input-label</code> function turns a package into a
label/package pair. It does add a little bit of run-time overhead, but
nothing really measurable.</p><p>There are also cases where package definitions, in particular for
package variants, would directly manipulate input lists as returned by
<code>package-inputs</code> and related procedures. It’s a case where packagers
had to be aware of input labels, and they would typically use
<a href="https://www.gnu.org/software/guile/manual/html_node/Association-Lists.html">association list (or “alist”) manipulation
procedures</a>
and similar construct—this is context #3 above. To replace those
idioms, we defined a higher-level construct that does not assume input
labels. For example, a common idiom when defining a package variant
with additional dependencies goes like this:</p><pre><code class="language-scheme">(define hello-with-additional-dependencies
(package
(inherit hello)
(name "hello-with-bells-and-whistles")
(inputs `(("guile" ,guile-3.0)
("libtextstyle" ,libtextstyle)
,@(package-inputs hello)))))</code></pre><p>The variant defined above adds two inputs to those of <code>hello</code>. We
introduced a macro,
<a href="https://guix.gnu.org/manual/devel/en/html_node/Defining-Package-Variants.html#index-modify_002dinputs"><code>modify-inputs</code></a>,
which allows packagers to express that in a higher-level (and less
cryptic) fashion, in a way that does not refer to input labels. Using
this other linguistic device (ha ha!), the snippet above becomes:</p><pre><code class="language-scheme">(define hello-with-additional-dependencies
(package
(inherit hello)
(name "hello-with-bells-and-whistles")
(inputs (modify-inputs (package-inputs hello)
(prepend guile-3.0 libtextstyle)))))</code></pre><p>Similarly, <code>modify-inputs</code> advantageously subsumes <code>alist-delete</code> and
whatnot when willing to replace or remove inputs, like so:</p><pre><code class="language-scheme">(modify-inputs (package-inputs hello)
(replace "gawk" my-special-gawk)
(delete "guile"))</code></pre><p>On the build side—context #2 above—, we also provide new procedures that
allow packagers to avoid relying on input labels: <a href="https://guix.gnu.org/manual/devel/en/html_node/Build-Utilities.html#index-search_002dinput_002dfile"><code>search-input-file</code>
and
<code>search-input-directory</code></a>.
Instead of having build phases that run code like:</p><pre><code class="language-scheme">(lambda* (#:key inputs #:allow-other-keys)
;; Replace references to “/sbin/ip” by references to
;; the actual “ip” command in /gnu/store.
(substitute* "client/scripts/linux"
(("/sbin/ip")
;; Look up the input labeled “iproute”.
(string-append (assoc-ref inputs "iproute")
"/sbin/ip"))))</code></pre><p>… you’d now write:</p><pre><code class="language-scheme">(lambda* (#:key inputs #:allow-other-keys)
;; Replace references to “/sbin/ip” by references to
;; the actual “ip” command in /gnu/store.
(substitute* "client/scripts/linux"
(("/sbin/ip")
;; Search “/sbin/ip” among all the inputs.
(search-input-file inputs "/sbin/ip"))))</code></pre><p>Nothing revolutionary but a couple of advantages: code is no longer tied
to input labels or package names, and <code>search-input-file</code> raises an
exception when the file is not found, which is better than building an
incorrect file name.</p><p>That was a deep dive into packaging! If you’re already packaging
software for Guix, you hopefully see how to do things “the new way”.
Either way, it’s interesting to see the wide-ranging implications of
what initially looks like a simple change. Things get complex when you
have to consider all the idioms that 20,000 packages make use of.</p><h1>Adapting to the new style</h1><p>It’s nice to have a plan to change the style of package definitions, but
how do you make it happen concretely? The last thing we want is, for
the next five years, to have to explain <em>two</em> package styles to
newcomers instead of one.</p><p>First, <a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-import.html"><code>guix import</code></a>
now returns packages in the new style. But what about existing package
definitions?</p><p>Fortunately, most of the changes above can be automated: that’s the job
of the <a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-style.html"><code>guix style</code></a>
command that we added for this purpose, but which may eventually be
extended to make package definitions prettier in all sorts of ways.
From a checkout, one can run:</p><pre><code>./pre-inst-env guix style</code></pre><p>Whenever it can be done safely, package inputs in <em>every</em> package
definition are rewritten to the new style: removing input labels, and
using <code>modify-inputs</code> where appropriate. If you maintain your own
channel, you can also run it for your packages:</p><pre><code>guix style -L /path/to/channel my-package1 my-package2 …</code></pre><p>We recommend waiting until the next Guix release is out though, which
could be a month from now, so that your channel remains usable by those
who pinned an older revision of the <code>guix</code> channel.</p><p>We ran <code>guix style</code> a couple of days ago on the whole repository,
leading to <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=8394619baceb118df92e355377fd543bb1aa501a">the biggest
commit</a>
in Guix history:</p><pre><code>460 files changed, 37699 insertions(+), 49782 deletions(-)</code></pre><p>Woohoo! Less than 15% of the packages (not counting Rust packages,
which are a bit special) have yet to be adjusted. In most cases,
package inputs that were left unchanged are either those where we cannot
automatically determine whether the change would break the package, for
instance because build-side code expects certain labels, and those that
are “complex”—e.g., inputs that include conditionals.</p><p>The key challenge here was making sure <code>guix style</code> transformations are
correct; by default, we even want to be sure that changes introduced by
<code>guix style</code> do not trigger a rebuild—that package
<a href="https://guix.gnu.org/manual/devel/en/html_node/Derivations.html">derivations</a>
are unchanged.</p><p>To achieve that, <code>guix style</code> <em>correlates</em> the source code of each
package definition with the corresponding live package record. That
allows it to answer questions such as “is this label the name of the
input package”. That way, it can tell that, say:</p><pre><code class="language-scheme">(inputs `(("libx11" ,libx11)
("libxcb" ,libxcb)))</code></pre><p>can be rewritten without introducing a rebuild, because labels match
actual package names, whereas something like this cannot, due to label
mismatches:</p><pre><code class="language-scheme">(inputs `(("libX11" ,libx11)
("xcb" ,libxcb)))</code></pre><p><code>guix style</code> can also determine situations where changes would trigger a
rebuild but would still be “safe”—without any observable effect. You
can force it to make such changes by running:</p><pre><code>guix style --input-simplification=safe</code></pre><p>Because we’re all nitpicky when it comes to code formatting, <code>guix style</code> had to produce nicely formatted code, and to make local changes
as opposed to rewriting complete package definitions. Lisps are famous
for being <a href="https://en.wikipedia.org/wiki/Homoiconicity">homoiconic</a>,
which comes in handy in such a situation.</p><p>But the tools at our disposal are not capable enough for this
application. First, Scheme’s standard
<a href="https://www.gnu.org/software/guile/manual/html_node/Scheme-Read.html"><code>read</code></a>
procedure, which reads an sexp (or an abstract syntax tree if you will)
from a byte stream and returns it, does not preserve comments.
Obviously we’d rather not throw away comments, so we came up with our
own <code>read</code> variant that preserves comments. Similarly, we have a custom
pretty printer that can write comments, allowing it to achieve changes
like this:</p><pre><code class="language-diff">@@ -2749,18 +2707,17 @@ (define-public debops
"debops-debops-defaults-fall-back-to-less.patch"))))
(build-system python-build-system)
(native-inputs
- `(("git" ,git)))
+ (list git))
(inputs
- `(("ansible" ,ansible)
- ("encfs" ,encfs)
- ("fuse" ,fuse)
- ("util-linux" ,util-linux) ;for umount
- ("findutils" ,findutils)
- ("gnupg" ,gnupg)
- ("which" ,which)))
+ (list ansible
+ encfs
+ fuse
+ util-linux ;for umount
+ findutils
+ gnupg
+ which))
(propagated-inputs
- `(("python-future" ,python-future)
- ("python-distro" ,python-distro)))
+ (list python-future python-distro))
(arguments
`(#:tests? #f</code></pre><p>The pretty printer also has special rules for input lists. For
instance, lists of five inputs or less go into a single line, if
possible, whereas longer lists end up with one input per line, which is
often more convenient, especially when visualizing diffs. It also has
rules to format <code>modify-inputs</code> in the same way we’d do it in Emacs:</p><pre><code class="language-diff">@@ -171,9 +170,9 @@ (define-public arcan-sdl
(inherit arcan)
(name "arcan-sdl")
(inputs
- `(("sdl" ,sdl)
- ,@(fold alist-delete (package-inputs arcan)
- '("libdrm"))))
+ (modify-inputs (package-inputs arcan)
+ (delete "libdrm")
+ (prepend sdl)))
(arguments</code></pre><p>Overall that makes <code>guix style</code> a pretty fun meta-project!</p><h1>“Worse is better” or “the Right Thing”?</h1><p>There are several lessons here. One is that having an <em>embedded</em>
domain-specific language is what makes such changes possible: yes
package definitions have a declarative field, but we do not hide the
fact that their meaning is determined by the broader Guix framework,
starting with the <code>(guix packages)</code> module, which defines the package
record type and associated procedures. Having a single repository
containing both package definitions <em>and</em> “the package manager” is also
a key enabler; we can change the framework, add new linguistic tools,
<em>and</em> adjust package definitions at the same time. This is in contrast,
for instance, with the approach taken by Nix, where the language
implementation lives separately from the package collection.</p><p>Another one is that a strong, consistent community leads to consistent
changes—<a href="https://en.wikipedia.org/wiki/Conway%27s_law">not surprisingly in
fact</a>. It’s a pleasure to
see that we, collectively, can undertake such overarching changes and
all look in the same direction.</p><p>The last lesson is in how we approach design issues in a project that is
now a little more than nine years old. Over these nine years it’s clear
that we have usually favored <a href="https://en.wikipedia.org/wiki/Worse_is_better#The_MIT_approach">“the right
thing”</a>
in our design—but not at any cost. This whole change is an illustration
of this. It was clear from the early days of Guix that input labels and
the associated machinery were suboptimal. But it took us a few years to
design an implement the missing pieces: G-expressions, self-referential
records, and the various idioms that allow package definitions to
benefit from these. In the meantime, we built a complete system <em>and a
community</em> and we gained experience. We cannot claim we attained <em>the</em>
right thing, if such a thing exists, but certainly package definitions
today are closer to the declarative ideal and easier to teach.</p><h1>It’s here today!</h1><p>This big change, along with countless other improvements and package
upgrades, is just one <code>guix pull</code> away! After months of development, we
have just merged the “core updates” branch bringing so many new
things—from GNOME 41, to GCC 10 by default, to hardened Python packages
and <a href="https://guix.gnu.org/en/blog/2021/taming-the-stat-storm-with-a-loader-cache/">improved executable startup
times</a>.
This paves the way for the upcoming release, most likely labeled “1.4”,
unless a closer review of the changes that have landed leads us to think
“2.0” would be more appropriate… Stay tuned!</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package manager and
an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects user
freedom</a>.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, AArch64 and POWER9 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2021/taming-the-stat-storm-with-a-loader-cache//Taming the ‘stat’ storm with a loader cacheLudovic Courtès2021-08-02T15:00:00Z2021-08-02T15:00:00Z It was one of these days where some of us on IRC were rehashing that old
problem—that application startup in Guix causes a
“ stat storm”—and lamenting the
lack of a solution when suddenly, Ricardo
proposes what,
in hindsight, looks like an obvious solution: “maybe we could use a
per-application ld cache?”. A moment where collective thinking exceeds
the sum of our individual thoughts. The result is one of the many
features that made it in the core-updates branch, slated to be merged
in the coming weeks, one that reduces application startup time. ELF files and their…<p>It was one of these days where some of us on IRC were rehashing that old
problem—that application startup in Guix causes a
“<a href="https://linux.die.net/man/2/stat"><code>stat</code></a> storm”—and lamenting the
lack of a solution when suddenly, Ricardo
<a href="https://logs.guix.gnu.org/guix/2020-11-24.log#183934">proposes</a> what,
in hindsight, looks like an obvious solution: “maybe we could use a
per-application ld cache?”. A moment where collective thinking exceeds
the sum of our individual thoughts. The result is one of the many
features that made it in the <code>core-updates</code> branch, slated to be merged
in the coming weeks, one that reduces application startup time.</p><h1>ELF files and their dependencies</h1><p>Before going into detail, let’s look at what those “<code>stat</code> storms” look
like and where they come from. Loading an
<a href="https://en.wikipedia.org/wiki/Executable_and_Linkable_Format">ELF</a>
executable involves loading the shared libraries (the <code>.so</code> files, for
“shared objects”) it depends on, recursively. This is the job of the
<em>loader</em> (or <em>dynamic linker</em>), <code>ld.so</code>, which is part of the GNU C
Library (glibc) package. What shared libraries an executable like that
of Emacs depends on? The <code>ldd</code> command answers that question:</p><pre><code>$ ldd $(type -P .emacs-27.2-real)
linux-vdso.so.1 (0x00007fff565bb000)
libtiff.so.5 => /gnu/store/l1wwr5c34593gqxvp34qbwdkaf7xhdbd-libtiff-4.2.0/lib/libtiff.so.5 (0x00007fd5aa2b1000)
libjpeg.so.62 => /gnu/store/5khkwz9g6vza1n4z8xlmdrwhazz7m8wp-libjpeg-turbo-2.0.5/lib/libjpeg.so.62 (0x00007fd5aa219000)
libpng16.so.16 => /gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/libpng16.so.16 (0x00007fd5aa1e4000)
libz.so.1 => /gnu/store/rykm237xkmq7rl1p0nwass01p090p88x-zlib-1.2.11/lib/libz.so.1 (0x00007fd5aa1c2000)
libgif.so.7 => /gnu/store/bpw826hypzlnl4gr6d0v8m63dd0k8waw-giflib-5.2.1/lib/libgif.so.7 (0x00007fd5aa1b8000)
libXpm.so.4 => /gnu/store/jgdsl6whyimkz4hxsp2vrl77338kpl0i-libxpm-3.5.13/lib/libXpm.so.4 (0x00007fd5aa1a4000)
[…]
$ ldd $(type -P .emacs-27.2-real) | wc -l
89</code></pre><p>(If you’re wondering why we’re looking at <code>.emacs-27.2-real</code> rather than
<code>emacs-27.2</code>, it’s because in Guix the latter is a tiny shell wrapper
around the former.)</p><p>To load a graphical program like Emacs, the loader needs to load more
than 80 shared libraries! Each is in its own <code>/gnu/store</code> sub-directory
in Guix, one directory per package.</p><p>But how does <code>ld.so</code> know where to find these libraries in the first
place? In Guix, during the link phase that produces an ELF file
(executable or shared library), we tell the
<a href="https://en.wikipedia.org/wiki/Linker_%28computing%29">linker</a> to
populate the <code>RUNPATH</code> entry of the ELF file with the list of
directories where its dependencies may be found. This is done by
passing
<a href="https://sourceware.org/binutils/docs/ld/Options.html#index-_002drpath_003ddir"><code>-rpath</code></a>
options to the linker, which Guix’s <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/ld-wrapper.in">“linker
wrapper”</a>
takes care of. The <code>RUNPATH</code> is the <em>run-time library search path</em>:
it’s a colon-separated list of directories where <code>ld.so</code> will look for
shared libraries when it loads an ELF file. We can look at the
<code>RUNPATH</code> of our Emacs executable like this:</p><pre><code>$ objdump -x $(type -P .emacs-27.2-real) | grep RUNPATH
RUNPATH /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/lib:/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib:/gnu/store/l1wwr5c34593gqxvp34qbwdkaf7xhdbd-libtiff-4.2.0/lib:/gnu/store/5khkwz9g6vza1n4z8xlmdrwhazz7m8wp-libjpeg-turbo-2.0.5/lib:[…]</code></pre><p>This <code>RUNPATH</code> has 39 entries, which roughly corresponds to the number
of direct dependencies of the executable—dependencies are listed as
<code>NEEDED</code> entries in the ELF file:</p><pre><code>$ objdump -x $(type -P .emacs-27.2-real) | grep NEED | head
NEEDED libtiff.so.5
NEEDED libjpeg.so.62
NEEDED libpng16.so.16
NEEDED libz.so.1
NEEDED libgif.so.7
NEEDED libXpm.so.4
NEEDED libgtk-3.so.0
NEEDED libgdk-3.so.0
NEEDED libpangocairo-1.0.so.0
NEEDED libpango-1.0.so.0
$ objdump -x $(type -P .emacs-27.2-real) | grep NEED | wc -l
52</code></pre><p>(Some of these <code>.so</code> files live in the same directory, which is why
there are more <code>NEEDED</code> entries than directories in the <code>RUNPATH</code>.)</p><p>A system such as Debian that follows the <a href="https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard">file system hierarchy
standard</a>
(FHS), where all libraries are in <code>/lib</code> or <code>/usr/lib</code>, does not have to
bother with <code>RUNPATH</code>: all <code>.so</code> files are known to be found in one of
these two “standard” locations. Anyway, let’s get back to our initial
topic: the “<code>stat</code> storm”.</p><h1>Walking search paths</h1><p>As you can guess, when we run Emacs, the loader first needs to locate
and load the 80+ shared libraries it depends on. That’s where things
get pretty inefficient: the loader will search each <code>.so</code> file Emacs
depends on in one of the 39 directories listed in its <code>RUNPATH</code>.
Likewise, when it finally finds <code>libgtk-3.so</code>, it’ll look for its
dependencies in each of the directories in its <code>RUNPATH</code>. We can see
that at play by tracing system calls with the
<a href="https://strace.io/"><code>strace</code></a> command:</p><pre><code>$ strace -c emacs --version
GNU Emacs 27.2
Copyright (C) 2021 Free Software Foundation, Inc.
GNU Emacs comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GNU Emacs
under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING.
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
55.46 0.006629 3 1851 1742 openat
16.06 0.001919 4 422 mmap
11.46 0.001370 2 501 477 stat
4.79 0.000573 4 122 mprotect
3.84 0.000459 4 111 read
2.45 0.000293 2 109 fstat
2.34 0.000280 2 111 close
[…]
------ ----------- ----------- --------- --------- ----------------
100.00 0.011952 3 3325 2227 total</code></pre><p>For this simple <code>emacs --version</code> command, the loader and <code>emacs</code> probed
for more than 2,200 files, with the
<a href="https://linux.die.net/man/2/openat"><code>openat</code></a> and
<a href="https://linux.die.net/man/2/stat"><code>stat</code></a> system calls, and most of
these probes were unsuccessful (counted as “errors” here, meaning that
the call returned an error). The fraction of “erroneous” system calls
is no less than 67% (2,227 over 3,325). We can see the desperate search
of <code>.so</code> files by looking at individual calls:</p><pre><code>$ strace -e openat,stat emacs --version
[…]
openat(AT_FDCWD, "/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/lib/libpng16.so.16", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/libpng16.so.16", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/gnu/store/l1wwr5c34593gqxvp34qbwdkaf7xhdbd-libtiff-4.2.0/lib/libpng16.so.16", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/gnu/store/5khkwz9g6vza1n4z8xlmdrwhazz7m8wp-libjpeg-turbo-2.0.5/lib/libpng16.so.16", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/tls/haswell/x86_64/libpng16.so.16", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/tls/haswell/x86_64", 0x7ffe428a1c70) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/tls/haswell/libpng16.so.16", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/tls/haswell", 0x7ffe428a1c70) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/tls/x86_64/libpng16.so.16", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/tls/x86_64", 0x7ffe428a1c70) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/tls/libpng16.so.16", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/tls", 0x7ffe428a1c70) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/haswell/x86_64/libpng16.so.16", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/haswell/x86_64", 0x7ffe428a1c70) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/haswell/libpng16.so.16", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/haswell", 0x7ffe428a1c70) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/x86_64/libpng16.so.16", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/x86_64", 0x7ffe428a1c70) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/libpng16.so.16", O_RDONLY|O_CLOEXEC) = 3
[…]</code></pre><p>Above is the sequence where we see <code>ld.so</code> look for <code>libpng16.so.16</code>,
searching in locations where we <em>know</em> it’s not going to find it. A bit
ridiculous. How does this affect performance? The impact is small in
the most favorable case—on a hot cache, with fast solid state device
(SSD) storage. But it likely has a visible effect in other cases—on a
cold cache, with a slower spinning hard disk drive (HDD), on a network
file system (NFS).</p><h1>Enter the per-package loader cache</h1><p>The idea that Ricardo submitted, using a loader cache, makes a lot of
sense: we know from the start that <code>libpng.so</code> may only be found in
<code>/gnu/store/…-libpng-1.6.37</code>, no need to look elsewhere. In fact, it’s
not new: glibc has had such a cache “forever”; it’s the
<code>/etc/ld.so.cache</code> file you can see on FHS distros and which is
typically created by running
<a href="https://linux.die.net/man/8/ldconfig"><code>ldconfig</code></a> when a package has
been installed. Roughly, the cache maps library <code>SONAME</code>s, such as
<code>libpng16.so.16</code>, to their file name on disk, say
<code>/usr/lib/libpng16.so.16</code>.</p><p>The problem is that this cache is inherently system-wide: it assumes
that there is only <em>one</em> <code>libpng16.so</code> on the system; any binary that
depends on <code>libpng16.so</code> will load it from its one and only location.
This models perfectly matches the FHS, but it’s at odds with the
flexibility offered by Guix, where several variants or versions of the
library can coexist on the system, used by different applications.
That’s the reason why Guix and other non-FHS distros such as NixOS or
GoboLinux typically <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/base.scm?id=a92dfbce30777de6ca05031e275410cf9f56c84c#n716">turn
off</a>
that feature altogether… and pay the cost of those <code>stat</code> storms.</p><p>The insight we gained on that Tuesday evening IRC conversation is that
we could <em>adapt</em> glibc’s loader cache to our setting: instead of a
system-wide cache, we’d have a <em>per-application loader cache</em>. As one
of the last package <a href="https://guix.gnu.org/manual/en/html_node/Build-Phases.html">build
phases</a>,
we’d run <code>ldconfig</code> to create <code>etc/ld.so.cache</code> within that package’s
<code>/gnu/store</code> sub-directory. We then need to modify the loader so it
would look for <code>${ORIGIN}/../etc/ld.so.cache</code> instead of
<code>/etc/ld.so.cache</code>, where <code>${ORIGIN}</code> is the location of the ELF file
being loaded. A discussion of these changes is <a href="https://issues.guix.gnu.org/44899">in the issue
tracker</a>; you can see <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/patches/glibc-dl-cache.patch?id=0236013cd0fc86ff4a042885c735e3f36a7f5c25">the glibc
patch</a>
and the new <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/gnu-build-system.scm?id=0236013cd0fc86ff4a042885c735e3f36a7f5c25#n735"><code>make-dynamic-linker-cache</code> build
phase</a>.
In short, the <code>make-dynamic-linker-cache</code> phase computes the set of
direct and indirect dependencies of an ELF file using the
<a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/gremlin.scm?id=0236013cd0fc86ff4a042885c735e3f36a7f5c25#n265"><code>file-needed/recursive</code></a>
procedure and derives from that the library search path, creates a
temporary <code>ld.so.conf</code> file containing this search path for use by
<code>ldconfig</code>, and finally runs <code>ldconfig</code> to actually build the cache.</p><p>How does this play out in practice? Let’s try an <code>emacs</code> build that
uses this new loader cache:</p><pre><code>$ strace -c /gnu/store/ijgcbf790z4x2mkjx2ha893hhmqrj29j-emacs-27.2/bin/emacs --version
GNU Emacs 27.2
Copyright (C) 2021 Free Software Foundation, Inc.
GNU Emacs comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GNU Emacs
under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING.
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
28.68 0.002909 26 110 13 openat
25.13 0.002549 26 96 read
20.41 0.002070 4 418 mmap
9.34 0.000947 10 90 pread64
6.60 0.000669 5 123 mprotect
4.12 0.000418 3 107 1 newfstatat
2.19 0.000222 2 99 close
[…]
------ ----------- ----------- --------- --------- ----------------
100.00 0.010144 8 1128 24 total</code></pre><p>Compared to what we have above, the total number of system calls has
been divided by 3, and the fraction of erroneous system calls goes from
67% to 0.2%. Quite a difference! We count on you, dear users, to <a href="https://guix.gnu.org/en/contact">let
us know</a> how this impacts load time for
you.</p><h1>Flexibility without <code>stat</code> storms</h1><p>With <a href="https://www.gnu.org/software/stow">GNU Stow</a> in the 1990s, and
then Nix, Guix, and other distros, the benefits of flexible file layouts
rather than the rigid Unix-inherited FHS have been demonstrated—nowadays
I see it as an antidote to opaque and bloated application bundles à la
Docker. Luckily, few of our system tools have FHS assumptions baked in,
probably in large part thanks to GNU’s insistence on a <a href="https://www.gnu.org/prep/standards/html_node/Directory-Variables.html">rigorous
installation directory
categorization</a>
in the early days rather than hard-coded directory names. The loader
cache is one of the few exceptions. Adapting it to a non-FHS context is
fruitful for Guix and for the other distros and packaging tools in a
similar situation; perhaps it could become an option in glibc proper?</p><p>This is not the end of <code>stat</code> storms, though. Interpreters and language
run-time systems rely on search paths—<code>GUILE_LOAD_PATH</code> for Guile,
<code>PYTHONPATH</code> for Python, <code>OCAMLPATH</code> for OCaml, etc.—and are equally
prone to stormy application startups. Unlike ELF, they do not have a
mechanism akin to <code>RUNPATH</code>, let alone a run-time search path cache. We
have yet to find ways to address these.</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package manager and
an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects user
freedom</a>.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, AArch64 and POWER9 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2021/reproducible-data-processing-pipelines//Reproducible data processing pipelinesLudovic Courtès2021-06-11T17:00:00Z2021-06-11T17:00:00Z Last week, we at Guix-HPC published videos of
a workshop on reproducible software
environments
we organized on-line. The videos are well worth watching—especially if
you’re into reproducible research, and especially if you speak French or
want to practice. This post, though, is more of a meta-post: it’s about
how we processed these videos. “A workshop on reproducibility ought to
have a reproducible video pipeline”, we thought. So this is what we
did ! From BigBlueButton to WebM Over the last year and half, perhaps you had the “opportunity” to
participate in an on-line conference,…<p>Last week, <a href="https://hpc.guix.info">we at Guix-HPC</a> published <a href="https://hpc.guix.info/events/2021/atelier-reproductibilit%C3%A9-environnements/">videos of
a workshop on reproducible software
environments</a>
we organized on-line. The videos are well worth watching—especially if
you’re into reproducible research, and especially if you speak French or
want to practice. This post, though, is more of a meta-post: it’s about
how we processed these videos. “A workshop on reproducibility <em>ought to
have</em> a reproducible video pipeline”, we thought. So this is what we
<a href="https://gitlab.inria.fr/guix-hpc/website/-/blob/master/doc/atelier-reproductibilit%C3%A9/render-videos.scm">did</a>!</p><h1>From BigBlueButton to WebM</h1><p>Over the last year and half, perhaps you had the “opportunity” to
participate in an on-line conference, or even to organize one. If so,
chances are that you already know
<a href="https://bigbluebutton.org/">BigBlueButton</a> (BBB), the free software
video conferencing suite initially designed for on-line teaching. In a
nutshell, it allows participants to chat (audio, video, and keyboard),
and speakers can share their screen or a PDF slide deck. Organizers can
also record the session.</p><p>BBB then creates a link to recorded sessions with a custom JavaScript
player that replays everything: typed chat, audio and video (webcams),
shared screens, and slide decks. This BBB replay a bit too rough though
and often not the thing you’d like to publish after the conference.
Instead, you’d rather do a bit of editing: adjusting the start and end
time of each talk, removing live chat from what’s displayed (which
allows you to remove info that personally identifies participants,
too!), and so forth. Turns out this kind of post-processing is a bit of
work, primarily because BBB does “the right thing” of recording each
stream separately, in the most appropriate form: webcam and screen
shares are recorded as separate videos, chat is recorded as text with
timings, slide decks is recorded as a bunch of PNGs plus timings, and
then there’s a bunch of XML files with metadata putting it all together.</p><p>Anyway, with a bit of searching, we quickly found the handy
<a href="https://github.com/plugorgau/bbb-render">bbb-render</a> tool, which can
first
<a href="https://github.com/plugorgau/bbb-render/blob/master/download.py">download</a>
all these files and then
<a href="https://github.com/plugorgau/bbb-render/blob/master/make-xges.py">assemble</a>
them using the Python interface to the <a href="https://gstreamer.freedesktop.org/documentation/gst-editing-services/index.html">GStreamer Editing Services
(GES)</a>.
Good thing: we don’t have to figure out all these things; we “just” have
to run these two scripts in an environment with the right dependencies.
And guess what: we know of a great tool to control execution
environments!</p><h1>A “deployment-aware Makefile”</h1><p>So we have a process that takes input files—those PNGs, videos, and XML
files—and produces output files—WebM video files. As developers we
immediately recognize a pattern and the timeless tool to deal with it:
<a href="https://www.gnu.org/software/make"><code>make</code></a>. The web already seems to
contain countless BBB post-processing makefiles (and shell scripts,
too). We were going to contribute to this while we suddenly realized
that we know of <em>another</em> great tool to express such processes: Guix!
Bonus: while a makefile would address just the tip of the
iceberg—running bbb-render—Guix can also take care of the tedious task
of deploying the <em>right</em> environment to run bbb-render in.</p><p>What we did was to write some sort of a <em>deployment-aware makefile</em>.
It’s still a relatively unconventional way to use Guix, but one that’s
very convenient. We’re talking about videos, but really, you could use
the same approach for any kind of processing graph where you’d be
tempted to just use <code>make</code>.</p><p>The end result here is a <a href="https://gitlab.inria.fr/guix-hpc/website/-/blob/6977da4618814c790e767618da5cf9ec2cab0742/doc/atelier-reproductibilit%C3%A9/render-videos.scm">Guix
file</a>
that returns a <em>manifest</em>—a list of videos to “build”. You can build
the videos with:</p><pre><code>guix build -m render-videos.scm</code></pre><p>Overall, the file defines a bunch of functions (<em>procedures</em> in
traditional Scheme parlance), each of which takes input files and
produces output files. More accurately, these functions returns objects
that <em>describe</em> how to build their output from the input files—similar
to how a <a href="https://www.gnu.org/software/make/manual/html_node/Rule-Introduction.html">makefile
rule</a>
describes how to build its target(s) from its prerequisite(s). (The
reader familiar with functional programming may recognize a monad here,
and indeed, those build descriptions can be thought of as monadic values
in a hypothetical “Guix build” monad; technically though, they’re
regular Scheme values.)</p><p>Let’s take a guided tour of this 300-line file.</p><h1>Rendering</h1><p>The <a href="https://gitlab.inria.fr/guix-hpc/website/-/blob/6977da4618814c790e767618da5cf9ec2cab0742/doc/atelier-reproductibilit%C3%A9/render-videos.scm#L23-75">first
step</a>
in this file describes where bbb-render can be found and how to run it
to produce a GES “project” file, which we’ll use later to render the
video:</p><pre><code class="language-scheme">(define bbb-render
(origin
(method git-fetch)
(uri (git-reference (url "https://github.com/plugorgau/bbb-render")
(commit "a3c10518aedc1bd9e2b71a4af54903adf1d972e5")))
(file-name "bbb-render-checkout")
(sha256
(base32 "1sf99xp334aa0qgp99byvh8k39kc88al8l2wy77zx7fyvknxjy98"))))
(define rendering-profile
(profile
(content (specifications->manifest
'("gstreamer" "gst-editing-services" "gobject-introspection"
"gst-plugins-base" "gst-plugins-good"
"python-wrapper" "python-pygobject" "python-intervaltree")))))
(define* (video-ges-project bbb-data start end
#:key (webcam-size 25))
"Return a GStreamer Editing Services (GES) project for the video,
starting at START seconds and ending at END seconds. BBB-DATA is the raw
BigBlueButton directory as fetched by bbb-render's 'download.py' script.
WEBCAM-SIZE is the percentage of the screen occupied by the webcam."
(computed-file "video.ges"
(with-extensions (list (specification->package "guile-gcrypt"))
(with-imported-modules (source-module-closure
'((guix build utils)
(guix profiles)))
#~(begin
(use-modules (guix build utils) (guix profiles)
(guix search-paths) (ice-9 match))
(define search-paths
(profile-search-paths #+rendering-profile))
(for-each (match-lambda
((spec . value)
(setenv
(search-path-specification-variable
spec)
value)))
search-paths)
(invoke "python"
#+(file-append bbb-render "/make-xges.py")
#+bbb-data #$output
"--start" #$(number->string start)
"--end" #$(number->string end)
"--webcam-size"
#$(number->string webcam-size)))))))</code></pre><p>First it defines the source code location of bbb-render as an
<a href="https://guix.gnu.org/manual/en/html_node/origin-Reference.html">“origin”</a>.
Second, it defines <code>rendering-profile</code> as a
<a href="https://guix.gnu.org/manual/en/html_node/Getting-Started.html#index-profile">“profile”</a>
containing all the packages needed to run bbb-render’s <code>make-xges.py</code>
script. The <code>specification->manifest</code> procedure creates a <em>manifest</em>
from a set of packages specs, and likewise <code>specification->package</code>
returns the package that matches a given spec. You can try these things at
the <a href="https://guix.gnu.org/manual/en/html_node/Invoking-guix-repl.html"><code>guix repl</code></a>
prompt:</p><pre><code>$ guix repl
GNU Guile 3.0.7
Copyright (C) 1995-2021 Free Software Foundation, Inc.
Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.
Enter `,help' for help.
scheme@(guix-user)> ,use(guix profiles)
scheme@(guix-user)> ,use(gnu)
scheme@(guix-user)> (specification->package "guile@2.0")
$1 = #<package guile@2.0.14 gnu/packages/guile.scm:139 7f416be776e0>
scheme@(guix-user)> (specifications->manifest '("guile" "gstreamer" "python"))
$2 = #<<manifest> entries: (#<<manifest-entry> name: "guile" version: "3.0.7" …> #<<manifest-entry> name: "gstreamer" version: "1.18.2" …> …)</code></pre><p>Last, it defines <code>video-ges-project</code> as a function that takes the BBB
raw data, a start and end time, and produces a <code>video.ges</code> file. There
are three key elements here:</p><ol><li><a href="https://guix.gnu.org/manual/en/html_node/G_002dExpressions.html#index-computed_002dfile"><code>computed-file</code></a>
is a function to produce a file, <code>video.ges</code> in this case, by
running the code you give it as its second argument—the <em>recipe</em>,
in makefile terms.</li><li>The recipe passed to <code>computed-file</code> is a
<a href="https://guix.gnu.org/manual/en/html_node/G_002dExpressions.html"><em>G-expression</em></a>
(or “gexp”), introduced by this fancy <code>#~</code> (hash tilde) notation.
G-expressions are a way to <em>stage</em> code, to mark it for eventual
execution. Indeed, that code will only be executed if and when we
run <code>guix build</code> (without <code>--dry-run</code>), and only if the result is
not already in <a href="https://guix.gnu.org/manual/en/html_node/The-Store.html">the
store</a>.</li><li>The gexp refers to <code>rendering-profile</code>, to <code>bbb-render</code>, to
<code>bbb-data</code> and so on by <em>escaping</em> with the <code>#+</code> or <code>#$</code> syntax
(they’re equivalent, unless doing cross-compilation). During
build, these reference items in the store, such as
<code>/gnu/store/…-bbb-render</code>, which is itself the result of “building”
the origin we’ve seen above. The <code>#$output</code> reference corresponds
to the build result of this <code>computed-file</code>, the complete file name
of <code>video.ges</code> under <code>/gnu/store</code>.</li></ol><p>That’s quite a lot already! Of course, this real-world example is
more intimidating than the toy examples you’d find in the manual, but
really, pretty much everything’s there. Let’s see in more detail at
what’s inside this gexp.</p><p>The gexp first imports a bunch of helper modules with <a href="https://guix.gnu.org/manual/en/html_node/Build-Utilities.html">build
utilities</a>
and tools to manipulate profiles and search path environment variables.
The <code>for-each</code> call iterates over search path environment
variables—<code>PATH</code>, <code>PYTHONPATH</code>, and so on—, setting them so that the
<code>python</code> command is found and so that the needed Python modules are
found.</p><p>The <code>with-imported-modules</code> form above indicates that the <code>(guix build utils)</code> and <code>(guix profiles)</code> modules, which are part of Guix, along
with their dependencies (their <em>closure</em>), need to be imported in the
build environment. What about <code>with-extensions</code>? Those <code>(guix …)</code>
module indirectly depend on additional modules, provided by the
<code>guile-gcrypt</code> package, hence this spec.</p><p>Next comes the
<a href="https://gitlab.inria.fr/guix-hpc/website/-/blob/6977da4618814c790e767618da5cf9ec2cab0742/doc/atelier-reproductibilit%C3%A9/render-videos.scm#L77-106"><code>ges->webm</code></a>
function which, as the name implies, takes a <code>.ges</code> file and produces a
WebM video file by invoking <code>ges-launch-1.0</code>. The end result is a video
containing the recording’s audio, the webcam and screen share (or slide
deck), but not the chat.</p><h1>Opening and closing</h1><p>We have a WebM video, so we’re pretty much done, right? But… we’d also
like to have an opening, showing the talk title and the speaker’s name,
as well as a closing. How do we get that done?</p><p>Perhaps a bit of a sledgehammer, but it turns out that we chose to
produce those still images with LaTeX/Beamer, from
<a href="https://gitlab.inria.fr/guix-hpc/website/-/blob/6977da4618814c790e767618da5cf9ec2cab0742/doc/atelier-reproductibilit%C3%A9/opening.tex">these</a>
<a href="https://gitlab.inria.fr/guix-hpc/website/-/blob/6977da4618814c790e767618da5cf9ec2cab0742/doc/atelier-reproductibilit%C3%A9/closing.tex">templates</a>.</p><p>We need again several processing steps:</p><ol><li>We first define the
<a href="https://gitlab.inria.fr/guix-hpc/website/-/blob/6977da4618814c790e767618da5cf9ec2cab0742/doc/atelier-reproductibilit%C3%A9/render-videos.scm#L140-166"><code>latex->pdf</code></a>
function that takes a template <code>.tex</code> file, a speaker name and
title. It copies the template, replaces placeholders with the
speaker name and title, and runs <code>pdflatex</code> to produce the PDF.</li><li>The
<a href="https://gitlab.inria.fr/guix-hpc/website/-/blob/6977da4618814c790e767618da5cf9ec2cab0742/doc/atelier-reproductibilit%C3%A9/render-videos.scm#L168-175"><code>pdf->bitmap</code></a>
function takes a PDF and returns a suitably-sized JPEG.</li><li><a href="https://gitlab.inria.fr/guix-hpc/website/-/blob/6977da4618814c790e767618da5cf9ec2cab0742/doc/atelier-reproductibilit%C3%A9/render-videos.scm#L177-200"><code>image->webm</code></a>
takes that JPEG and invokes <code>ffmpeg</code> to render it as WebM, with the
right resolution, frame rate, and audio track.</li></ol><p>With that in place, we define a sweet and small function that produces
the opening WebM file for a given talk:</p><pre><code class="language-scheme">(define (opening title speaker)
(image->webm
(pdf->bitmap (latex->pdf (local-file "opening.tex") "opening.pdf"
#:title title #:speaker speaker)
"opening.jpg")
"opening.webm" #:duration 5))</code></pre><p>We need one last function,
<a href="https://gitlab.inria.fr/guix-hpc/website/-/blob/6977da4618814c790e767618da5cf9ec2cab0742/doc/atelier-reproductibilit%C3%A9/render-videos.scm#L216-236"><code>video-with-opening/closing</code></a>,
that given a talk, an opening, and a closing, concatenates them by
invoking <code>ffmpeg</code>.</p><h1>Putting it all together</h1><p>Now we have all the building blocks!</p><p>We use
<a href="https://guix.gnu.org/manual/en/html_node/G_002dExpressions.html#index-local_002dfile"><code>local-file</code></a>
to refer to the raw BBB data, taken from disk:</p><pre><code class="language-scheme">(define raw-bbb-data/monday
;; The raw BigBlueButton data as returned by './download.py URL', where
;; 'download.py' is part of bbb-render.
(local-file "bbb-video-data.monday" "bbb-video-data"
#:recursive? #t))
(define raw-bbb-data/tuesday
(local-file "bbb-video-data.tuesday" "bbb-video-data"
#:recursive? #t))</code></pre><p>No, the raw data is not in the Git repository (it’s too big and contains
personally-identifying information about participants), so this assumes
that there’s a <code>bbb-video-data.monday</code> and a <code>bbb-video-data.tuesday</code> in
the same directory as <code>render-videos.scm</code>.</p><p>For good measure, we define a
<a href="https://gitlab.inria.fr/guix-hpc/website/-/blob/6977da4618814c790e767618da5cf9ec2cab0742/doc/atelier-reproductibilit%C3%A9/render-videos.scm#L243-251"><code><talk></code></a>
data type:</p><pre><code class="language-scheme">(define-record-type <talk>
(talk title speaker start end cam-size data)
talk?
(title talk-title)
(speaker talk-speaker)
(start talk-start) ;start time in seconds
(end talk-end) ;end time
(cam-size talk-webcam-size) ;percentage used for the webcam
(data talk-bbb-data)) ;BigBlueButton data</code></pre><p>… such that we can easily <a href="https://gitlab.inria.fr/guix-hpc/website/-/blob/6977da4618814c790e767618da5cf9ec2cab0742/doc/atelier-reproductibilit%C3%A9/render-videos.scm#L263-288">define
talks</a>,
along with
<a href="https://gitlab.inria.fr/guix-hpc/website/-/blob/6977da4618814c790e767618da5cf9ec2cab0742/doc/atelier-reproductibilit%C3%A9/render-videos.scm#L297-311"><code>talk->video</code></a>,
which takes a talk and return a complete, final video:</p><pre><code class="language-scheme">(define (talk->video talk)
"Given a talk, return a complete video, with opening and closing."
(define file-name
(string-append (canonicalize-string (talk-speaker talk))
".webm"))
(let ((raw (ges->webm (video-ges-project (talk-bbb-data talk)
(talk-start talk)
(talk-end talk)
#:webcam-size
(talk-webcam-size talk))
file-name))
(opening (opening (talk-title talk) (talk-speaker talk))))
(video-with-opening/closing file-name raw
opening closing.webm)))</code></pre><p>The <a href="https://gitlab.inria.fr/guix-hpc/website/-/blob/6977da4618814c790e767618da5cf9ec2cab0742/doc/atelier-reproductibilit%C3%A9/render-videos.scm#L313-319">very last
bit</a>
iterates over the talks and returns a manifest containing all the final
videos. Now we can build the ready-to-be-published videos, all at once:</p><pre><code>$ guix build -m render-videos.scm
[… time passes…]
/gnu/store/…-emmanuel-agullo.webm
/gnu/store/…-francois-rue.webm
…</code></pre><p><a href="https://hpc.guix.info/events/2021/atelier-reproductibilité-environnements/">Voilà!</a></p><p><img src="/static/blog/img/2021-video-tv-screen.png" alt="Image of an old TV screen showing a video opening." /></p><h1>Why all the fuss?</h1><p>OK, maybe you’re thinking “this is just another hackish script to fiddle
with videos”, and that’s right! It’s also worth mentioning another
approach: <a href="https://lang.video/">Racket’s video language</a>, which is
designed to manipulate video abstractions, similar to GES but with a
sweet high-level functional interface.</p><p>But look, this one’s different: it’s
self-contained, it’s reproducible, and it has the right abstraction
level. Self-contained is a big thing; it means you can run it and it
knows what software to deploy, what environment variables to set, and so
on, for each step of the pipeline. Granted, it could be simplified with
appropriate high-level interfaces in Guix. But remember: the
alternative is a makefile (“deployment-unaware”) completed by a <code>README</code>
file giving a vague idea of the dependencies needed. The reproducible
bit is pretty nice too (especially for a workshop <em>on</em> reproducibility).
It also means there’s caching: videos or intermediate byproducts already
in the store don’t need to be recomputed. Last, we have access to a
general-purpose programming language where we can <em>build abstractions</em>,
such as the <code><talk></code> data type, that makes the whole thing more pleasant
to work with and more maintainable.</p><p>Hopefully that’ll inspire you to have a reproducible video pipeline for
your next on-line event, or maybe that’ll inspire you to replace your
old makefile and shelly habits for data processing!</p><p>High-performance computing (HPC) people might be wondering how to go
from here and build “computing-resource-aware” or
“storage-resource-aware” pipelines where each computing step could be
submitted to the job scheduler of an HPC cluster and use distributed
file systems for intermediate results rather than <code>/gnu/store</code>. If
you’re one of these folks, do take a look at how the <a href="https://guixwl.org/">Guix Workflow
Language</a> addresses these issues.</p><h1>Acknowledgments</h1><p>Thanks to Konrad Hinsen for valuable feedback on an earlier draft.</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package manager and
an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects user
freedom</a>.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, AArch64 and POWER9 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2021/getting-bytes-to-disk-more-quickly//Getting bytes to disk more quicklyLudovic Courtès2021-03-26T14:00:00Z2021-03-26T14:00:00Z Let’s face it: functional package managers like Guix provide unequaled
support for reproducibility and transactional upgrades, but the price to
pay is that users often spend a fair amount of time downloading (or
building) packages. Download times are okay on day-to-day use but
they’re a pain point when performing large upgrades or when installing
Guix System for the first time. With that in mind, it should come as no surprise that Michael
Stapelberg’s excellent 2020 Arch Linux Conference
talk
and the installation speed achieved by distri were a great motivation
boost. Michael proposes radical ideas to speed up package installation,
such as downloading and…<p>Let’s face it: functional package managers like Guix provide unequaled
support for reproducibility and transactional upgrades, but the price to
pay is that users often spend a fair amount of time downloading (or
building) packages. Download times are okay on day-to-day use but
they’re a pain point when performing large upgrades or when installing
Guix System for the first time.</p><p>With that in mind, it should come as no surprise that Michael
Stapelberg’s <a href="https://media.ccc.de/v/arch-conf-online-2020-6387-distri-researching-fast-linux-package-management">excellent 2020 Arch Linux Conference
talk</a>
and the installation speed achieved by distri were a great motivation
boost. Michael proposes radical ideas to speed up package installation,
such as downloading and mounting ready-to-use SquashFS images. Not
everything can be transposed to Guix as-is, but it certainly got us
thinking.</p><p>This article dives into improvements made over the last few months that
will be in the upcoming 1.2.1 release, and which are already <a href="https://guix.gnu.org/manual/en/html_node/Upgrading-Guix.html">one <code>guix pull</code>
away</a>;
they all contribute to making <a href="https://guix.gnu.org/manual/en/html_node/Substitutes.html">substitute
download</a> and
installation “faster”. This is an evolution of the existing mechanisms
rather than a revolution, but one that users will surely welcome.</p><h1>Reducing latency</h1><p>One of the first things we notice is latency between subsequent
substitute downloads. It may sound ridiculous, but <code>guix-daemon</code> would
spawn one helper process (running the internal <code>guix substitute</code>
command) for each substitute download. That means that not only would
it create a new process each time, it would also be unable to reuse
connections to the substitute servers. This is particularly noticeable
and wasteful when downloading many substitutes in a row, such as when
installing Guix System for the first time.</p><p>The latency in between subsequent substitute downloads was primarily
this: <code>fork</code> (creating a new process), TCP connection establishment and
TLS connection handshake. When running <code>perf timechart record guix build …</code> (what a wonderful tool <a href="https://perf.wiki.kernel.org/"><code>perf timechart</code></a> is!), we get this Gantt
diagram, which gives an idea of all the time wasted creating these
processes, waiting for them to initialize and connect to the substitute
server, and so on:</p><p><img src="https://guix.gnu.org/static/blog/img/substitute-chart-one-process-per-substitute.png" alt="Gantt diagram without substitute agent." /></p><p>Why was it done this way? Because the daemon, written in C++ and
inherited from Nix, would historically delegate substitution to helper
programs, with a flexible but naive protocol between the daemon and
those helper programs.</p><p>Back in December, we <a href="https://issues.guix.gnu.org/45018">tackled this
issue</a>
(<a href="https://issues.guix.gnu.org/45253">followup</a>): the daemon would now
launch a single <code>guix substitute</code> process and reuse it for subsequent
substitute downloads. In turn, <code>guix substitute</code> would cache
connections so we save time on connection establishment.</p><p>Another observation we made is that <code>guix-daemon</code> implemented
post-processing steps after each download that would contribute to
latency. Once <code>guix substitute</code> had completed a download and extracted
the archive to the store, the daemon would traverse this store item a
couple of times to reset file timestamps/permissions (“<em>metadata
canonicalization</em>”), to deduplicate files, and to verify the integrity
of the whole store item. This is I/O-intensive and particularly
wasteful for store items with many files. Our next step was to delegate
all this work to <code>guix substitute</code>, which allows it to <a href="https://issues.guix.gnu.org/45253">pipeline archive
extraction, integrity checks, deduplication, and metadata
canonicalization</a>. All this happens
as the bytes flow in and no extra traversal is needed.</p><p>The speedup offered by these optimizations depends on several factors,
including the latency of your network, the speed of your CPU and that of
your hard disk, and it grows with the number of substitutes fetched in a
row. What we can say is that even in a “favorable”
situation—low-latency network, fast CPU, fast storage device—it
definitely feels snappier.</p><h1>Increasing bandwidth</h1><p>With the latency issue pretty much solved, the next step was to look at
bandwidth. When we <a href="https://guix.gnu.org/en/blog/2019/substitutes-are-now-available-as-lzip/">introduced lzip-compressed
substitutes</a>
back in 2019, the assumption was that a much higher compression ratio
(compared to gzip), would inevitably translate to faster downloads.
That is true… but only for some bandwidth/CPU power configurations.</p><p>Specifically, it turns out that, for someone with a fast connection,
such as fiber-to-the-home (FTTH), <a href="https://lists.gnu.org/archive/html/guix-devel/2020-12/msg00177.html">downloads of lzip-compressed
substitutes are actually
CPU-bound</a>.
In other words, the limiting factor is the processing time to decompress
those lzip’d archives—not the available bandwidth. Lzip currently
achieves the best compression ratios so far, and that’s great, but
decompression is compute-intensive.</p><p>Ever-increasing bandwidth is what drove the design and implementation of
newer compression methods, such as “Z standard”, colloquially known as
<a href="https://facebook.github.io/zstd/">“zstd”</a>. The decompression speed of
zstd is on the order of 3.5 higher than that of gzip and an order of
magnitude higher than that of lzip; but unlike the venerable
<a href="http://www.oberhumer.com/opensource/lzo">lzo</a>, zstd achieves high
compression ratios: at level 19, the compression ratio of zstd is lower
than that of lzip but in the same ballpark. Guillaume Le Vaillant
<a href="https://lists.gnu.org/archive/html/guix-devel/2021-01/msg00097.html">provided</a>
an insightful comparison of gzip, lzip, and zstd on a plot showing their
decompression speed as a function of the compression ratio:</p><p><img src="https://guix.gnu.org/static/blog/img/decompression-speed-plot.png" alt="Decompression speed vs. compression ratio of gzip, lzip, and zstd." /></p><p>There are several takeaways. First, zstd decompression is always faster
than the alternatives. Second, zstd compresses better than gzip
starting from level 2, so gzip “loses” on both criteria. Last, zstd at
level 19 achieves a compression ratio comparable to lzip level 5 or
6—lzip level 9 remains better in that regard.</p><p>With brand new <a href="https://notabug.org/guile-zstd/guile-zstd">Guile bindings to
zstd</a>, we were able to add
zstd support to <a href="https://guix.gnu.org/manual/en/html_node/Invoking-guix-publish.html"><code>guix publish</code></a>
and <code>guix substitute</code>. But what policy should be adopted for the
official substitute server at <code>ci.guix.gnu.org</code>?</p><p>Our goal is to maximize substitute download speed. In some
configurations, for example on relatively slow connections, fetching
lzip substitutes is not CPU-bound; in those cases, fetching lzip
substitutes remains the fastest option. And of course, there are these
other configurations where lzip decompression is the limiting factor,
and where we’d rather fetch a slightly less compressed zstd substitute
(or even a much less compressed gzip substitutes, sometimes).</p><p>The solution we came up with is a naive but efficient solution:
<a href="https://issues.guix.gnu.org/47137">client-side adaptive compression
selection</a>. When it fetches and
decompresses a substitute, <code>guix substitute</code> monitors its CPU usage as
reported by
<a href="https://www.gnu.org/software/libc/manual/html_node/Processor-Time.html#index-times"><code>times</code></a>.
If the user time to wall-clock time ratio is close to one, that probably
means the substitute fetch and decompression process was CPU-bound; in
that case, choose a compression method that yields faster decompression
next time—assuming the substitute server offers several options.
Conversely, if CPU usage is close to zero, the process is probably
network-bound, so next time we’ll choose a better-compressed option.</p><p>The official server at <code>ci.guix.gnu.org</code> now provides zstd-compressed
substitutes in addition to lzip and gzip. An <a href="https://lists.gnu.org/archive/html/guix-devel/2021-03/msg00333.html">extra
complication</a>
is that we cannot drop gzip compression just yet because pre-1.1.0 Guix
installations understand nothing but gzip. To ease transition, the
server will probably offer substitutes in all three compression formats
for one more year or so.</p><p>There’s a good reason to upgrade your daemon though: you’ll be able to
benefit from those zstd substitutes and if you have high bandwidth,
you’ll quickly see the difference!</p><h1>Grabbing substitutes from your neighbors</h1><p>When it comes to increasing download speeds, another option is to
download from your neighbors rather than from a server far away. This
is particularly useful at the workplace, where machines around you are
likely to already have the store items you’re looking for, or, say, at
Guix gatherings—at least when we can meet physically again…</p><p>The idea had been <a href="https://guix.gnu.org/en/blog/2017/reproducible-builds-a-status-update/">floating
around</a>
for some time as a direct benefit of <a href="https://reproducible-builds.org/">reproducible
builds</a> and co-maintainer Mathieu
Othacehe <a href="https://othacehe.org/substitute-server-discovery.html">recently implemented
it</a>. As Mathieu
explains in his blog post, <code>guix publish</code> can now advertise itself on
the local network using the mDNS/DNS-SD protocol <em>via</em>
<a href="https://avahi.org">Avahi</a>; when <code>guix-daemon</code> is passed the
<code>--discover</code> option, <code>guix substitute</code> automatically discovers local
substitute servers and adds them as the preferred download location.
The Guix System installation image even allows you to enable it to speed
up the installation process:</p><p><img src="https://guix.gnu.org/static/blog/img/installer-substitute-discovery.png" alt="Enabling substitute server discovery when installing Guix System." /></p><p>Again, that only speeds things up if substitute servers use a
compression method with fast decompression, and with either a cache or
fast compression if they compress things on the fly. Zstd comes in
handy for that: Guillaume’s measurements show that <a href="https://lists.gnu.org/archive/html/guix-devel/2021-01/pnglYLfp1DXNC.png">zstd compression is
inexpensive at low compression levels, while achieving higher
compression ratios than
gzip</a>.</p><h1>Going further</h1><p>Put together, these changes have the potential to noticeably improve
user experience. But as I wrote in the introduction, it’s not
revolutionary either—users still have to download all these things.</p><p>The next big step might come from fetching substitutes over
peer-to-peer, content-addressed networks such as
<a href="https://issues.guix.gnu.org/33899">IPFS</a> or
<a href="https://issues.guix.gnu.org/46800#3">GNUnet</a>. Their content-addressed
nature could allow users to download less. The performance
characteristics of these networks is less clear though.</p><p>There is still value in a plain HTTP-based substitute protocol like the
one currently used that is easy to set up, though. In that spirit, an
option would be to “upgrade” the existing substitute protocol to take
advantage of content-addressability. After all, the daemon already
performs file-level deduplication and there’s <a href="https://lists.gnu.org/archive/html/guix-devel/2020-12/msg00258.html">a fair amount of
identical files between subsequent builds of the same
package</a>.
So… <a href="https://lists.gnu.org/archive/html/guix-devel/2021-01/msg00080.html">what if we only downloaded those files not already available
locally?</a>
This idea is appealing but we need to go beyond the prototyping phase to
get a better idea of its viability.</p><p>For completeness, another option currently investigated by the Nix
developers is that of “<em>content-addressed derivations</em>”. While
currently store file names contain a hash of the <em>inputs</em> used to
produce them, the idea of content-addressed derivations is to make it a
content hash; that way, if an input change has no effect on the build
result, the output is the same and nothing needs to be re-downloaded
(this is what Eelco Dolstra described as the <em>intensional model</em> in
Chapter 6 of his <a href="http://nixos.org/~eelco/pubs/phd-thesis.pdf">seminal PhD
thesis</a>). This option is
appealing, also for other reasons, but it’s a fundamental change with
what looks like a <a href="https://github.com/tweag/rfcs/blob/cas-rfc/rfcs/0062-content-addressed-paths.md">high implementation complexity and transition
cost</a>.
We have yet to gauge the pros and cons of following this approach.</p><p>Until then, we hope Guix 1.2.1 will bring your faster substitutes and
happiness!</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package manager and
an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects user
freedom</a>.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2020/securing-updates//Securing updatesLudovic Courtès2020-07-01T17:40:00Z2020-07-01T17:40:00Z Software deployment tools like Guix are in a key position when it comes
to securing the “software supply chain”—taking source code fresh from
repositories and providing users with ready-to-use binaries. We have
been paying attention to several aspects of this problem in Guix:
authentication of pre-built
binaries ,
reproducible
builds ,
bootstrapping , and
security
updates . A couple of weeks ago, we addressed the elephant in the room:
authentication of Guix code itself by guix pull ,
the tool that updates Guix and its package collection. This article
looks at what we set out to address, how we achieved it,…<p>Software deployment tools like Guix are in a key position when it comes
to securing the “software supply chain”—taking source code fresh from
repositories and providing users with ready-to-use binaries. We have
been paying attention to several aspects of this problem in Guix:
<a href="https://guix.gnu.org/manual/en/html_node/Substitute-Authentication.html">authentication of pre-built
binaries</a>,
<a href="https://guix.gnu.org/blog/tags/reproducible-builds/">reproducible
builds</a>,
<a href="https://guix.gnu.org/blog/tags/bootstrapping/">bootstrapping</a>, and
<a href="https://guix.gnu.org/blog/tags/security/">security
updates</a>.</p><p>A couple of weeks ago, we addressed the elephant in the room:
authentication of Guix code itself by <a href="https://guix.gnu.org/manual/en/html_node/Invoking-guix-pull.html"><code>guix pull</code></a>,
the tool that updates Guix and its package collection. This article
looks at what we set out to address, how we achieved it, and how it
compares to existing work in this area.</p><h1>What updates should be protected against</h1><p>The problem of securing distro updates is often viewed through the lens
of binary distributions such as Debian, where the main asset to be
protected are binaries themselves. The functional deployment model that
Guix and Nix implement is very different: conceptually, Guix is a
<em>source distribution</em>, like Gentoo if you will.</p><p>Pre-built binaries are of course available and very useful, but they’re
optional; we call them
<a href="https://guix.gnu.org/manual/en/html_node/Substitutes.html"><em>substitutes</em></a>
because they’re just that: substitutes for local builds. When you do
choose to accept substitutes, they must be signed by one of the keys you
authorized (this has been the case since <a href="https://guix.gnu.org/blog/2014/gnu-guix-06-released/">version 0.6 in
2014</a>).</p><p>Guix consists of source code for the tools as well as all the <a href="https://guix.gnu.org/manual/en/html_node/Defining-Packages.html">package
definitions</a>—the
<em>distro</em>. When users run <code>guix pull</code>, what happens behind the scene is
equivalent to <code>git clone</code> or <code>git pull</code>. There are many ways this can
go wrong. An attacker can trick the user into pulling code from an
alternate repository that contains malicious code or definitions for
backdoored packages. This is made more difficult by the fact that code
is fetched over HTTPS from
<a href="https://git.savannah.gnu.org/cgit/guix.git/">Savannah</a> by default. If
Savannah is compromised (<a href="https://www.fsf.org/blogs/sysadmin/savannah-and-www.gnu.org-downtime">as happened in
2010</a>),
an attacker can push code to the Guix repository, which everyone would
pull. The change might even go unnoticed and remain in the repository
forever. An attacker with access to Savannah can also reset the main
branch to an earlier revision, leading users to install outdated
software with known vulnerabilities—a <em>downgrade attack</em>. These are
the kind of attacks we want to protect against.</p><h1>Authenticating Git checkouts</h1><p>If we take a step back, the problem we’re trying to solve is not
specific to Guix and to software deployment tools: it’s about
<em>authenticating Git checkouts</em>. By that, we mean that when <code>guix pull</code>
obtains code from Git, it should be able to tell that all the commits it
fetched were pushed by authorized developers of the project. We’re
really looking at individual commits, not tags, because users can choose
to pull arbitrary points in the commit history of Guix and third-party
<a href="https://guix.gnu.org/manual/en/html_node/Channels.html">channels</a>.</p><p>Checkout authentication requires <a href="https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work">cryptographically signed
commits</a>.
By signing a commit, a Guix developer asserts that they are the one who
made the commit; they may be its author, or they may be the person who
applied somebody else’s changes after review. It also requires a notion
of authorization: we don’t simply want commits to have a valid
signature, we want them to be signed by an authorized key. The set of
authorized keys changes over time as people join and leave the project.</p><p>To implement that, we came up with the following mechanism and rule:</p><ol><li>The repository contains a <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/.guix-authorizations"><code>.guix-authorizations</code>
file</a>
that lists the OpenPGP key fingerprints of authorized committers.</li><li>A commit is considered authentic if and only if it is signed by one
of the keys listed in the <code>.guix-authorizations</code> file of each of
its parents. This is the <em>authorization invariant</em>.</li></ol><p>(Remember that Git commits form a directed acyclic graph (DAG) where
each commit can have zero or more parents; merge commits have two parent
commits, for instance. Do not miss <a href="https://eagain.net/articles/git-for-computer-scientists/"><em>Git for Computer
Scientists</em></a>
for a pedagogical overview!)</p><p>Let’s take an example to illustrate. In the figure below, each box is a
commit, and each arrow is a parent relationship:</p><p><img src="https://guix.gnu.org/static/blog/img/commit-graph.svg" alt="Example commit graph." /></p><p>This figure shows two lines of development: the orange line may be the
main development branch, while the purple line may correspond to a
feature branch that was eventually merged in commit <em>F</em>. <em>F</em> is a merge
commit, so it has two parents: <em>D</em> and <em>E</em>.</p><p>Labels next to boxes show who’s in <code>.guix-authorizations</code>: for commit A,
only Alice is an authorized committer, and for all the other commits,
both Bob and Alice are authorized committers. For each commit, we see
that the authorization invariant holds; for example:</p><ul><li>commit <em>B</em> was made by Alice, who was the only authorized committer
in its parent, commit <em>A</em>;</li><li>commit <em>C</em> was made by Bob, who was among the authorized committers
as of commit <em>B</em>;</li><li>commit <em>F</em> was made by Alice, who was among the authorized
committers of both parents, commits <em>D</em> and <em>E</em>.</li></ul><p>The authorization invariant has the nice property that it’s simple to
state, and it’s simple to check and enforce. This is what <code>guix pull</code>
implements. If your current Guix, <a href="https://guix.gnu.org/manual/en/html_node/Invoking-guix-describe.html">as returned by <code>guix describe</code></a>,
is at commit <em>A</em> and you want to pull to commit <em>F</em>, <code>guix pull</code>
traverses all these commits and checks the authorization invariant.</p><p>Once a commit has been authenticated, all the commits in its <a href="https://en.wikipedia.org/wiki/Transitive_closure#In_graph_theory">transitive
closure</a>
are known to be already authenticated. <code>guix pull</code> keeps a local cache
of the commits it has previously authenticated, which allows it to
traverse only new commits. For instance, if you’re at commit <em>F</em> and
later update to a descendant of <em>F</em>, authentication starts at <em>F</em>.</p><p>Since <code>.guix-authorizations</code> is a regular file under version control,
granting or revoking commit authorization does not require special
support. In the example above, commit <em>B</em> is an authorized commit by
Alice that adds Bob’s key to <code>.guix-authorizations</code>. Revocation is
similar: any authorized committer can remove entries from
<code>.guix-authorizations</code>. Key rotation can be handled similarly: a
committer can remove their former key and add their new key in a single
commit, signed by the former key.</p><p>The authorization invariant satisfies our needs for Guix. It has one
downside: it prevents pull-request-style workflows. Indeed, merging the
branch of a contributor not listed in <code>.guix-authorizations</code> would break
the authorization invariant. It’s a good tradeoff for Guix because our
workflow relies on <a href="https://lwn.net/Articles/702177/">patches carved into stone
tablets</a> (<a href="https://issues.guix.gnu.org/">patch
tracker</a>), but it’s not suitable for every
project out there.</p><h1>Bootstrapping</h1><p>The attentive reader may have noticed that something’s missing from the
explanation above: what do we do about commit <em>A</em> in the example above?
In other words, which commit do we pick as the first one where we
can start verifying the authorization invariant?</p><p>We solve this bootstrapping issue by defining <em>channel introductions</em>.
Previously, one would identify a channel simply by its URL. Now, when
introducing a channel to users, one needs to provide an additional piece
of information: the first commit where the authorization invariant
holds, and the fingerprint of the OpenPGP key used to sign that commit
(it’s not strictly necessary but provides an additional check).
Consider this commit graph:</p><p><img src="https://guix.gnu.org/static/blog/img/commit-graph-intro.svg" alt="Example commit graph with introduction." /></p><p>On this figure, <em>B</em> is the introduction commit. Its ancestors, such as
<em>A</em> are considered authentic. To authenticate, <em>C</em>, <em>D</em>, <em>E</em>, and <em>F</em>,
we check the authorization invariant.</p><p>As always when it comes to establishing trust, distributing channel
introductions is very sensitive. The introduction of the official
<code>guix</code> channel is <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/channels.scm?id=d2fde340adf197cc42bc4e0187deaf3a7bd3968d#n155">built into
Guix</a>.
Users obtain it when they install Guix the first time; hopefully they
verify the signature on the Guix tarball or ISO image, <a href="https://guix.gnu.org/manual/en/html_node/Binary-Installation.html">as noted in the
installation
instructions</a>,
which reduces chances of getting the “wrong” Guix, but it is still very much
<a href="https://en.wikipedia.org/wiki/Trust_on_first_use">trust-on-first-use</a>
(TOFU).</p><p>For signed third-party channels, users have to provide the channel’s
introduction in their <code>channels.scm</code> file, like so:</p><pre><code class="language-scheme">(channel
(name 'my-channel)
(url "https://example.org/my-channel.git")
(introduction
(make-channel-introduction
"6f0d8cc0d88abb59c324b2990bfee2876016bb86"
(openpgp-fingerprint
"CABB A931 C0FF EEC6 900D 0CFB 090B 1199 3D9A EBB5"))))</code></pre><p>The <code>guix describe</code> command now prints the introduction if there’s one.
That way, one can share their channel configuration, including
introductions, without having to be an expert.</p><p>Channel introductions also solve another problem: forks. Respecting the
authorization invariant “forever” would effectively prevent
“unauthorized” forks—forks made by someone who’s not in <code>.guix-authorizations</code>.
Someone publishing a fork simply needs to emit a new introduction
for their fork, pointing to a different starting commit.</p><p>Last, channel introductions give a <em>point of reference</em>: if an attacker
manipulates branch heads on Savannah to have them point to unrelated
commits (such as commits on an orphan branch that do not share any
history with the “official” branches), authentication will necessarily
fail as it stumbles upon the first unauthorized commit made by the
attacker. In the figure above, the red branch with commits <em>G</em> and <em>H</em>
cannot be authenticated because it starts from <em>A</em>, which lacks
<code>.guix-authorizations</code> and thus fails the authorization invariant.</p><p>That’s all for authentication! I’m glad you read this far. At this
point you can take a break or continue with the next section on how
<code>guix pull</code> prevents downgrade attacks.</p><h1>Downgrade attacks</h1><p>An important threat for software deployment tools is <em>downgrade</em> or
<em>roll-back</em> attacks. The attack consists in tricking users into
installing older, known-vulnerable software packages, which in turn may
offer new ways to break into their system. This is not strictly related
to the authentication issue we’ve been discussing, except that it’s
another important issue in this area that we took the opportunity to
address.</p><p>Guix saves provenance info for itself: <code>guix describe</code> prints that
information, essentially the Git commits of the channels used during
<code>git pull</code>:</p><pre><code>$ guix describe
Generation 149 Jun 17 2020 20:00:14 (current)
guix 8b1f7c0
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: 8b1f7c03d239ca703b56f2a6e5f228c79bc1857e</code></pre><p>Thus, <code>guix pull</code>, once it has retrieved the latest commit of the
selected branch, can verify that it is doing a <em>fast-forward update</em> in
Git parlance—just like <code>git pull</code> does, but compared to the
previously-deployed Guix. A fast-forward update is when the new commit
is a descendant of the current commit. Going back to the figure above,
going from commit <em>A</em> to commit <em>F</em> is a fast-forward update, but going
from <em>F</em> to <em>A</em> or from <em>D</em> to <em>E</em> is not.</p><p>Not doing a fast-forward update would mean that the user is deploying an
older version of the Guix currently used, or deploying an unrelated
version from another branch. In both cases, the user is at risk of
ending up installing older, vulnerable software.</p><p>By default <code>guix pull</code> now errors out on non-fast-forward updates,
thereby protecting from roll-backs. Users <a href="https://issues.guix.gnu.org/41882">who understand the
risks</a> can override that by passing
<code>--allow-downgrades</code>.</p><p>Authentication and roll-back prevention allow users to safely refer to
mirrors of the Git repository. If <code>git.savannah.gnu.org</code> is down, one
can still update by fetching from a mirror, for instance with:</p><pre><code>guix pull --url=https://github.com/guix-mirror/guix</code></pre><p>If the repository at this URL is behind what the user already deployed,
or if it’s not a genuine mirror, <code>guix pull</code> will abort. In other
cases, it will proceed.</p><p>Unfortunately, there is no way to answer the general question “<em>is</em> X
<em>the latest commit of branch</em> B <em>?</em>”. Rollback detection prevents just
that, rollbacks, but there’s no mechanism in place to tell whether a
given mirror is stale. To mitigate that, channel authors can specify,
in the repository, the channel’s <em>primary URL</em>. This piece of
information lives in the <code>.guix-channel</code> file, in the repository, so
it’s authenticated. <code>guix pull</code> uses it to print a warning when the
user pulls from a mirror:</p><pre><code>$ guix pull --url=https://github.com/guix-mirror/guix
Updating channel 'guix' from Git repository at 'https://github.com/guix-mirror/guix'...
Authenticating channel 'guix', commits 9edb3f6 to 3e51f9e (44 new commits)...
guix pull: warning: pulled channel 'guix' from a mirror of https://git.savannah.gnu.org/git/guix.git, which might be stale
Building from this channel:
guix https://github.com/guix-mirror/guix 3e51f9e
…</code></pre><p>So far we talked about mechanics in a rather abstract way. That might
satisfy the graph theorist or the Git geek in you, but if you’re up for
a quick tour of the implementation, the next section is for you!</p><h1>A long process</h1><p>We’re kinda celebrating these days, but the <a href="https://issues.guix.gnu.org/22883">initial bug
report</a> was opened… in 2016. One of
the reasons was that we were hoping the general problem was solved
already and we’d “just” have to adapt what others had done. As for the
actual design: you would think it can be implemented in ten lines of
shell script invoking <code>gpgv</code> and <code>git</code>. Perhaps that’s a possibility,
but the resulting performance would be problematic—keep in mind that
users may routinely have to authenticate hundreds of commits. So we
took a long road, but the end result is worth it. Let’s recap.</p><p>Back in April 2016, committers <a href="https://issues.guix.gnu.org/22883#4">started signing
commits</a>, with a <a href="https://issues.guix.gnu.org/22883#36">server-side hook
prohibiting unsigned commits</a>. In
July 2016, we had <a href="https://issues.guix.gnu.org/22883#33">proof-of-concept libgit2
bindings</a> with the primitives
needed to verify signatures on commits, passing them to <code>gpgv</code>; later
<a href="https://gitlab.com/guile-git/guile-git/">Guile-Git</a> was born, providing
good coverage of the libgit2 interface. Then there was a two-year
hiatus during which no code was produced in that area.</p><p>Everything went faster starting from December 2019. Progress was
incremental and may have been hard to follow, even for die-hard Guix
hackers, so here are the major milestones:</p><ul><li>In December 2019, a first <a href="https://issues.guix.gnu.org/22883#48">authentication program for use by Guix
developers</a> landed; it could
be run from a checkout with the <code>make authenticate</code> command. It
would use Guile-Git but call out to <code>gpgv</code> for signature
verification, which made it rather slow. The list of authorized
keys was hard-coded.</li><li>In April 2020, we had <a href="https://issues.guix.gnu.org/22883#61">an implementation of OpenPGP for signature
verification purposes only</a>
available as <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/openpgp.scm"><code>(guix openpgp)</code></a>.
The code is based on Göran Weinholt’s pure-Scheme
<a href="https://github.com/weinholt/industria/">Industria</a> library, with
the addition of a few features and using
<a href="https://notabug.org/cwebber/guile-gcrypt">Guile-Gcrypt</a> for faster
crypto. That led to a tenfold speedup compared to invoking <code>gpgv</code>,
which is primarily due to the fact that our code <a href="https://issues.guix.gnu.org/22883#62">foregoes OpenPGP
bells and whistles</a> and
focuses on “just” signature verification. Notably, it ignores key
expiration and revocation.</li><li>In May, <a href="https://issues.guix.gnu.org/22883#64"><code>.guix-authorizations</code> support was
added</a>, superseding the
hard-coded list of authorized keys. The OpenPGP keyring could now
be loaded straight from a <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/?h=keyring">Git branch containing all the OpenPGP
keys ever
used</a>.</li><li>Early June, the authentication code was <a href="https://issues.guix.gnu.org/41653">moved to its own
module</a>, <code>(guix git-authenticate)</code>, with a test suite.</li><li>Soon after, Git authentication was <a href="https://issues.guix.gnu.org/41767">integrated in
channels</a>: <code>guix pull</code> would now
authenticate the <code>guix</code> channel, <a href="https://issues.guix.gnu.org/22883">closing the 4-year old
mega-issue</a>.</li><li>Just today, we added the final bits, <a href="https://issues.guix.gnu.org/42048">allowing channel authors to
benefit from the feature</a>.</li></ul><p>Whether you’re a channel author or a user, the feature is now <a href="https://guix.gnu.org/manual/devel/en/html_node/Channels.html">fully
documented in the
manual</a>,
and we’d love to get your feedback!</p><h1>SHA-1</h1><p>We can’t really discuss Git commit signing without mentioning
<a href="https://en.wikipedia.org/wiki/SHA-1">SHA-1</a>. The venerable
crytographic hash function is approaching end of life, as evidenced by
<a href="https://shattered.io/">recent</a>
<a href="https://sha-mbles.github.io/">breakthroughs</a>. Signing a Git commit
boils down to signing a SHA-1 hash, because all objects in
the Git store are identified by their SHA-1 hash.</p><p>Git now relies on a <a href="https://www.usenix.org/system/files/conference/usenixsecurity17/sec17-stevens.pdf">collision attack detection
library</a>
to mitigate practical attacks. Furthermore, the Git
project is planning a <a href="https://git-scm.com/docs/hash-function-transition/">hash function
transition</a> to
address the problem.</p><p>Some projects such as Bitcoin Core choose to not rely on SHA-1 at all.
Instead, for the commits they sign, they include in the commit log the
SHA512 hash of the tree, which the <a href="https://github.com/bitcoin/bitcoin/tree/master/contrib/verify-commits">verification scripts
check</a>.</p><p>Computing a tree hash <em>for each commit</em> in Guix would probably be
prohibitively costly. For now, for lack of a better solution, we rely
on Git’s collision attack detection and look forward to a hash function
transition.</p><p>As for SHA-1 in an OpenPGP context: our authentication code <a href="https://issues.guix.gnu.org/41787">rejects
SHA-1 OpenPGP signatures</a>, as
recommended.</p><h1>Related work</h1><p>A lot of work has gone into securing the software supply chain, often in
the context of binary distros, sometimes in a more general context; more
recent work also looks into Git authentication and related issues.
This section attempts to summarize how Guix relates to similar work that
we’re aware of in these two areas. More detailed discussions can be
found in the <a href="https://issues.guix.gnu.org/22883">issue tracker</a>.</p><p><a href="https://theupdateframework.io/">The Update Framework</a> (TUF) is a
reference for secure update systems, with <a href="https://github.com/theupdateframework/specification/blob/master/tuf-spec.md#the-update-framework-specification">a well-structured
spec</a>
and a number of
<a href="https://github.com/theupdateframework/specification/blob/master/tuf-spec.md#the-update-framework-specification">implementations</a>.
TUF is a great source of inspiration to think about this problem space.
Many of its goals are shared by Guix. Not all the attacks it aims to
protect against (Section 1.5.2 of the spec) are addressed by what’s
presented in this post: <em>indefinite freeze attacks</em>,
where updates never become available, are not addressed <em>per se</em> (though
easily observable), and <em>slow retrieval attacks</em> aren’t addressed
either. The notion of <em>role</em> is also something currently missing from
the Guix authentication model, where any authorized committer can touch
any files, though the model and <code>.guix-authorizations</code> format leave room
for such an extension.</p><p>However, both in its goals and system descriptions, TUF is biased
towards systems that distribute binaries as plain files with associated
meta-data. That creates a fundamental impedance mismatch. As an
example, attacks such as <em>fast-forward attacks</em> or <em>mix-and-match
attacks</em> don’t apply in the context of Guix; likewise, the <em>repository</em>
depicted in Section 3 of the spec has little in common with a Git
repository.</p><p>Developers of OPAM, the OCaml package manager, <a href="http://opam.ocaml.org/blog/Signing-the-opam-repository/">adapted TUF for use with
their Git-based package
repository</a>,
later updated to write <a href="https://github.com/hannesm/conex">Conex</a>, a
separate tool to authenticate OPAM repositories. OPAM is interesting
because like Guix it’s a source distro and its <a href="https://github.com/ocaml/opam-repository">package
repository</a> is a Git
repository containing “build recipe”. To date, it appears that <code>opam update</code> itself does not authenticate repositories though; it’s up to
users or developer to run Conex.</p><p>Another very insightful piece of work is the 2016 paper <a href="https://www.usenix.org/system/files/conference/usenixsecurity16/sec16_paper_torres-arias.pdf"><em>On omitting
commits and committing
omissions</em></a>.
The paper focuses on the impact of malicious modifications to Git
repository meta-data. An attacker with access to the repository can
modify, for instance, branch references, to cause a rollback attack or a
“teleport” attack, causing users to pull an older commit or an unrelated
commit. As written above, <code>guix pull</code> would detect such attacks.
However, <code>guix pull</code> would fail to detect cases where metadata
modification does not yield a rollback or teleport, yet gives users a
different view than the intended one—for instance, a user is directed to
an authentic but different branch rather than the intended one. The
“secure push” operation and the associated <em>reference state log</em> (RSL)
the authors propose would be an improvement.</p><h1>Wrap-up and outlook</h1><p>Guix now has a mechanism that allows it to authenticate updates. If
you’ve run <code>guix pull</code> recently, perhaps you’ve noticed additional
output and a progress bar as new commits are being authenticated. Apart
from that, the switch has been completely transparent. The
authentication mechanism is built around the commit graph of Git; in
fact, it’s a mechanism to <em>authenticate Git checkouts</em> and in that sense
it is not tied to Guix and its application domain. It is available not
only for the main <code>guix</code> channel, but also for third-party channels.</p><p>To bootstrap trust, we added the notion of <em>channel introductions</em>.
These are now visible in the user interface, in particular in the output
of <code>guix describe</code> and in the configuration file of <code>guix pull</code> and
<code>guix time-machine</code>. While channel configuration remains a few lines of
code that users typically paste, this extra bit of configuration might
be intimidating. It certainly gives an incentive to provide a
command-line interface to manage the user’s list of channels: <code>guix channel add</code>, etc.</p><p>The solution here is built around the assumption that Guix is
fundamentally a source-based distribution, and is thus completely
orthogonal to the <a href="https://guix.gnu.org/manual/en/html_node/Substitute-Server-Authorization.html">public key infrastructure (PKI) Guix uses for the
signature of
substitutes</a>.
Yet, the substitute PKI could probably benefit from the fact that we now
have a secure update mechanism for the Guix source code: since <code>guix pull</code> can securely retrieve a new substitute signing key, perhaps it
could somehow handle substitute signing key revocation and delegation
automatically? Related to that, channels could perhaps advertise a
substitute URL and its signing key, possibly allowing users to register
those when they first pull from the channel. All this requires more
thought, but it looks like there are new opportunities here.</p><p>Until then, if you’re a user or a channel author, we’d love to hear from
you! We’ve already gotten feedback that these new mechanisms <a href="https://issues.guix.gnu.org/41882">broke
someone’s workflow</a>; hopefully it
didn’t break yours, but either way your input is important in improving
the system. If you’re into security and think this design is terrible
or awesome, please do provide feedback.</p><p>It’s a long and article describing a long ride on a path we discovered
as we went, and it felt like an important milestone to share!</p><h1>Acknowledgments</h1><p>Thanks to everyone who provided feedback, ideas, or carried out code
review during this long process, notably (in no particular order):
Christopher Lemmer Webber, Leo Famulari, David Thompson, Mike Gerwitz,
Ricardo Wurmus, Werner Koch, Justus Winter, Vagrant Cascadian, Maxim
Cournoyer, Simon Tournier, John Soo, and Jakub Kądziołka. Thanks also
to janneke, Ricardo, Marius, and Simon for reviewing an earlier draft of
this post.</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package
manager and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects
user
freedom</a>.
Guix can be used on top of any system running the kernel Linux, or it
can be used as a standalone operating system distribution for i686,
x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2020/gnu-shepherd-user-services//GNU Shepherd user servicesEfraim Flashner2020-05-17T20:00:00Z2020-05-17T20:00:00Z One of the things which sets Guix apart from other GNU/Linux
distributions is that it uses GNU
Shepherd instead of the now
ubiquitous systemd. A side effect of this is that user systemd units do
not work on Guix System. Love, hate or extremely ambivalent toward
systemd, this means that users cannot rely on already written systemd
unit files for their regular user-level services. There are a couple of benefits to using GNU Shepherd, and not all of
them are due to it already being installed on Guix. Becoming comfortable
with using Shepherd and understanding how to write and edit Shepherd
service…<p>One of the things which sets Guix apart from other GNU/Linux
distributions is that it uses <a href="https://www.gnu.org/software/shepherd/">GNU
Shepherd</a> instead of the now
ubiquitous systemd. A side effect of this is that user systemd units do
not work on Guix System. Love, hate or extremely ambivalent toward
systemd, this means that users cannot rely on already written systemd
unit files for their regular user-level services.</p><p>There are a couple of benefits to using GNU Shepherd, and not all of
them are due to it already being installed on Guix. Becoming comfortable
with using Shepherd and understanding how to write and edit Shepherd
service configurations makes the transition from other GNU/Linux
distributions to Guix System easier. More complex services with their
own logic tree, using the full power of <a href="https://www.gnu.org/software/guile/">GNU
Guile</a>, are also possible. This
means you can have one service that behaves differently if it's running
on a different system or architecture without needing to call out to
shell scripts or using minimally different service definitions.</p><p>The GNU Shepherd manual
<a href="https://www.gnu.org/software/shepherd/manual/html_node/Jump-Start.html#index-Configuration-file">suggests</a>
putting all the services inside a
monolithic <code>init.scm</code> file, located by default at
<code>$XDG_CONFIG_DIR/shepherd/init.scm</code>. While this does make it easy to keep
everything in one place, it does create one glaring issue: any changes
to the file mean that all the services need to be stopped and restarted
in order for any changes to take place.</p><p>Luckily there's a nice function called <code>scandir</code> hiding in <a href="https://www.gnu.org/software/guile/manual/html_node/File-Tree-Walk.html#index-scandir"><code>ice-9 ftw</code></a>
which returns a list of all files in a specified directory (with options
for narrowing down the list or sorting it). This means that our <code>init.scm</code>
can contain a minimum of code and all actual services can be loaded from
individual files.</p><p>First the minimal <code>init.scm</code>:</p><pre><code class="language-scheme">(use-modules (shepherd service)
((ice-9 ftw) #:select (scandir)))
;; Load all the files in the directory 'init.d' with a suffix '.scm'.
(for-each
(lambda (file)
(load (string-append "init.d/" file)))
(scandir (string-append (dirname (current-filename)) "/init.d")
(lambda (file)
(string-suffix? ".scm" file))))
;; Send shepherd into the background
(action 'shepherd 'daemonize)</code></pre><p>Let's take a sample service for running syncthing, as defined in
<code>$XDG_CONFIG_DIR/shepherd/init.d/syncthing.scm</code>:</p><pre><code class="language-scheme">(define syncthing
(make <service>
#:provides '(syncthing)
#:docstring "Run `syncthing' without calling the browser"
#:start (make-forkexec-constructor
'("syncthing" "-no-browser")
#:log-file (string-append (getenv "HOME")
"/log/syncthing.log"))
#:stop (make-kill-destructor)
#:respawn? #t))
(register-services syncthing)
(start syncthing)</code></pre><p>As with any other shepherd service it is defined and registered, and in
this case it will start automatically. When the file is loaded by
shepherd after being discovered by scandir everything works exactly as
though the service definition were located directly inside the <code>init.scm</code>.</p><p>Now lets make a change. Since syncthing already has a <code>-logfile</code> flag and
it has built-in log rotation that sounds better than using shepherd's
<code>#:log-file</code> option. First we'll make our changes to the service:</p><pre><code class="language-scheme">(define syncthing
(make <service>
#:provides '(syncthing)
#:docstring "Run `syncthing' without calling the browser"
#:start (make-forkexec-constructor
'("syncthing" "-no-browser"
"-logflags=3" ; prefix with date & time
"-logfile=/home/user/log/syncthing.log"))
#:stop (make-kill-destructor)
#:respawn? #t))
(register-services syncthing)
(start syncthing)</code></pre><p>Now we stop syncthing:</p><pre><code class="language-sh">$ herd stop syncthing</code></pre><p>And we load the new service:</p><pre><code class="language-sh">$ herd load root ~/.config/shepherd/init.d/syncthing.scm</code></pre><p>This allows for quickly iterating on services without needing to stop
all the services! Let's take a look at another service:</p><pre><code class="language-scheme">(define fccache
(make <service>
#:provides '(fccache)
#:docstring "Run 'fc-cache -frv'"
#:start (make-forkexec-constructor
'("guix" "environment" "--ad-hoc" "fontconfig" "--"
"fc-cache" "-frv")
#:log-file (string-append (getenv "HOME")
"/log/fccache.log"))
#:one-shot? #t))
(register-services fccache)</code></pre><p>In this example I want to refresh my font cache but I don't want to
actually install fontconfig either system-wide or in my profile.</p><pre><code class="language-sh">$ which fc-cache
which: no fc-cache in (/home/user/.config/guix/current/bin:/home/user/.guix-profile/bin:/home/user/.guix-profile/sbin:/run/setuid-programs:/run/current-system/profile/bin:/run/current-system/profile/sbin)
$ herd start fccache
Service fccache has been started.</code></pre><p>Of course we can import other modules and leverage the code already
written there. In this case, instead of using the string "guix
environment --ad-hoc fontutils -- fc-cache -frv" let's use the <code>guix environment</code> function already available in <code>guix scripts environment</code>:</p><pre><code class="language-scheme">(use-modules (guix scripts environment))
(define fccache
(make <service>
#:provides '(fccache)
#:docstring "Run 'fc-cache -frv'"
#:start (lambda () ; Don't run immediately when registered!
(guix-environment "--ad-hoc" "fontconfig" "--" "fc-cache" "-frv"))
#:one-shot? #t))
(register-services fccache)</code></pre><pre><code class="language-sh">$ herd load root ~/.config/shepherd/init.d/fccache.scm
Loading /home/user/.config/shepherd/init.d/fccache.scm.
$ herd start fccache
/gnu/store/hbqlzgd8hcf6ndcmx7q7miqrsxb4dmkk-gs-fonts-8.11/share/fonts: caching, new cache contents: 0 fonts, 1 dirs
/gnu/store/hbqlzgd8hcf6ndcmx7q7miqrsxb4dmkk-gs-fonts-8.11/share/fonts/type1: caching, new cache contents: 0 fonts, 1 dirs
/gnu/store/hbqlzgd8hcf6ndcmx7q7miqrsxb4dmkk-gs-fonts-8.11/share/fonts/type1/ghostscript: caching, new cache contents: 35 fonts, 0 dirs
/home/user/.guix-profile/share/fonts: caching, new cache contents: 0 fonts, 7 dirs
/home/user/.guix-profile/share/fonts/opentype: caching, new cache contents: 8 fonts, 0 dirs
/home/user/.guix-profile/share/fonts/otf: caching, new cache contents: 12 fonts, 0 dirs
/home/user/.guix-profile/share/fonts/terminus: caching, new cache contents: 18 fonts, 0 dirs
/home/user/.guix-profile/share/fonts/truetype: caching, new cache contents: 58 fonts, 0 dirs
/home/user/.guix-profile/share/fonts/ttf: caching, new cache contents: 12 fonts, 0 dirs
/home/user/.guix-profile/share/fonts/type1: caching, new cache contents: 0 fonts, 1 dirs
/home/user/.guix-profile/share/fonts/type1/ghostscript: caching, new cache contents: 35 fonts, 0 dirs
/home/user/.guix-profile/share/fonts/woff: caching, new cache contents: 1 fonts, 0 dirs
/run/current-system/profile/share/fonts: skipping, no such directory
/home/user/.local/share/fonts: skipping, no such directory
/home/user/.fonts: skipping, no such directory
/gnu/store/hbqlzgd8hcf6ndcmx7q7miqrsxb4dmkk-gs-fonts-8.11/share/fonts/type1: skipping, looped directory detected
/home/user/.guix-profile/share/fonts/opentype: skipping, looped directory detected
/home/user/.guix-profile/share/fonts/otf: skipping, looped directory detected
/home/user/.guix-profile/share/fonts/terminus: skipping, looped directory detected
/home/user/.guix-profile/share/fonts/truetype: skipping, looped directory detected
/home/user/.guix-profile/share/fonts/ttf: skipping, looped directory detected
/home/user/.guix-profile/share/fonts/type1: skipping, looped directory detected
/home/user/.guix-profile/share/fonts/woff: skipping, looped directory detected
/gnu/store/hbqlzgd8hcf6ndcmx7q7miqrsxb4dmkk-gs-fonts-8.11/share/fonts/type1/ghostscript: skipping, looped directory detected
/home/user/.guix-profile/share/fonts/type1/ghostscript: skipping, looped directory detected
/var/cache/fontconfig: not cleaning unwritable cache directory
/home/user/.cache/fontconfig: cleaning cache directory
/home/user/.fontconfig: not cleaning non-existent cache directory
fc-cache: succeeded
herd: exception caught while executing 'start' on service 'fccache':
Throw to key `quit' with args `(0)'.</code></pre><p>The problem with this approach is that <code>guix-environment</code> returns the
<a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/scripts/environment.scm?h=v1.1.0#n771">exit
code</a>
of the programs it calls and <code>#:start</code> expects a
<a href="https://www.gnu.org/software/shepherd/manual/html_node/Slots-of-services.html#index-Starting-a-service-1"><code>constructor</code></a>
to return <code>#t</code> or <code>#f</code> so there's some work to be done here.</p><p>This was just a quick peek into what's possible with GNU Shepherd when
run as a user. Next time we'll take a look at integrating
<a href="https://www.gnu.org/software/mcron/">mcron</a> to replicate some of
systemd's timer functionality.</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package
manager and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects
user
freedom</a>.
Guix can be used on top of any system running the kernel Linux, or it
can be used as a standalone operating system distribution for i686,
x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2020/grafts-continued//Grafts, continuedLudovic Courtès2020-05-06T15:00:00Z2020-05-06T15:00:00Z Guix includes a mechanism called grafts that allows us to provide
users with security
updates
in a timely fashion, even for core packages deep down in the dependency
graph. Most users value the benefits of grafts, but newcomers were also
unavoidably struck by what turned out to be the undesirable side effect
of our graft implementation on user experience. This had been a
well-known problem for a while, but 1.1.0 finally addressed these
issues . This article recaps how grafts are implemented, what problems that
caused, and how we solved it. It’s a deep dive into core Guix,…<p>Guix includes a mechanism called <em>grafts</em> that allows us to provide
users with <a href="https://guix.gnu.org/manual/en/html_node/Security-Updates.html">security
updates</a>
in a timely fashion, even for core packages deep down in the dependency
graph. Most users value the benefits of grafts, but newcomers were also
unavoidably struck by what turned out to be the undesirable side effect
of our graft implementation on user experience. This had been a
well-known problem for a while, but <a href="https://guix.gnu.org/blog/2020/gnu-guix-1.1.0-released/">1.1.0 finally addressed these
issues</a>.</p><p>This article recaps how grafts are implemented, what problems that
caused, and how we solved it. It’s a deep dive into core Guix, and I
hope it’ll be insightful to all and intriguing to the functional
programming geeks among us!</p><h1>What’s this “graft” thing anyway?</h1><p>Grafts were introduced in the early days of Guix to <a href="https://guix.gnu.org/blog/2016/timely-delivery-of-security-updates/">address probably
the main practical shortcomings of functional software
deployment</a>.
In a nutshell, functional deployment as implemented by Nix and Guix
means that, when a package changes, everything that depends on it must
be rebuilt (or re-downloaded). To deploy a security fix in the C
library or in Bash, you would thus need to rebuild everything. Even
with a huge build farm, that can significantly delay the deployment of
fixes; users would find themselves either rebuilding things locally or,
at best, re-downloading binaries of everything.</p><p>To address this, Guix developers can instead specify a <em>replacement</em> in
a <a href="https://guix.gnu.org/manual/en/html_node/Defining-Packages.html">package
definition</a>.
If we have a bug-fix for, say, libc, developers would (1) define a
package for the fixed libc, and (2) add a <code>replacement</code> field in the
original libc package pointing to that fixed package. The effect is
that <em>only the bug-fix libc needs to be built</em>. When building a
package, the bug-fix libc is automatically <em>grafted onto that package</em>,
such that the resulting package refers to the bug-fix libc. <a href="https://guix.gnu.org/manual/en/html_node/Security-Updates.html">See the
manual</a>
for more.</p><p>When “lowering” a high-level <a href="https://guix.gnu.org/manual/en/html_node/Defining-Packages.html">package
definition</a>
to a low-level
<a href="https://guix.gnu.org/manual/en/html_node/Derivations.html"><em>derivation</em></a>,
Guix traverses the package dependency graph and identifies a set of
potentially applicable grafts. Why “potentially applicable”? Consider
this scenario: assume <code>perl</code> has a <code>replacement</code>; <code>coreutils</code> has a
dependency on <code>perl</code>, but it’s a build-time dependency: <code>coreutils</code> does
not depend on <code>perl</code> at run time. Thus, <code>coreutils</code> can be used as is,
there is no need to graft it.</p><p>But how do we know whether a dependency is a built-time-only dependency?
The <a href="https://guix.gnu.org/manual/en/html_node/package-Reference.html#index-package"><code>native-inputs</code>
field</a>
of a package usually lists build-time dependencies, but it’s more of a
hint. Ultimately, the set of run-time dependencies, which we call the
<em>references</em>, is the subset of the build-time dependencies that the
garbage collector (GC) in the build daemon finds <em>in the build
result</em>—Section 5.5.1 of <a href="http://nixos.org/~eelco/pubs/phd-thesis.pdf">Eelco Dolstra’s PhD
thesis</a> describes how the
GC
scans for references. In our example, we first have to actually build
<code>coreutils</code> before we can tell whether it depends on <code>perl</code> at
run time.</p><p>Guix arranges to graft only when necessary. In this example, <code>guix build coreutils</code> would return the same as <code>guix build coreutils --no-grafts</code>. Conversely, since <code>inkscape</code> has a run-time dependency on
<code>perl</code>, <code>guix build inkscape</code> returns a derivation that grafts the
<code>perl</code> replacement onto the original <code>inkscape</code> build result, the one
returned by <code>guix build inkscape --no-grafts</code>. The (simplified)
dependency graph of the derivation for the grafted <code>inkscape</code> looks like
this:</p><p><img src="https://guix.gnu.org/static/blog/img/inkscape-graft.svg" alt="Dependency graph of the graft derivation of Inkscape." /></p><p>Grafts are a form of what <a href="https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf"><em>Build Systems à la
Carte</em></a>
by Mokhov <em>et al.</em> (a good read!) refers to as <em>dynamic dependencies</em>:
grafting depends on intermediate <em>build results</em>.</p><p>Still here? With the background in place, let’s look at the problems
that arose.</p><h1>Grafts, the user interface, and performance</h1><p>Conceptually, to decide whether to graft a package, we examine the
references of the build result of the ungrafted package. However, we
usually want <code>guix install</code> & co. to first display an overall build
plan, especially when invoked with <code>--dry-run</code>:</p><pre><code>$ guix install inkscape
The following package will be installed:
inkscape 1.0
71.3 MB will be downloaded:
/gnu/store/xq64iaxx2gmlcgnipj31wjxlf1yd2g2p-gsl-2.6
/gnu/store/zrmhnj3pwchn2msphgnwzwd3q89m46rn-aspell-0.60.8
/gnu/store/5g31zf21lk8nrfd2b8qrm19nwdm9gir9-potrace-1.16
/gnu/store/qpr7387bsjs7rpl6flrwdvn2zbzh5bch-ghostscript-with-cups-9.52
/gnu/store/7y3lvk3xf4im8n44337mc6y0ccysvfia-font-dejavu-2.37
/gnu/store/95n3zvzxzq2bxvdvz292cg04757ij30x-cups-minimal-2.3.1
…</code></pre><p>To accommodate that, the pre-1.1.0 implementation of grafts did the
following: when
<a href="https://guix.gnu.org/manual/en/html_node/Substitutes.html">substitutes</a>
were enabled, it would get the list of references of ungrafted packages
from substitutes; only when substitutes for an ungrafted package are
missing would it first try to build that package. Thus, when
substitutes are available, <code>guix install</code> and similar commands would be
able to display the build plan upfront. However, when a packages had no
substitutes, you’d see Guix start building it without saying a word
about the build plan, which was <a href="https://issues.guix.gnu.org/issue/28310">arguably
confusing</a>.</p><p>But it’s worse than that. Grafting is per-package, so every time you
would lower a package to a derivation, you would need to answer the
question “does <em>this</em> specific package have substitutes, and if so,
should it be grafted?” The end result was <a href="https://issues.guix.gnu.org/issue/22990">poor resource usage and
terrible user interface
feedback</a>. For every package
that is a graft candidate, the user would see that infamous line:</p><pre><code>updating substitutes from 'https://ci.guix.gnu.org'...</code></pre><p>The problem was particularly acute when building whole systems with
<code>guix system</code> because there would typically be a large number of such
packages. Furthermore, each of these lines would correspond to
(roughly) a single HTTP GET request on a fresh TLS connection. That can
be slow… and annoying. Perhaps to some users this <code>updating substitutes</code> stuttering was the proof of the developers’ incompetence
and perhaps, truth be told, to some of us developers it was a small
price to pay for the sophistication of grafts.</p><p>For users who disable substitutes and build everything locally, the
situation wasn’t much better: all the packages candidate for grafting
would be built one by one, thereby missing parallelization opportunities
as specified by
<a href="https://guix.gnu.org/manual/en/html_node/Invoking-guix_002ddaemon.html"><code>--max-jobs</code></a>.</p><h1>Gathering dynamic dependencies</h1><p>To address this, all these individual dynamic dependencies need to be
gathered somehow instead of being treated one by one. Conceptually, we
would like to, roughly, do a first pass lowering packages to derivations
as if grafting was disabled, build all these derivations, and then do a
second pass to determine which packages in the graph need to be grafted and
to compute the relevant grafting derivation. That would address the
performance issue: we’d now have as much parallelism as possible, so we
wouldn’t query substitutes or build packages one by one. If we reify
that second pass to the user interface code, it also addresses the user
interface issue by allowing it to display, possibly, two build plans:
the “ungrafted” one followed by the grafted one.</p><p>The problem is that our API is inherently serial: the
<code>package-derivation</code> function takes <em>one</em> package, lowers it, and
returns its derivation:</p><pre><code class="language-scheme">(use-modules (guix)
(gnu packages base)
(gnu packages inkscape))
(define s (open-connection))
(package-derivation s coreutils)
⇒ #<derivation /gnu/store/rpfdbax1py483m9ydhvp73s7dgmn6xh4-coreutils-8.31.drv => /gnu/store/jkj7wxybgcpdamkl6fz7wwbb1ak5wxvk-coreutils-8.31-debug /gnu/store/zibwkb5xavnv6z3gzknfqjsxb9b0izh0-coreutils-8.31 7f6c92e3a000>
(package-derivation s coreutils #:graft? #f)
⇒ #<derivation /gnu/store/rpfdbax1py483m9ydhvp73s7dgmn6xh4-coreutils-8.31.drv => /gnu/store/jkj7wxybgcpdamkl6fz7wwbb1ak5wxvk-coreutils-8.31-debug /gnu/store/zibwkb5xavnv6z3gzknfqjsxb9b0izh0-coreutils-8.31 7f6c92e3a000>
(package-derivation s inkscape)
⇒ #<derivation /gnu/store/jzm2zsq8m0rj8wdsmikc0p2ik0cprrcf-inkscape-0.92.4.drv => /gnu/store/clj8rjnsip8a35hyd9nf4l65w7ahn0gs-inkscape-0.92.4 7f6c9c15b730>
(package-derivation s inkscape #:graft? #f)
⇒ #<derivation /gnu/store/psd31x1fq0v2g594z217mh56xzg21dym-inkscape-0.92.4.drv => /gnu/store/zz28ckjwfxwkx3gsm8sc452kmvfiky6y-inkscape-0.92.4 7f6c90ad4f50></code></pre><p>Lowering includes dealing with grafts, and
that’s why we ended up with one-by-one inefficiencies. An option would
be to make all the API “plural”: have <code>package-derivation</code> and its
friends accept a <em>list</em> of packages instead of a single one. That would
be a huge amount of work and the end result would be unpleasant to use:
it’s easier to reason one-by-one.</p><p>The solution implemented in 1.1.0 instead starts from this observation:
the call graph of <code>package-derivation</code> mirrors the package graph. Thus,
we could gather dynamic dependencies using <a href="https://guix.gnu.org/manual/en/html_node/The-Store-Monad.html">monad
trickery</a>
or using “control effects”. We went for the latter, which didn’t have
the “contamination” problem of monads and led to simpler code.</p><p>The starting point is that, by definition, code with dynamic
dependencies necessarily calls
<a href="https://guix.gnu.org/manual/en/html_node/The-Store.html#index-build_002dderivations"><code>build-derivations</code></a>.
Taking advantage of <a href="https://www.gnu.org/software/guile/manual/html_node/Prompts.html">delimited continuations in
Guile</a>,
<code>build-derivations</code> is instrumented to <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=041b340da409078951267b6a8c43b27716e6b7ec">abort to a “build handler”
prompt</a>
when it’s called. The build handler receives the list of derivations to
build along with a continuation to invoke to resume the aborted
computation and start building things. User interface code can install
a build handler that displays what’s going to be built:</p><pre><code class="language-scheme">(with-build-handler (lambda (continue store things mode)
(show-what-to-build store things)
(continue #t))
…)</code></pre><p>To implement dry runs, simply omit the call to <code>continue</code> and nothing
will be built. (This is a slightly simplified artist view, see
<a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=07ce23e011d18460e7ff5553d4ff640f7073075b"><code>build-notifier</code></a>
for the real thing.)</p><p>Now, we need to take advantage of this mechanism to gather the
individual <code>build-derivations</code> calls so we can later emit a single
<code>build-derivations</code> call for all the gathered derivations. The goal is
to effectively gather all the calls for ungrafted packages, build them
all at once, and then resume graft computation.</p><p>To achieve that, we write a build handler that, when invoked, returns an
<code><unresolved></code> object that captures what to build and the continuation.
In addition, we provide a primitive to <em>introduce parallelism</em> such
that, if a dynamic dependency is encountered, we keep going and attempt
to compute as much as possible without resolving that dynamic
dependency. These are <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=c40bf5816cb3ffb59920a61f71bd34b53cac3637"><code>build-accumulator</code> and
<code>map/accumulate-builds</code></a>.
<code>map/accumulate-builds</code> is like <code>map</code>, except that it accumulates and
gathers <code>build-derivations</code> request.</p><p>By using <code>map/accumulate-builds</code> instead of <code>map</code> in a few
<a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=584dfdac3795541ff020aca3f488ceaf2ddd7fc3">key</a>
<a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=25af35fa32bf6c991510406a330d4a42bd5beba8">places</a>,
we obtain a good approximation of what we wanted, as illustrated in this
run:</p><pre><code>$ guix install zile-on-guile vim-full
The following packages will be installed:
zile-on-guile 2.4.14-0.fd09781
vim-full 8.2.0411
9.4 MB will be downloaded:
/gnu/store/vf7w4yiax38ra7x8aqqvbnc38c0ldgpm-zile-on-guile-2.4.14-0.fd09781
/gnu/store/dnj9wljcck9vdwgp7dwxk00qnnk1g3c5-vim-full-8.2.0411
downloading from https://ci.guix.gnu.org/nar/lzip/dnj9wljcck9vdwgp7dwxk00qnnk1g3c5-vim-full-8.2.0411...
vim-full-8.2.0411 8.9MiB 7.6MiB/s 00:01 [##################] 100.0%
downloading from https://ci.guix.gnu.org/nar/lzip/vf7w4yiax38ra7x8aqqvbnc38c0ldgpm-zile-on-guile-2.4.14-0.fd09781...
zile-on-guile-2.4.14-0.fd09781 140KiB 1.8MiB/s 00:00 [##################] 100.0%
The following derivation will be built:
/gnu/store/d9xms78917w67xq71pqsx5x9s6dmq6d7-profile.drv
The following graft will be made:
/gnu/store/4n6dmg6iwjg0adpcvqygr9wgsnclswss-vim-full-8.2.0411.drv
applying 8 grafts for /gnu/store/4n6dmg6iwjg0adpcvqygr9wgsnclswss-vim-full-8.2.0411.drv...
building /gnu/store/d9xms78917w67xq71pqsx5x9s6dmq6d7-profile.drv...</code></pre><p>What we see above is first a build plan that downloads binaries for the
two ungrafted packages, followed by a build plan for one grafting
derivations: we have successfully preserved parallelism.</p><p>The solution resembles the <code>suspending</code> scheduler discussed in the <em>à
la Carte</em> paper, though decomposition is not as principled as what the
paper describes. It remains an approximation and not the
optimal way to deal with dynamic dependencies. There are still
situations <a href="https://issues.guix.gnu.org/issue/40612">where that shows</a>,
but overall, it’s a significant improvement. Unlike <a href="https://issues.guix.gnu.org/issue/22990#7">other solutions
prototyped before</a>, this one
has the advantage of being orthogonal and simple: less than <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=c40bf5816cb3ffb59920a61f71bd34b53cac3637">100 new
lines of
code</a>,
and even <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=4b75a7060058bc2e959dcb4145067f6bba3e34e5">about 30 lines
removed</a>
from the graft implementation. That alone contributes a lot to the
author’s satisfaction. :-)</p><h1>Interlude: a scatter/gather pattern?</h1><p>In the end, we’re just gathering all the <code>build-derivations</code> calls,
turning them into a single call, and finally calling all the original
site continuations with the result. The same kind of issue shows up
when dealing with sequences of remote procedure calls (RPCs) and HTTP
requests, and it seems there’s a more general pattern lurking here.
Consider code like this:</p><pre><code class="language-scheme">(map (lambda (thing)
(http-get (thing->url thing)))
lst)</code></pre><p>Wouldn’t it be nice if we could somehow capture all the <code>http-get</code>
calls, turn them into a series of <a href="https://en.wikipedia.org/wiki/HTTP_pipelining">pipelined GET
requests</a>, and resume the
continuations with their result?</p><p>I haven’t found a standard functional pattern to address this and would
welcome ideas!</p><h1>Dynamic dependencies of all shapes</h1><p>We have seen how Guix deals with dynamic dependencies. Nix supports a
similar but limited form of dynamic dependencies through
the <code>import</code> primitive of the
Nix language, <a href="https://github.com/NixOS/nix/blob/master/src/libexpr/primops.cc#L74">which can take the result of a derivation
build</a>;
it does not attempt to gather the resulting <code>buildPaths</code> calls.</p><p>If we take a step back, we can see that Nix and Guix actually support
other forms of dynamic dependencies. For example, it’s possible to
write derivations whose result is a function of the reference graph of
another derivation’s build result. This is achieved in Guix by passing
the <a href="https://guix.gnu.org/manual/en/html_node/G_002dExpressions.html#index-gexp_002d_003ederivation"><code>#:references-graphs</code> argument of
<code>gexp->derivation</code></a>,
which leads the build daemon to <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/nix/libstore/build.cc?id=298fb2907e3f432cea7dee9f58e89ab8d9dbd56f#n1763">include the reference graph in the
build
environment</a>.</p><p>Another form of dynamic dependency is <em>derivation-building derivations</em>
or <em>recursive derivations</em>, which were <a href="https://github.com/NixOS/nix/pull/3205">recently implemented in
Nix</a>. It supports another form
of dynamic dependency where the build process of a derivation can itself
create and build derivations (these are <a href="https://en.wikipedia.org/wiki/Parallel_task_scheduling_problem"><em>moldable
tasks</em></a>
in scheduling parlance). It’s a great feature because in a nutshell, it
allows Nix to be used not only to compose packages, but also at a finer
grain as part of a package build process.</p><p>Guix supports yet another form of dynamic dependencies. The newfangled
<a href="https://guix.gnu.org/manual/en/html_node/Invoking-guix-deploy.html"><code>guix deploy</code>
tool</a>
works by <a href="https://guix.gnu.org/blog/2019/managing-servers-with-gnu-guix-a-tutorial/">evaluating g-expressions (gexps)
remotely</a>.
For example, before actually deploying an operating system, it first
runs code on the remote node to perform sanity checks: checking whether
the declared file system UUIDs or labels are valid, checking whether
additional kernel modules should be added to the initial RAM disk, and
so forth. To do that,
<a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/remote.scm#n109"><code>remote-eval</code></a>
first builds a derivation that produces a Scheme program, deploys it
along with all its dependencies on that target machine, runs it, and
retrieves the result. This form of dynamic dependency also benefits
from the gathering machinery discussed above.</p><h1>Conclusion</h1><p>This is a long article on what may look like a fine point of Guix design
and implementation, but there’s much to say about it! Grafts are key to
the use of functional deployment in production because they enable quick
security updates, and it’s a lot better if they don’t harm the user
experience.</p><p>The pre-1.1.0 implementation of grafts had a negative impact on the user
interface and on performance, which was due to the sequential handling
of grafts, one package at a time. In 1.1.0 we addressed it by using
delimited continuations to gather dynamic dependencies such as grafts,
perform builds in bulk, and resume each derivation computation.</p><p>As it turned out, the implementation of dynamic dependencies raises lots
of interesting design and implementation issues, and it’s probably not
the end of the road!</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package
manager and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects
user
freedom</a>.
Guix can be used on top of any system running the kernel Linux, or it
can be used as a standalone operating system distribution for i686,
x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2020/guile-3-and-guix//Guile 3 & GuixLudovic Courtès2020-01-24T15:00:00Z2020-01-24T15:00:00Z Version 3.0 of GNU Guile, an implementation of the Scheme programming
language , was released just last
week .
This is a major milestone for Guile, which gets compiler improvements
and just-in-time (JIT) native code generation, leading to significant
performance improvements over 2.2. It’s also great news for all the
users of Guile, and in particular for Guix! This post discusses what it means for Guix to migrate to Guile 3 and
how that migration is already taking place. Guile in Guix Most users interact with Guix through its command-line interface, and we
work hard to make it…<p>Version 3.0 of GNU Guile, an implementation of the <a href="https://schemers.org">Scheme programming
language</a>, <a href="https://www.gnu.org/software/guile/news/gnu-guile-300-released.html">was released just last
week</a>.
This is a major milestone for Guile, which gets compiler improvements
and just-in-time (JIT) native code generation, leading to significant
performance improvements over 2.2. It’s also great news for all the
users of Guile, and in particular for Guix!</p><p><img src="https://guix.gnu.org/static/blog/img/guile-3.png" alt="Guile 3 logo." /></p><p>This post discusses what it means for Guix to migrate to Guile 3 and
how that migration is already taking place.</p><h1>Guile in Guix</h1><p>Most users interact with Guix through its command-line interface, and we
work hard to make it as approachable as possible. As any user quickly
notices, Guix uses the Scheme programming language uniformly for its
configuration—from
<a href="https://guix.gnu.org/manual/devel/en/html_node/Channels.html">channels</a>
to
<a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-package.html#profile_002dmanifest">manifests</a>
and <a href="https://guix.gnu.org/manual/devel/en/html_node/Using-the-Configuration-System.html">operating
systems</a>—and
anyone who starts packaging software knows that <a href="https://guix.gnu.org/manual/devel/en/html_node/Defining-Packages.html">package
definitions</a>
are in fact Scheme code as well.</p><p>This is a significant departure from many other, and in particular from
<a href="https://nixos.org/nix/">Nix</a>. While Nix defines several
domain-specific languages (DSLs) for these aspects—the <a href="https://nixos.org/nix/manual/#chap-writing-nix-expressions">Nix
language</a>
but also specific
<a href="https://nixos.org/nix/manual/#sec-conf-file">configuration</a>
<a href="https://nixos.org/nix/manual/#chap-distributed-builds">languages</a>—Guix
chooses Scheme as the single language for all this, <a href="https://hal.inria.fr/hal-00824004/en">together with the
definition of high-level embedded domain-specific languages
(EDSLs)</a>.</p><p>It goes beyond that: in Guix System, all the things traditionally
implemented in C or as a set of Perl or shell scripts are implemented in
Scheme. That includes <a href="https://www.gnu.org/software/shepherd/">the init
system</a>, <a href="https://guix.gnu.org/manual/en/html_node/Build-Systems.html">package
builds</a>,
<a href="https://guix.gnu.org/manual/en/html_node/Initial-RAM-Disk.html">the initial RAM disk
(initrd)</a>,
<a href="https://guix.gnu.org/blog/2016/guixsd-system-tests/">system tests</a>, and
more. Because this leads to several layers of Scheme code, executed at
different points in time, Guix includes a <a href="https://hal.inria.fr/hal-01580582/en"><em>code staging</em>
mechanism</a> built upon the nice
properties of Scheme.</p><p>Why do that? The arguments, right from the start, were twofold: using a
general-purpose language allows us to benefit from its implementation
tooling, and having interfaces for “everything” in Scheme makes it easy
for users to navigate their distro or OS code and to reuse code to build
new features or applications. Guix developers benefit from the ease of
code reuse every day; demonstrative examples include the <a href="https://guix.gnu.org/blog/2017/running-system-services-in-containers/">use of Guix
container facilities in the init
system</a>,
the development of
<a href="https://guix.gnu.org/manual/devel/en/html_node/Development.html">many</a>
<a href="https://guix.gnu.org/manual/devel/en/html_node/Utilities.html">tools</a>
providing facilities around packages, the implementation of additional
<a href="https://emacs-guix.gitlab.io/website/">user</a>
<a href="https://github.com/UMCUGenetics/hpcguix-web/">interfaces</a>, and work on
applications that use Guix as a library such as the <a href="https://www.guixwl.org/">Guix Workflow
Language</a> and
<a href="https://hpc.guix.info/blog/2019/10/towards-reproducible-jupyter-notebooks/">Guix-Jupyter</a>.</p><p>As for the benefits of the host general-purpose language, these are
rather obvious: Guix developers benefit from an expressive language, an
optimizing compiler, a debugger, a powerful <a href="https://www.gnu.org/software/guile/manual/html_node/Using-Guile-Interactively.html">read-eval-print loop
(REPL)</a>,
an <a href="https://nongnu.org/geiser/">interactive development environment</a>,
and all sorts of libraries. Moving to Guile 3 should add to that better
performance, essentially for free. To be comprehensive, Guile 3 may
well come with a set of brand new bugs too, but so far we seem to be
doing OK!</p><h1>Migrating to Guile 3</h1><p>What does it mean for Guix to migrate to Guile 3? We’ve seen above
different ways in which Guix relies on Guile. In short, we can say that
migration is threefold:</p><ol><li>Guix is a distro that ships Guile-related packages. Like any other
distro, it will have to upgrade its <code>guile</code> package to 3.0 and to
ensure packages that depend on it and updated as well.</li><li>Guix is a program written in Guile. As such, we need to make sure
that all its dependencies (half a dozen of Guile libraries) work
with Guile 3 and that Guix itself runs fine with Guile 3.</li><li>Guix ties together operating system components. In particular, the
init system (the Shepherd) and other boot-time facilities will also
migrate.</li></ol><h2>The packages</h2><p>Updating the distro is the boring part, but it’s best to get it right.
Guix makes it possible to have unrelated versions of variants of
packages in different environments or different profiles, which is very
nice. We’ll have performed a smooth transition if users and tools see
that the packages named <code>guile</code> and <code>guile-ssh</code> (say) transparently move
from Guile 2.2 to 3.0, <em>in lockstep</em>.</p><p>Put differently, most of the upgrade work upon a programming language
version bump deals with conventions, and in particular package names.
Currently, <code>guile</code> corresponds to the 2.2 stable series and all the
<code>guile-*</code> packages are built against it. In the meantime, the package
for Guile 3 is named <code>guile-next</code> and packages built against it are
called <code>guile3.0-*</code>. Over the last few weeks we created <code>guile3.0-</code>
variants for most Guile packages, something that’s <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=89a99d53f56c7c383659d821c28286b6d71e458d">easily achieved with
Guix</a>.</p><p>The big switch will consist in renaming all current <code>guile-*</code> packages
to <code>guile2.2-*</code> packages, for use with the legacy 2.2 series, and
renaming all the <code>guile3.0-*</code> packages to <code>guile-*</code>. We will switch
soon, but before getting there, we’re making sure important packages are
available for 3.0.</p><h2>Guix-the-program</h2><p>A more interesting part is “porting” Guix itself from Guile 2.2 to
Guile 3. It seems that developers have become wary of 2-to-3
transitions for programming languages. Fear not! Switching from
Guile 2 to Guile 3 turned out to be an easy task. In fact, very little
changed in the language itself; what did change—e.g., semantics on fine
points of the module system, support for structured exceptions—is either
optional or backwards-compatible.</p><p>As Guile 2.9 pre-releases trickled in, we started testing all the Guile
libraries Guix relies on against 2.9. For the vast majority of them,
all we had to do was to <a href="https://gitlab.com/gnutls/gnutls/commit/763e31d351933222281bf9c11ff0bddb89bb701d">update their <code>configure.ac</code> to allow builds
with
3.0</a>.</p><p>Guix itself was a bit more work, mostly because it’s a rather large code
base with a picky test suite. The bit that required most work has to do
with the introduction of <a href="https://www.gnu.org/software/guile/manual/html_node/Declarative-Modules.html"><em>declarative
modules</em></a>,
an optional semantic change in modules to support more compiler
optimizations. We had several <a href="https://en.wikipedia.org/wiki/White-box_testing">“white-box
tests”</a> where tests
would happily peek at private module bindings through <a href="https://www.gnu.org/software/guile/manual/html_node/Using-Guile-Modules.html#index-_0040_0040">the magical-evil
<code>@@</code>
operator</a>.
Because we chose to enable declarative modules, we also had to adjust
our tests to no longer do that. And well, that’s about it!</p><p>At that point, we were able to <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=da7651806102d637253cb9f5677b96d6a178fc05">create a <code>guile3.0-guix</code> package
variant</a>,
primarily for testing purposes. Soon after, we told <a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-pull.html"><code>guix pull</code></a>
to <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=8234fe653e61d0090138cbd4c48d877568355439">build Guix with 3.0 instead of
2.2</a>.
Thus, Guix users who upgrade will transparently find themselves running
Guix on Guile 3.0.</p><p>The main benefit is improved performance. Guile 3 is known to be <a href="https://www.gnu.org/software/guile/news/gnu-guile-300-released.html">up to
32 times faster than
Guile 2.2</a>
on some micro-benchmarks. Assessing the performance gains on a
“real-world” application like Guix is the real test. What would be a
relevant benchmark? At its core, Guix is essentially a compiler from
high-level descriptions of packages, operating systems, and the like, to
low-level build instructions
(<a href="https://guix.gnu.org/manual/devel/en/html_node/Derivations.html"><em>derivations</em></a>).
Thus, a good benchmark is a command that exercises little more than this
compilation step:</p><pre><code>guix build libreoffice ghc-pandoc guix --dry-run --derivation</code></pre><p>or:</p><pre><code>guix system build config.scm --dry-run --derivation</code></pre><p>On x86_64, the <code>guix build</code> command above on Guile 3 is 7% faster than
on Guile 2.2, and <code>guix system build</code>, which is more
computing-intensive, is 10% faster (heap usage is ~5% higher). This is
lower than the skyrocketing speedups observed on some microbenchmarks,
but it’s probably no surprise: these <code>guix</code> commands are short-lived (a
couple of seconds) and they’re rather I/O- and GC-intensive—something
JIT compilation cannot help with.</p><p>On 32-bit ARM, we temporarily disabled JIT <a href="https://issues.guix.gnu.org/issue/39208">due to a
bug</a>; there we observe a slight
<em>slowdown</em> compared to 2.2. This can be explained by the fact that
<a href="https://wingolog.org/archives/2018/01/17/instruction-explosion-in-guile">virtual machine (VM) instructions in 3.0 are lower-level than in
2.2</a>
and will hopefully be more than compensated for when JIT is re-enabled.</p><h2>Gluing it all together</h2><p>The last part of the Guile 3 migration has to do with how Guix, and in
particular Guix System, glues things together. As explained above, Guix
manipulates several stages of Scheme code that will run a different
points in time.</p><p>Firstly, the code that runs package builds, such as <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/gnu-build-system.scm">the one that runs
<code>./configure && make && make install</code></a>,
is Guile code. Currently that code runs on Guile 2.2, but on the next
major rebuild-the-world upgrade, we will switch to Guile 3.</p><p>Additionally, Guix produces Scheme code consumed by <a href="https://www.gnu.org/software/shepherd">the
Shepherd</a>, by
<a href="https://www.gnu.org/software/mcron">GNU mcron</a>, and for <a href="https://guix.gnu.org/manual/en/html_node/Guided-Graphical-Installation.html">the graphical
installer</a>.
These will soon switch to Guile 3 as well. This kind of change is made
easy by the fact that both the package definitions and the staged code
that depends on those packages live in the same repository.</p><h1>Long live Guile 3!</h1><p>Migrating Guix to Guile 3 is a bit of work because of the many ways Guix
interacts with Guile and because of the sheer size of the code base.
For a “2-to-3” transition though, it was easy. And fundamentally, it
remains a cheap transition compared to what it brings: better
performance and new features. That’s another benefit of using a
general-purpose language.</p><p>Thumbs up to everyone involved in its development, and long live
Guile 3!</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package
manager and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects
user
freedom</a>.
Guix can be used on top of any system running the kernel Linux, or it
can be used as a standalone operating system distribution for i686,
x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2019/spreading-the-news//Spreading the newsLudovic Courtès2019-11-12T14:00:00Z2019-11-12T14:00:00Z Like most free software projects, Guix has no shortage of communication
channels: there’s this blog , the NEWS
file for release
notes, a bunch of mailing lists , an IRC
channel, there’s also an unofficial
sub-Reddit and certainly more. Yet, as
developers, we often find it hard to communicate important changes to
our users. Starting from a few weeks ago, guix pull --news tells
users what’s new, and it already feels very helpful! This post is about
our motivations and the implementation of this new feature. Getting the word out Developers keep…<p>Like most free software projects, Guix has no shortage of communication
channels: there’s <a href="https://guix.gnu.org/blog">this blog</a>, the <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/NEWS"><code>NEWS</code>
file</a> for release
notes, a bunch of <a href="https://guix.gnu.org/contact">mailing lists</a>, an IRC
channel, there’s also an <a href="https://www.reddit.com/r/GUIX/">unofficial
sub-Reddit</a> and certainly more. Yet, as
developers, we often find it hard to communicate important changes to
our users. Starting from a few weeks ago, <code>guix pull --news</code> tells
users what’s new, and it already feels very helpful! This post is about
our motivations and the implementation of this new feature.</p><h1>Getting the word out</h1><p>Developers keep adding crazy features, fixing bugs, and generally
improving things. But how good is it if users aren’t aware of these new
things? As an example, since June, our build farm has been <a href="https://guix.gnu.org/blog/2019/substitutes-are-now-available-as-lzip/">offering
lzip-compressed
binaries</a>,
which results in better performance when installing software. But to
take advantage of that, users need to be aware of its existence, and
they need to upgrade their Guix daemon. Likewise, how do we get people
to learn about <a href="https://guix.gnu.org/blog/2019/managing-servers-with-gnu-guix-a-tutorial/">the new <code>guix deploy</code>
command</a>
that’s now available at their fingertips, about <a href="https://guix.gnu.org/blog/tags/security-advisory/">security issues that
were fixed</a>, about
important <a href="https://lists.gnu.org/archive/html/info-guix/2019-06/msg00001.html">infrastructure
changes</a>,
new options added to existing commands, and so forth?</p><p>Our (frustrating!) experience has been that release notes, blog posts,
and mailing list announcements aren’t quite enough to get the word out.
There’s always people who’ll miss important info and realize when it’s
already late, sometimes too late. Hence this simple idea: wouldn’t it
be nice if important information would reach users right in their
terminal?</p><h1><code>guix pull</code> news</h1><p>Alright, that’s not exactly a novel idea! In Debian for example,
<a href="https://manpages.debian.org/testing/apt-listchanges/apt-listchanges.1.en.html"><code>apt-listchanges</code></a>
shows news at the level of individual packages, taken from the
<code>NEWS.Debian</code> or <code>changelog.Debian</code> files that package maintainers
update. In addition, <code>apt dist-upgrade</code> and similar commands typically
display dialog boxes and menus when special actions need to be taken
when upgrading. That’s more or less what we’re looking for.</p><p>The situation in Guix is a little different: all of it lives in <a href="https://git.savannah.gnu.org/cgit/guix.git/">a
single Git repository</a> that
contains the core, the command-line interfaces, as well as package
definitions and operating system service definitions. The <a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-pull.html"><code>guix pull</code></a>
command is sort-of equivalent to <code>apt update</code>, except that it updates
not only the set of available packages but also the <code>guix</code> tools
themselves and all the <a href="https://guix.gnu.org/manual/devel/en/html_node/Using-the-Configuration-System.html">operating system
interfaces</a>.
Under the hood, <code>guix pull</code> essentially does <code>git pull</code> and consequently
updates all of this. Guix very much follows a “rolling release” model.</p><p>For some time already, <code>guix pull</code> has been able to extract information
about new and upgraded packages and <a href="https://guix.gnu.org/blog/2018/multi-dimensional-transactions-and-rollbacks-oh-my/">to present it to the
user</a>.
Our goal was to complement it with high-level information about
important changes written with users in mind. Clearly, showing the Git
commit log is not an option: commit logs are meant for developers and
there’s roughly a thousand commits per month—way too much information.
We needed high-level news entries, explicitly written for users.</p><p>The end result is this: <code>guix pull</code> now displays, in addition to a
summary of the new and upgraded packages, the headlines of applicable
news entries contributed by developers. Users can view the details by
running <code>guix pull --news</code>:</p><p><img src="https://guix.gnu.org/static/blog/img/guix-pull-news.gif" alt="'guix pull' displaying news." /></p><p>Users can no longer miss the news, for the benefit of both users and
developers!</p><h1>Under the hood</h1><p>How does this all work? There were several goals and constraints.
First, like commit logs, our high-level news entries should be anchored
in the Git history. Second, unlike commit logs, it should be possible
to amend them—to fix typos, provide additional info, and so on. Third,
the project has been paying a lot of attention to internationalization,
with translations available for <a href="https://translationproject.org/domain/guix.html">user interface
messages</a>, for <a href="https://translationproject.org/domain/guix-packages.html">package
descriptions</a>,
and for the <a href="https://guix.gnu.org/manual/">user manual</a>—it’s one of
these things that helps free software reach out to more people; thus, we
naturally wanted news to be internationalized. Last, since Guix
supports third-party
<a href="https://guix.gnu.org/manual/devel/en/html_node/Channels.html">“channels”</a>,
which are extensions of the official <code>guix</code> channel, why not provide
channel authors access to that news feature?</p><p>With all these things in mind, we designed a simple <a href="https://guix.gnu.org/manual/devel/en/html_node/Channels.html#Writing-Channel-News">news
format</a>.
In essence, channel authors, including Guix developers, can provide a
news file that looks <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/etc/news.scm">like
this</a>:</p><pre><code class="language-scheme">(channel-news
(version 0)
(entry (commit "3e962e59d849e4300e447d94487684102d9d412e")
(title (en "@command{guix graph} now supports package
transformations")
(de "@command{guix graph} unterstützt nun Paketumwandlungen"))
(body
(en "The @command{guix graph} command now supports the common package
transformation options (see @command{info \"(guix) Package Transformation
Options\"}). This is useful in particular to see the effect of the
@option{--with-input} dependency graph rewriting option.")
(de "Der Befehl @command{guix graph} unterstützt nun die mit anderen
Befehlen gemeinsamen Umwandlungsoptionen (siehe @command{info \"(guix.de)
Paketumwandlungsoptionen\"}). Sie helfen insbesondere dabei, die Wirkung der
Befehlszeilenoption @option{--with-input} zum Umschreiben des
Abhängigkeitsgraphen zu sehen.")))
(entry (commit "49af34cfac89d384c46269bfd9388b2c73b1220a")
(title (en "@command{guix pull} now honors
@file{/etc/guix/channels.scm}")
(es "Ahora @command{guix pull} tiene en cuenta
@file{/etc/guix/channels.scm}"))
(body
(en "The @command{guix pull} command will now read the
@file{/etc/guix/channels.scm} file if it exists and if the per-user
@file{~/.config/guix/channels.scm} is not present. This allows administrators
of multi-user systems to define site-wide defaults.")
(es "Ahora la orden @command{guix pull} lee el fichero
@file{/etc/guix/channels.scm} si existe y el fichero personalizable
@file{~/.config/guix/channels.scm} no está presente. Esto permite a quienes
administran sistemas con múltiples usuarias definir valores predeterminados
en el sistema."))))</code></pre><p>Each news entry refers to a commit, the commit that introduced the
change it documents, and it has a title and body. Those can use Texinfo
markup for rich formatting, and translations can be provided directly
within the news file.</p><p>When <code>guix pull --news</code> runs, it determines which news entries are
applicable given the user’s previous Guix instance. The <code>(guix channels)</code> module provides a simple programming interface for that:</p><pre><code class="language-scheme">(use-modules (guix channels) (srfi srfi-1))
(channel-news-for-commit (first %default-channels)
"66b707a7d2325daadeed5ca913637eea3a2628e7"
"ac19950507e941b6263f62f4ee4e8934c1b1598e")
⇒ (#<<channel-news-entry> commit: "3e962e59d849e4300e447d94487684102d9d412e" tag: #f title: (("en" . "@command{guix graph} now supports package\ntransformations") …) body: …> #<<channel-news-entry> commit: "49af34cfac89d384c46269bfd9388b2c73b1220a" tag: #f title: (("en" . "@command{guix pull} now honors\n@file{/etc/guix/channels.scm}") …) body: …>)</code></pre><p>One thing is quite unusual (one might say “weird” :-)) about this news
format: it refers to commit IDs “in-band”. In other words, unlike Git
commit logs, which are “out-of-band”, the news file is contained
<em>inside</em> the repository that it refers to. Technically, it means that
before pushing a news entry, one must make sure it refers to the right
commit ID (the news format allows you to refer to tags as well, but one
may not want to create tags for every “news-worthy” change). Likewise,
rebasing might invalidate commit IDs that appear in the news file. So
this whole “in-band” log has drawbacks, but the big win is that it
allows us to amend news entries to fix typos, add translations, and so
on.</p><h1>In other news…</h1><p>Since it was applied <a href="https://issues.guix.gnu.org/issue/37413">a bit more than a month
ago</a>, we’ve already put the
news mechanism to good use on quite a few occasions: giving users
instructions on how to deal with locales after the last glibc upgrade,
giving them upgrade info for
<a href="https://guix.gnu.org/blog/2019/insecure-permissions-on-profile-directory-cve-2019-18192/">CVE-2019-18192</a>,
telling them about new command-line options, and more.</p><p>In parallel, given that reading the mailing lists is akin to “drinking
from a fire hose” as they say, Christopher Baines has been thinking
about how to provide regular development updates to interested users and
developers. Chris <a href="https://lists.gnu.org/archive/html/guix-devel/2019-11/msg00016.html">announced last
week</a>
a prototype of a “Guix Weekly News” web site that would aggregate
information about package updates automatically extracted from the <a href="http://data.guix.gnu.org/">Guix
Data Service</a>, along with manually written
updates. It would seem that this service could readily grab info from
channel news as well.</p><p>What about you, what do you expect in terms of news distribution? Join
us <a href="https://guix.gnu.org/contact">on the mailing list and on IRC</a> and
let us know!</p><h4>About GNU Guix</h4><p><a href="https://www.gnu.org/software/guix">GNU Guix</a> is a transactional package
manager and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects
user
freedom</a>.
Guix can be used on top of any system running the kernel Linux, or it
can be used as a standalone operating system distribution for i686,
x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2019/managing-servers-with-gnu-guix-a-tutorial//Managing Servers with GNU Guix: A TutorialJakob L. Kreuze2019-11-04T15:00:00Z2019-11-04T15:00:00Z The outcome of this year's
GSoC
is a Guile-based programming interface named guix deploy for
automatically creating, upgrading, and changing the configurations of
machines running the Guix System. The tool is comparable to
Ansible or
NixOps , but makes use of the system
configuration facilities provided by Guix. A post from earlier this
summer
described an early version of the programming interface, but we're
already a few months into autumn, so it's time for guide on how you
can use guix deploy in production. Simple case: managing a home server If the machine you…<p>The outcome of this year's
<a href="https://summerofcode.withgoogle.com/projects/#5232565294727168">GSoC</a>
is a Guile-based programming interface named <code>guix deploy</code> for
automatically creating, upgrading, and changing the configurations of
machines running the Guix System. The tool is comparable to
<a href="https://www.ansible.com/">Ansible</a> or
<a href="https://nixos.org/nixops/">NixOps</a>, but makes use of the system
configuration facilities provided by Guix. A <a href="http://guix.gnu.org/blog/2019/towards-guix-for-devops/">post from earlier this
summer</a>
described an early version of the programming interface, but we're
already a few months into autumn, so it's time for guide on how you
can use <code>guix deploy</code> in production.</p><h4>Simple case: managing a home server</h4><p>If the machine you need to manage is already running the Guix System,
it shouldn't be too hard to incorporate <code>guix deploy</code> into your
workflow. All that's needed is the <code><operating-system></code> declaration
you've been passing to <code>guix system reconfigure</code> and some information
about the machine (specifically its IP address and architecture). The
<code>guix deploy</code> command is invoked with the filename of a "deployment
specification" as an argument, whose contents should look something
like this:</p><pre><code class="language-scheme">;; Module imports
(use-modules (gnu) (guix))
(use-service-modules networking ssh)
(use-package-modules bootloaders)
;; Operating system description
(define os
(operating-system
(locale "en_US.utf8")
(timezone "America/New_York")
(keyboard-layout (keyboard-layout "us" "altgr-intl"))
(bootloader (bootloader-configuration
(bootloader grub-bootloader)
(target "/dev/sda")
(keyboard-layout keyboard-layout)))
(file-systems (cons* (file-system
(mount-point "/")
(device "/dev/sda1")
(type "ext4"))
%base-file-systems))
(host-name "alyssas-home-server")
(users (cons* (user-account
(name "alyssa")
(comment "Alyssa")
(group "users")
(home-directory "/home/alyssa")
(supplementary-groups
'("wheel" "netdev" "audio" "video")))
%base-user-accounts))
(sudoers-file (plain-file "sudoers" "\
root ALL=(ALL) ALL
%wheel ALL=NOPASSWD: ALL\n"))
(services (append
(list (service openssh-service-type
(openssh-configuration
(permit-root-login #t)))
(service dhcp-client-service-type))
%base-services))))
;; List of machines to deploy
(list (machine
(operating-system os)
(environment managed-host-environment-type)
(configuration (machine-ssh-configuration
(host-name "alyssa-p-hacker.tld")
(system "i686-linux")
(identity "/path/to/ssh-key")))))</code></pre><p>Even if Scheme isn't your forté, parts of this should look familiar if
you've used Guix before. The "operating system description" section
in particular is something you might use with <code>guix system reconfigure</code>. What's new is the last part: We construct a <code>list</code>
containing one <code>machine</code> of the <code>managed-host-environment-type</code>, for
which we've specified that <code>os</code> is the <code>operating-system</code>
declaration that we want to install on it, and that we can connect to
it using the parameters specified by the <code>machine-ssh-configuration</code>.</p><p>Let's take a step back for a moment and explain what a <code>machine</code> is.
<code>guix deploy</code> aims to support a number of different use-cases, which
we abstract as "environment types". We'll see other environment types
later in this article, but the general idea is that these environments
specify how resources should be "provisioned" or created. For
example, an environment type designed for working with a Virtual
Private Server (VPS) provider might make calls the provider's API to
request a virtual machine before installing the <code>machine</code>'s
<code>operating-system</code> declaration on it.</p><p>The environment type used in this example,
<code>managed-host-environment-type</code>, is intended for machines that are
already running Guix System and are accessible over SSH. It expects
that the <code>configuration</code> field of the <code>machine</code> be an instance of
<code>machine-ssh-configuration</code>, whose available fields are described in
the
<a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-deploy.html">manual</a>.
This gives <code>guix deploy</code> the information it needs to connect to the
machine's SSH daemon.</p><p>Running <code>guix deploy</code> with this file would build the "operating system
closure" of <code>os</code> -- a bundle of the packages, configuration
files, and other dependencies necessary to realize that configuration
-- for the architecture specified by <code>system</code> (in this case
<code>i686-linux</code>), send it over SSH to <code>alyssa-p-hacker.tld</code>, and then
remotely "activate" the configuration by creating a new system
generation and upgrading running services. Sweet! Upgrading our
single server setup has been reduced to an endeavour involving just
over a dozen keystrokes.</p><h4>More advanced case: managing a virtual private server deployment</h4><p>One server not cutting it for you? <code>guix deploy</code> can still help.
Suppose we run a web service that we'd like to split up across
multiple machines for performance reasons.</p><pre><code class="language-scheme">(define %forum-server-count 4)
(define (forum-server n)
(operating-system
(host-name (format #f "forum-server-~a" n))
...
(services (append (list (service httpd-service-type
(httpd-configuration
...)))
%base-services))))
(map (lambda (n)
(machine
(system (forum-server n))
(environment digital-ocean-environment-type)
(configuration (digital-ocean-configuration
(region "nyc3")
(size "s-1vcpu-1gb")
(enable-ipv6 #t)))))
(iota %forum-server-count))</code></pre><p>This example isn't as concrete as the first one; I'm intentionally
omitting parts of the configuration to make the example clearer.
Here, we automate the creation of <code>%forum-server-count</code> Digital Ocean
"droplets" in their <code>NYC3</code> region by creating a list of 4 machines.</p><p>Assuming that the environment variable <code>GUIX_DIGITAL_OCEAN_TOKEN</code> is
properly set, running <code>guix deploy</code> with this file will do much of the
same as the previous example. The difference is that four virtual
machines will be automatically created on Digital Ocean.</p><p>One important thing to note about the <code>digital-ocean-environment-type</code>
is that, currently, it <em>does not</em> automatically clean up unused
virtual machines. If you change something in the deployment
specification and run <code>guix deploy</code> again, the virtual machines from
the previous deployment will remain until you destroy them yourself.</p><h4>A quick peek into the internals of <code>digital-ocean-environment-type</code></h4><p>It would be an overstatement to say that the process of implementing a
new environment type is easy, but a fair amount of the work has
already been done for you. We'll use the definition of
<code>digital-ocean-environment-type</code> as an example.</p><pre><code class="language-scheme">(define digital-ocean-environment-type
(environment-type
(machine-remote-eval digital-ocean-remote-eval)
(deploy-machine deploy-digital-ocean)
(roll-back-machine roll-back-digital-ocean)
(name 'digital-ocean-environment-type)
(description "Provisioning of \"droplets\": virtual machines
provided by the Digital Ocean virtual private server (VPS) service.")))</code></pre><p>The <code>environment-type</code> record specifies a small amount of metadata
(<code>name</code> and <code>description</code>), as well as the names of three procedures:
one for remotely evaluating a
<a href="http://guix.gnu.org/manual/en/html_node/G_002dExpressions.html#G_002dExpressions">G-Expression</a>
on the host (<code>machine-remote-eval</code>), one for deploying an
<code>operating-system</code> declaration to the host, and one for rolling the
host back one generation.</p><p>This might sound like a lot, but the pattern for these high-level
environment types is to somehow obtain a machine running Guix System,
set up an SSH daemon, and then delegate to
<code>managed-host-environment-type</code>. <code>digital-ocean-remote-eval</code> is a
pretty good example of this:</p><pre><code class="language-scheme">(define (digital-ocean-remote-eval target exp)
"Internal implementation of 'machine-remote-eval' for MACHINE instances with
an environment type of 'digital-ocean-environment-type'."
(mlet* %store-monad ((name (droplet-name target))
(network -> (droplet-public-ipv4-network name))
(address -> (hash-ref network "ip_address"))
(ssh-key -> (digital-ocean-configuration-ssh-key
(machine-configuration target)))
(delegate -> (machine
(inherit target)
(environment managed-host-environment-type)
(configuration
(machine-ssh-configuration
(host-name address)
(identity ssh-key)
(system "x86_64-linux"))))))
(machine-remote-eval delegate exp)))</code></pre><p>As you can see, you could reasonably go about implementing an
environment type without ever having to learn what a G-Expression is.
Here, <code>droplet-name</code> derives the name of the droplet from the
machine's <code>operating-system</code> declaration, the information necessary to
connect to the droplet is found using <code>droplet-public-ipv4-network</code>,
and that's used to create machine of <code>managed-host-environment-type</code>.</p><h4>In conclusion</h4><p>I sincerely hope that <code>guix deploy</code> proves to be a useful to anyone
dealing with system administration or software development.
Transactional upgrades should provide peace of mind to those managing
servers (it's worth noting that few existing tools are capable of
recovering from failed deployments), and I believe that
procedurally-generated deployment configurations could very well be
the future of distribution for software such as web services: Imagine
if setting up a <a href="https://joinmastodon.org">Mastodon</a> instance were as
easy as downloading a Scheme file and handing it off to <code>guix deploy</code>.
The ease of writing code that generates code isn't the only benefit of
using Guile for something like this. Guile is a general-purpose
programming language, so more advanced tooling can reasonably be built
atop <code>guix deploy</code>. A GTK or Emacs DevOps interface, perhaps? (If
that idea sounds outlandish, consider that the latter has <a href="https://emacs-guix.gitlab.io/website/">already
happened</a> for the package
management interface.)</p><p>It's been a great summer working alongside everyone in the Guix
community. <code>guix deploy</code> is brand new (and a little unstable!), but
we've had enthusiastic adoption by several on the mailing lists who
were quick to report any issues they found. I'd like to thank
everyone on the <code>#guix</code> IRC channel and the mailing lists who got me
up to speed with the code, answered my questions, gave feedback when I
submitted my patches, and put <code>guix deploy</code> under the pressure of use
in production. And of course, I want to thank my mentors Christopher
Lemmer Webber and David Thompson. I had to make some hard design
decisions, but this was made easier thanks to the guidance of two
experienced Guix veterans.</p><p>Oh, and this isn't a goodbye. I really feel I've found my place as a
Guix contributor, and I can't wait to see what the future will bring
for <code>guix deploy</code>. Catch ya on the mailing lists!</p><h4>Editor's note</h4><p>Thank you for all of your hard work, Jakob!</p><h4>About GNU Guix</h4><p><a href="https://www.gnu.org/software/guix">GNU Guix</a> is a transactional package
manager and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects
user
freedom</a>.
Guix can be used on top of any system running the kernel Linux, or it
can be used as a standalone operating system distribution for i686,
x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2019/towards-guix-for-devops//Towards Guix for DevOpsJakob L. Kreuze2019-07-12T19:00:00Z2019-07-12T19:00:00Z Hey, there! I'm Jakob, a Google Summer of Code intern and new contributor to
Guix. Since May, I've been working on a DevOps automation tool for the Guix
System, which we've been calling guix deploy . The idea for a Guix DevOps tool has been making rounds on the mailing lists for
some time now. Years, in fact; Dave Thompson and Chris Webber put together a
proof-of-concept for it way back in 2015. Thus, we've had plenty of time to gaze
upon the existing tools for this sort of thing -- Ansible , NixOps -- and
fantasize about a…<p>Hey, there! I'm Jakob, a Google Summer of Code intern and new contributor to
Guix. Since May, I've been working on a DevOps automation tool for the Guix
System, which we've been calling <code>guix deploy</code>.</p><p>The idea for a Guix DevOps tool has been making rounds on the mailing lists for
some time now. Years, in fact; Dave Thompson and Chris Webber put together a
proof-of-concept for it way back in 2015. Thus, we've had plenty of time to gaze
upon the existing tools for this sort of thing -- <a href="https://www.ansible.com/">Ansible</a>, <a href="https://nixos.org/nixops/">NixOps</a> -- and
fantasize about a similar tool, albeit with the expressive power of Guile scheme
and the wonderful system configuration facilities of Guix. And now, those
fantasies are becoming a reality.</p><p>"DevOps" is a term that might be unfamiliar to a fair number of Guix users. I'll
spare you the detour to Wikipedia and give a brief explanation of what <code>guix deploy</code> does.</p><p>Imagine that you've spent the afternoon playing around with Guile's <code>(web)</code>
module, developing software for a web forum. Awesome! But a web forum with no
users is pretty boring, so you decide to shell out a couple bucks for a virtual
private server to run your web forum. You feel that Wildebeest admirers on the
internet deserve a platform of their own for discussion, and decide to dedicate
the forum to that.</p><p>As it turns out, <em>C. gnou</em> is a more popular topic than you ever would have
imagined. Your web forum soon grows in size -- attracting hundreds of thousands
of simultaneous users. Despite Guile's impressive performance characteristics,
one lowly virtual machine is too feeble to support such a large population of
Wildebeest fanatics. So you decide to use Apache as a load-balancer, and shell
out a couple more bucks for a couple more virtual private servers. Now you've
got a problem on your hands; you're the proud owner of five or so virtual
machines, and you need to make sure they're all running the most recent version
of either your web forum software or Apache.</p><p>This is where <code>guix deploy</code> comes into play. Just as you'd use an
<code>operating-system</code> declaration to configure services and user accounts on a
computer running the Guix System, you can now use that same <code>operating-system</code>
declaration to remotely manage any number of machines. A "deployment" managing
your Wildebeest fan site setup might look something like this:</p><pre><code class="language-scheme">...
;; Service for our hypothetical guile web forum application.
(define guile-forum-service-type
(service-type (name 'guile-forum)
(extensions
(list (service-extension shepherd-root-service-type
guile-forum-shepherd-service)
(service-extension account-service-type
(const %guile-forum-accounts))))
(default-value (guile-forum-configuration))
(description "A web forum written in GNU Guile.")))
...
(define %forum-server-count 4)
(define (forum-server n)
(operating-system
(host-name (format #f "forum-server-~a" n))
...
(services
(append (list (service guile-forum-service-type
(guile-forum-configuration
"GNU Fan Forum!")))
%base-services))))
(define load-balancer-server
(operating-system
(host-name "load-balancer-server"
...
(services
(append (list (service httpd-service-type
(httpd-configuration
...)))
%base-services)))))
;; One machine running our load balancer.
(cons (machine
(system load-balancer-server)
(environment manged-host-environment-type)
(configuration (machine-ssh-configuration
...)))
;; And a couple running our forum software!
(let loop ((n 1)
(servers '()))
(if (> n %forum-server-count)
servers
(loop (1+ n)
(cons (machine
(system (forum-server n))
(environment manged-host-environment-type)
(configuration (machine-ssh-configuration
...)))
servers)))))</code></pre><p>The take-away from that example is that there's a new <code>machine</code> type atop the
good ol' <code>operating-system</code> type, specifying how the machine should be
<em>provisioned</em>. The version of <code>guix deploy</code> that's currently on the master
branch only supports <code>managed-host-environment-type</code>, which is used for machines
that are already up and running the Guix System. Provisioning, in that sense,
only really involves opening an SSH connection to the host. But I'm sure you can
imagine a <code>linode-environment-type</code> which automatically sets up a virtual
private server through Linode, or a <code>libvirt-environment-type</code> that spins up a
virtual machine for running your services. Those types are what I'll be working
on in the coming months, in addition to cleaning up the code that's there now.</p><p>And yes, you did read that right. <code>guix deploy</code> is on the Guix master branch
right now! In fact, we've already done a successful deployment right here on
<a href="http://ci.guix.gnu.org/">ci.guix.gnu.org</a>. So, if this sounds as though it'd be up your alley, run <code>guix pull</code>, crack open the <a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-deploy.html">manual</a>, and let us know how it goes!</p><h4>About GNU Guix</h4><p><a href="https://www.gnu.org/software/guix">GNU Guix</a> is a transactional package
manager and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects
user
freedom</a>.
Guix can be used on top of any system running the kernel Linux, or it
can be used as a standalone operating system distribution for i686,
x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2019/substitutes-are-now-available-as-lzip//Substitutes are now available as lzipLudovic Courtès2019-06-17T14:30:00Z2019-06-17T14:30:00Z For a long time, our build farm at ci.guix.gnu.org has been delivering
substitutes
(pre-built binaries) compressed with gzip. Gzip was never the best
choice in terms of compression ratio, but it was a reasonable and
convenient choice: it’s rock-solid, and zlib made it easy for us to have
Guile
bindings
to perform in-process compression in our multi-threaded guix publish
server. With the exception of building software from source, downloads take the
most time of Guix package upgrades. If users can download less,
upgrades become faster, and happiness ensues. Time has come to improve
on this, and starting…<p>For a long time, our build farm at ci.guix.gnu.org has been delivering
<a href="https://www.gnu.org/software/guix/manual/en/html_node/Substitutes.html">substitutes</a>
(pre-built binaries) compressed with gzip. Gzip was never the best
choice in terms of compression ratio, but it was a reasonable and
convenient choice: it’s rock-solid, and zlib made it easy for us to have
<a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/zlib.scm">Guile
bindings</a>
to perform in-process compression in our multi-threaded <a href="https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-publish.html"><code>guix publish</code></a>
server.</p><p>With the exception of building software from source, downloads take the
most time of Guix package upgrades. If users can download less,
upgrades become faster, and happiness ensues. Time has come to improve
on this, and starting from early June, Guix can publish and fetch
<a href="https://nongnu.org/lzip/">lzip</a>-compressed substitutes, in addition to
gzip.</p><h1>Lzip</h1><p><a href="https://nongnu.org/lzip/">Lzip</a> is a relatively little-known
compression format, initially developed by Antonio Diaz Diaz ca. 2013.
It has several C and C++ implementations with surprisingly few lines of
code, which is always reassuring. One of its distinguishing features is
a very good compression ratio with reasonable CPU and memory
requirements, <a href="https://nongnu.org/lzip/lzip_benchmark.html">according to benchmarks published by the
authors</a>.</p><p><a href="https://nongnu.org/lzip/lzlib.html">Lzlib</a> provides a well-documented C
interface and Pierre Neidhardt set out to write bindings for that
library, which eventually landed as the <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/lzlib.scm"><code>(guix lzlib)</code>
module</a>.</p><p>With this in place we were ready to start migrating our tools, and then
our build farm, to lzip compression, so we can all enjoy smaller
downloads. Well, easier said than done!</p><h1>Migrating</h1><p>The compression format used for substitutes is not a core component like
it can be in “traditional” binary package formats <a href="https://lwn.net/Articles/789449/">such as
<code>.deb</code></a> since Guix is conceptually a
“source-based” distro. However, deployed Guix installations did not
support lzip, so we couldn’t just switch our build farm to lzip
overnight; we needed to devise a transition strategy.</p><p>Guix asks for the availability of substitutes over HTTP. For example, a
question such as:</p><blockquote><p>“Dear server, do you happen to have a binary of
<code>/gnu/store/6yc4ngrsig781bpayax2cg6pncyhkjpq-emacs-26.2</code> that I could download?”</p></blockquote><p>translates into prose to an HTTP GET of
<a href="https://ci.guix.gnu.org/6yc4ngrsig781bpayax2cg6pncyhkjpq.narinfo">https://ci.guix.gnu.org/6yc4ngrsig781bpayax2cg6pncyhkjpq.narinfo</a>,
which returns something like:</p><pre><code>StorePath: /gnu/store/6yc4ngrsig781bpayax2cg6pncyhkjpq-emacs-26.2
URL: nar/gzip/6yc4ngrsig781bpayax2cg6pncyhkjpq-emacs-26.2
Compression: gzip
NarHash: sha256:0h2ibqpqyi3z0h16pf7ii6l4v7i2wmvbrxj4ilig0v9m469f6pm9
NarSize: 134407424
References: 2dk55i5wdhcbh2z8hhn3r55x4873iyp1-libxext-1.3.3 …
FileSize: 48501141
System: x86_64-linux
Deriver: 6xqibvc4v8cfppa28pgxh0acw9j8xzhz-emacs-26.2.drv
Signature: 1;berlin.guixsd.org;KHNpZ25hdHV…</code></pre><p>(This narinfo format is inherited from <a href="https://nixos.org/nix/">Nix</a> and
implemented
<a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/scripts/substitute.scm?id=121d9d1a7a2406a9b1cbe22c34343775f5955b34#n283">here</a>
and
<a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/scripts/publish.scm?id=121d9d1a7a2406a9b1cbe22c34343775f5955b34#n265">here</a>.)
This tells us we can download the actual binary from
<code>/nar/gzip/…-emacs-26.2</code>, and that it will be about 46 MiB (the
<code>FileSize</code> field.) This is what <code>guix publish</code> serves.</p><p>The trick we came up with was to allow <code>guix publish</code> to advertise
several URLs, one per compression format. Thus, for recently-built
substitutes, we get something <a href="https://ci.guix.gnu.org/mvhaar2iflscidl0a66x5009r44fss15.narinfo">like
this</a>:</p><pre><code>StorePath: /gnu/store/mvhaar2iflscidl0a66x5009r44fss15-gimp-2.10.12
URL: nar/gzip/mvhaar2iflscidl0a66x5009r44fss15-gimp-2.10.12
Compression: gzip
FileSize: 30872887
URL: nar/lzip/mvhaar2iflscidl0a66x5009r44fss15-gimp-2.10.12
Compression: lzip
FileSize: 18829088
NarHash: sha256:10n3nv3clxr00c9cnpv6x7y2c66034y45c788syjl8m6ga0hbkwy
NarSize: 94372664
References: 05zlxc7ckwflz56i6hmlngr86pmccam2-pcre-8.42 …
System: x86_64-linux
Deriver: vi2jkpm9fd043hm0839ibbb42qrv5xyr-gimp-2.10.12.drv
Signature: 1;berlin.guixsd.org;KHNpZ25hdHV…</code></pre><p>Notice that there are two occurrences of the <code>URL</code>, <code>Compression</code>, and
<code>FileSize</code> fields: one for gzip, and one for lzip. Old Guix instances
will just pick the first one, gzip; newer Guix will pick whichever
supported method provides the smallest <code>FileSize</code>, usually lzip. This
will make migration trivial in the future, should we add support for
other compression methods.</p><p>Users need to upgrade their Guix daemon to benefit from lzip. On a
“foreign distro”, simply run <code>guix pull</code> as root. On standalone Guix
systems, run <code>guix pull && sudo guix system reconfigure /etc/config.scm</code>. In both cases, the daemon has to be restarted, be it
with <code>systemctl restart guix-daemon.service</code> or with <code>herd restart guix-daemon</code>.</p><h1>First impressions</h1><p>This new gzip+lzip scheme has been deployed on ci.guix.gnu.org for a
week. Specifically, we run <code>guix publish -C gzip:9 -C lzip:9</code>, meaning
that we use the highest compression ratio for both compression methods.</p><p>Currently, only a small subset of the package substitutes are available
as both lzip and gzip; those that were already available as gzip have
not been recompressed. The following Guile program that taps into the
API of <a href="https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-weather.html"><code>guix weather</code></a>
allows us to get some insight:</p><pre><code class="language-scheme">(use-modules (gnu) (guix)
(guix monads)
(guix scripts substitute)
(srfi srfi-1)
(ice-9 match))
(define all-packages
(@@ (guix scripts weather) all-packages))
(define package-outputs
(@@ (guix scripts weather) package-outputs))
(define (fetch-lzip-narinfos)
(mlet %store-monad ((items (package-outputs (all-packages))))
(return
(filter (lambda (narinfo)
(member "lzip" (narinfo-compressions narinfo)))
(lookup-narinfos "https://ci.guix.gnu.org" items)))))
(define (lzip/gzip-ratio narinfo)
(match (narinfo-file-sizes narinfo)
((gzip lzip)
(/ lzip gzip))))
(define (average lst)
(/ (reduce + 0 lst)
(length lst) 1.))</code></pre><p>Let’s explore this at the
<a href="https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop">REPL</a>:</p><pre><code class="language-scheme">scheme@(guile-user)> (define lst
(with-store s
(run-with-store s (fetch-lzip-narinfos))))
computing 9,897 package derivations for x86_64-linux...
updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
scheme@(guile-user)> (length lst)
$4 = 2275
scheme@(guile-user)> (average (map lzip/gzip-ratio lst))
$5 = 0.7398994395478715</code></pre><p>As of this writing, around 20% of the package substitutes are
available as lzip, so take the following stats with a grain of salt.
Among those, the lzip-compressed substitute is on average 26% smaller
than the gzip-compressed one. What if we consider only packages bigger
than 5 MiB uncompressed?</p><pre><code class="language-scheme">scheme@(guile-user)> (define biggest
(filter (lambda (narinfo)
(> (narinfo-size narinfo)
(* 5 (expt 2 20))))
lst))
scheme@(guile-user)> (average (map lzip/gzip-ratio biggest))
$6 = 0.5974238562384483
scheme@(guile-user)> (length biggest)
$7 = 440</code></pre><p>For those packages, lzip yields substitutes that are 40% smaller on
average. Pretty nice! Lzip decompression is slightly more
CPU-intensive than gzip decompression, but downloads are
bandwidth-bound, so the benefits clearly outweigh the costs.</p><h1>Going forward</h1><p>The switch from gzip to lzip has the potential to make upgrades “feel”
faster, and that is great in itself.</p><p>Fundamentally though, we’ve always been looking in this project at
peer-to-peer solutions with envy. Of course, the main motivation is to
have a community-supported and resilient infrastructure, rather than a
centralized one, and that vision goes <a href="https://www.gnu.org/software/guix/blog/2017/reproducible-builds-a-status-update/">hand-in-hand with reproducible
builds</a>.</p><p>We started working on <a href="https://issues.guix.gnu.org/issue/33899">an extension to publish and fetch
substitutes</a> over
<a href="https://ipfs.io/">IPFS</a>. Thanks to its content-addressed nature, IPFS
has the potential to further reduce the amount of data that needs to be
downloaded on an upgrade.</p><p>The good news is that IPFS developers are also <a href="https://github.com/ipfs/package-managers">interested in working
with package manager
developers</a>, and I bet
there’ll be interesting discussions at <a href="https://camp.ipfs.io/">IPFS
Camp</a> in just a few days. We’re eager to pursue
our IPFS integration work, and if you’d like to join us and hack the
good hack, <a href="https://www.gnu.org/software/guix/contact/">let’s get in
touch!</a></p><h4>About GNU Guix</h4><p><a href="https://www.gnu.org/software/guix">GNU Guix</a> is a transactional package
manager and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects
user
freedom</a>.
Guix can be used on top of any system running the kernel Linux, or it
can be used as a standalone operating system distribution for i686,
x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2019/connecting-reproducible-deployment-to-a-long-term-source-code-archive//Connecting reproducible deployment to a long-term source code archiveLudovic Courtès2019-03-29T15:50:00Z2019-03-29T15:50:00Z GNU Guix can be used as a “package manager” to install and upgrade
software packages as is familiar to GNU/Linux users, or as an
environment manager, but it can also provision containers or virtual
machines, and manage the operating system running on your machine. One foundation that sets it apart from other tools in these areas is
reproducibility . From a high-level view, Guix allows users to
declare complete software environments and instantiate them. They can
share those environments with others, who can replicate them or adapt
them to their needs. This aspect is key to reproducible computational
experiments: scientists…<p>GNU Guix can be used as a “package manager” to install and upgrade
software packages as is familiar to GNU/Linux users, or as an
environment manager, but it can also provision containers or virtual
machines, and manage the operating system running on your machine.</p><p>One foundation that sets it apart from other tools in these areas is
<em>reproducibility</em>. From a high-level view, Guix allows users to
<em>declare</em> complete software environments and instantiate them. They can
share those environments with others, who can replicate them or adapt
them to their needs. This aspect is key to reproducible computational
experiments: scientists need to reproduce software environments before
they can reproduce experimental results, and this is one of the things
we are focusing on in the context of the
<a href="https://hpc.guix.info">Guix-HPC</a> effort. At a lower
level, the project, along with others in the <a href="https://reproducible-builds.org">Reproducible
Builds</a> community, is working to ensure
that software build outputs are <a href="https://reproducible-builds.org/docs/definition/">reproducible,
bit for bit</a>.</p><p>Work on reproducibility at all levels has been making great progress.
Guix, for instance, allows you to <a href="https://www.gnu.org/software/guix/blog/2018/multi-dimensional-transactions-and-rollbacks-oh-my/">travel back in
time</a>.
That Guix can travel back in time <em>and</em> build software reproducibly is a
great step forward. But there’s still an important piece that’s missing
to make this viable: a stable source code archive. This is where
<a href="https://www.softwareheritage.org">Software Heritage</a> (SWH for short)
comes in.</p><h1>When source code vanishes</h1><p>Guix contains thousands of package definitions. Each <a href="https://www.gnu.org/software/guix/manual/en/html_node/Defining-Packages.html">package
definition</a>
specifies the package’s source code URL and hash, the package’s
dependencies, and its build procedure. Most of the time, the package’s
source code is an archive (a “tarball”) fetched from a web site, but
more and more frequently the source code is a specific revision checked
out directly from a version control system.</p><p>The obvious question, then, is: what happens if the source code URL
becomes unreachable? The whole reproducibility endeavor collapses when
source code disappears. And source code <em>does</em> disappear, or, even
worse, it can be modified in place. At GNU we’re doing a good job of
having stable hosting that keeps releases around “forever”, unchanged
(modulo rare exceptions). But a lot of free software out there is
hosted on personal web pages with a short lifetime and on commercial
hosting services that come and go.</p><p>By default Guix would look up source code by hash in the caches of our
build farms. This comes for free: the <a href="https://www.gnu.org/software/guix/manual/en/html_node/Substitutes.html">“substitute”
mechanism</a>
extends to all “build artifacts”, including downloads. However, with
limited capacity, our build farms do not keep all the source code of all
the packages for a long time. Thus, one could very well find oneself
unable to rebuild a package months or years later, simply because its
source code disappeared or moved to a different location.</p><h1>Connecting to the archive</h1><p>It quickly became clear that reproducible builds had “reproducible
source code downloads”, so to speak, as a prerequisite. The Software
Heritage archive is the missing piece that would finally allow us to
reproduce software environments years later in spite of the volatility
of code hosting sites. Software Heritage’s mission is to archive
essentially “all” the source code ever published, including version
control history. Its archive already periodically ingests release
tarballs from the GNU servers, repositories from GitHub, packages from
PyPI, and much more.</p><p><img src="https://www.gnu.org/software/guix/static/blog/img/software-heritage-logo-title.svg" alt="Software Heritage logo" /></p><p>We quickly settled on a scheme where Guix would fall back to the
Software Heritage archive whenever it fails to download source code from
its original location. That way, package definitions don’t need to be
modified: they still refer to the original source code URL, but the
downloading machinery transparently goes to Software Heritage when
needed.</p><p>There are two types of source code downloads in Guix: tarball downloads,
and version control checkouts. In the former case, resorting to
Software Heritage is easy: Guix knows the SHA256 hash of the tarball so
it can look it up by hash using <a href="https://archive.softwareheritage.org/api/1/content/raw/">the <code>/content</code> endpoint of the
archive’s
interface</a>.</p><p>Fetching version control checkouts is more involved. In this case, the
downloader would first resolve the commit identifier to obtain a
<a href="https://archive.softwareheritage.org/api/1/revision/">Software Heritage
revision</a>. The
actual code for that revision is then fetched through the
<a href="https://docs.softwareheritage.org/devel/swh-vault/api.html"><em>vault</em></a>.</p><p>The vault conveniently allows users to fetch the tarball corresponding
to a revision. However, not all revisions are readily available as
tarballs (understandably), so the vault has an interface that allows you
to request the “<em>cooking</em>” of a specific revision. Cooking is
asynchronous and can take some time. Currently, if a revision hasn’t
been cooked yet, the Guix download machinery will request it and wait
until it is available. The process can take some time but will
eventually succeed.</p><p>Success! At this point, we have essentially bridged the gap between the
stable archive that Software Heritage provides and the reproducible
software deployment pipeline of Guix. This code <a href="https://issues.guix.info/issue/33432">was
integrated</a> in November 2018,
making it the first free software distribution backed by a stable
archive.</p><h1>The challenges ahead</h1><p>This milestone was encouraging: we had seemingly achieved our goal, but
we also knew of several shortcomings. First, even though the software
we package is still primarily distributed as tarballs, Software Heritage
keeps relatively few of these tarballs. Software Heritage does ingest
tarballs, notably those found on <a href="https://ftp.gnu.org/gnu/">the GNU
servers</a>, but the primary focus is on
preserving complete version control repositories rather than release
tarballs.</p><p>It is not yet clear to us what to do with plain old tarballs. On one
hand, they are here and cannot be ignored. Furthermore, some provide
artifacts that are not in version control, such as <code>configure</code> scripts,
and often enough they are accompanied by a cryptographic signature from
the developers that allows recipients to <em>authenticate</em> the code—an
important piece of information that’s often missing from version control
history. On the other hand, version control tags are increasingly
becoming the mechanism of choice to distribute software releases. It
may be that tags will become the primary mechanism for software release
distribution soon enough.</p><p>Version control tags turn out not to be ideal either, because they’re
mutable and per-repository. Conversely, Git commit identifiers are
unambiguous and repository-independent because they’re essentially
content-addressed, but our package definitions often refer to tags, not
commits, because that makes it clearer that we’re providing an actual
release and not an arbitrary revision (this is another illustration of
<a href="https://en.wikipedia.org/wiki/Zooko's_triangle">Zooko’s triangle</a>).</p><p>This leads to another limitation that stems from the mismatch between
the way Guix and Software Heritage compute hashes over version control
checkouts: both compute a hash over a serialized representation of the
directory, but they serialize the directory in a different way (SWH
serializes directories as Git trees, while Guix uses “normalized
archives” or Nars, the format the build daemon manipulates, which is
inherited from Nix.) That prevents Guix from looking up revisions by
content hash. The solution will probably involve changing Guix to
support the same method as Software Heritage, and/or adding Guix’s method
to Software Heritage.</p><p>Having to wait for “cooking” completion can also be problematic. The
Software Heritage team is investigating the possibility to
<a href="https://forge.softwareheritage.org/T1350">automatically cook all version control
tags</a>. That way, relevant
revisions would almost always be readily available through the vault.</p><p>Similarly, we have no guarantee that software provided by Guix is
available in the archive. Our plan is to <a href="https://forge.softwareheritage.org/T1352">extend Software
Heritage</a> such that it would
periodically archive the source code of software packaged by Guix.</p><h1>Going further</h1><p>In the process of adding support for Software Heritage, Guix <a href="https://issues.guix.info/issue/33432">gained
Guile bindings to the Software Heritage HTTP
interface</a>. Here’s a couple of
things we can do:</p><pre><code class="language-scheme">(use-modules (guix swh))
;; Check whether SWH has ever crawled our repository.
(define o (lookup-origin "https://git.savannah.gnu.org/git/guix.git"))
⇒ #<<origin> id: 86312956 …>
;; It did! When was its last visit?
(define last-visit
(first (origin-visits o)))
(date->string (visit-date last-visit))
⇒ "Fri Mar 29 10:07:45Z 2019"
;; Does it have our “v0.15.0” Git tag?
(lookup-origin-revision "https://git.savannah.gnu.org/git/guix.git" "v0.15.0")
⇒ #<<revision> id: "359fdda40f754bbf1b5dc261e7427b75463b59be" …></code></pre><p>Guix itself is a Guile library so when we combine the two, there are
interesting things we can do:</p><pre><code class="language-scheme">(use-modules (guix) (guix swh)
(gnu packages base)
(gnu packages golang))
;; This is our GNU Coreutils package.
coreutils
⇒ #<package coreutils@8.30 gnu/packages/base.scm:342 1c67b40>
;; Does SWH have its tarball?
(lookup-content (origin-sha256 (package-source coreutils))
"sha256")
⇒ #<<content> checksums: (("sha1" …)) data-url: …>
;; Our package for HashiCorp’s Configuration Language (HCL) is
;; built from a Git commit.
(define commit
(git-reference-commit
(origin-uri (package-source go-github-com-hashicorp-hcl))))
;; Is this particular commit available in the archive?
(lookup-revision commit)
⇒ #<<revision> id: "23c074d0eceb2b8a5bfdbb271ab780cde70f05a8" …></code></pre><p>We’re currently using a subset of this interface, but there’s certainly
more we could do. For example, we can compute archive coverage of the
Guix packages; we can also request the archival of each package’s source
code <em>via</em> the <a href="https://archive.softwareheritage.org/api/1/origin/save/">“save code”
interface</a>—though
all this is subject to <a href="https://archive.softwareheritage.org/api/#rate-limiting">rate
limiting</a>.</p><h1>Wrap-up</h1><p>Software Heritage support in Guix creates a bridge from the stable
source code archive to reproducible software deployment with complete
provenance tracking. For the first time it gives us a software package
distribution that can be rebuilt months or years later. This is
particularly beneficial in the context of reproducible science: finally
we can describe reproducible software environments, a prerequisite for
reproducible computational experiments.</p><p>Going further, we can provide a complete software supply tool chain with
provenance tracking that links revisions in the archive to
bit-reproducible build artifacts produced by Guix. Oh and Guix itself
<a href="https://archive.softwareheritage.org/api/1/origin/git/url/https://git.savannah.gnu.org/git/guix.git/">is
archived</a>,
so we have this meta-level where we could link Guix revisions to the
revisions of packages it provides… There are still technical challenges
to overcome, but that vision is shaping up.</p><h4>About GNU Guix</h4><p><a href="https://www.gnu.org/software/guix">GNU Guix</a> is a transactional package
manager and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects
user
freedom</a>.
Guix can be used on top of any system running the kernel Linux, or it
can be used as a standalone operating system distribution for i686,
x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2018/a-packaging-tutorial-for-guix//A packaging tutorial for GuixPierre Neidhardt2018-10-10T16:00:00Z2018-10-10T16:00:00Z Introduction GNU Guix stands out as the hackable package manager, mostly because it uses
GNU Guile , a powerful high-level programming language, one of the Scheme
dialects from the Lisp family . Package definitions are also written in Scheme, which empowers Guix in some very
unique ways, unlike most other package managers that use shell scripts or
simple languages. Use functions, structures, macros and all of Scheme expressiveness for your
package definitions. Inheritance makes it easy to customize a package by inheriting from it and
modifying only what is needed.…<h1>Introduction</h1><p>GNU Guix stands out as the <em>hackable</em> package manager, mostly because it uses
<a href="https://www.gnu.org/software/guile/">GNU Guile</a>, a powerful high-level programming language, one of the <a href="https://en.wikipedia.org/wiki/Scheme_(programming_language)">Scheme</a>
dialects from the <a href="https://en.wikipedia.org/wiki/Lisp_(programming_language)">Lisp family</a>.</p><p>Package definitions are also written in Scheme, which empowers Guix in some very
unique ways, unlike most other package managers that use shell scripts or
simple languages.</p><ul><li><p>Use functions, structures, macros and all of Scheme expressiveness for your
package definitions.</p></li><li><p>Inheritance makes it easy to customize a package by inheriting from it and
modifying only what is needed.</p></li><li><p>Batch processing: the whole package collection can be parsed, filtered and
processed. Building a headless server with all graphical interfaces stripped
out? It's possible. Want to rebuild everything from source using specific
compiler optimization flags? Pass the <code>#:make-flags "..."</code> argument to the
list of packages. It wouldn't be a stretch to think <a href="https://wiki.gentoo.org/wiki/USE_flag">Gentoo USE flags</a> here,
but this goes even further: the changes don't have to be thought out
beforehand by the packager, they can be <em>programmed</em> by the user!</p></li></ul><p>The following tutorial covers all the basics around package creation with Guix.
It does not assume much knowledge of the Guix system nor of the Lisp language.
The reader is only expected to be familiar with the command line and to have some
basic programming knowledge.</p><blockquote><p><strong>Update</strong>: Check out <a href="https://guix.gnu.org/cookbook/en/html_node/Packaging-Tutorial.html">the
Cookbook</a>
for an up-to-date and translated version of this tutorial!</p></blockquote><h1>A "Hello World" package</h1><p>The <a href="https://www.gnu.org/software/guix/manual/en/html_node/Defining-Packages.html">“Defining Packages” section of the manual</a> introduces the basics of Guix
packaging. In the following section, we will partly go over those basics again.</p><p><code>GNU hello</code> is a dummy project that serves as an idiomatic example for
packaging. It uses the GNU build system (<code>./configure && make && make install</code>).
Guix already provides a package definition which is a perfect example to start
with. You can look up its declaration with <code>guix edit hello</code> from the
command line. Let's see how it looks:</p><pre><code class="language-scheme">(define-public hello
(package
(name "hello")
(version "2.10")
(source (origin
(method url-fetch)
(uri (string-append "mirror://gnu/hello/hello-" version
".tar.gz"))
(sha256
(base32
"0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
(build-system gnu-build-system)
(synopsis "Hello, GNU world: An example GNU package")
(description
"GNU Hello prints the message \"Hello, world!\" and then exits. It
serves as an example of standard GNU coding practices. As such, it supports
command-line arguments, multiple languages, and so on.")
(home-page "https://www.gnu.org/software/hello/")
(license gpl3+)))</code></pre><p>As you can see, most of it is rather straightforward. But let's review the
fields together:</p><ul><li><strong>name:</strong> The project name. Using Scheme conventions, we prefer to keep it
lower case, without underscore and using dash-separated words.</li><li><p><strong>source:</strong> This field contains a description of the source code origin. The
<code>origin</code> record contains these fields:</p><ol><li>The method, here <code>url-fetch</code> to download via HTTP/FTP, but other methods
exist, such as <code>git-fetch</code> for Git repositories.</li><li>The URI, which is typically some <code>https://</code> location for <code>url-fetch</code>. Here
the special <code>mirror://gnu</code> refers to a set of well known locations, all of
which can be used by Guix to fetch the source, should some of them fail.</li><li>The <code>sha256</code> checksum of the requested file. This is essential to ensure
the source is not corrupted. Note that Guix works with base32 strings,
hence the call to the <code>base32</code> function.</li></ol></li><li><strong>build-system:</strong> This is where the power of abstraction provided by the Scheme
language really shines: in this case, the <code>gnu-build-system</code>
abstracts away the famous <code>./configure && make && make install</code> shell invocations. Other build systems include the
<code>trivial-build-system</code> which does not do anything and requires
from the packager to program all the build steps, the
<code>python-build-system</code>, the <code>emacs-build-system</code>, <a href="https://www.gnu.org/software/guix/manual/en/html_node/Build-Systems.html">and many
more</a>.</li><li><strong>synopsis:</strong> It should be a concise summary of what the package
does. For many packages a tagline from the
project's home page can be used as the synopsis.</li><li><strong>description:</strong> Same as for the synopsis, it's fine to re-use the project
description from the homepage. Note that Guix uses Texinfo
syntax.</li><li><strong>home-page:</strong> Use HTTPS if available.</li><li><strong>license:</strong> See <code>guix/licenses.scm</code> in the project source for a full list.</li></ul><p>Time to build our first package! Nothing fancy here for now: we will stick to a
dummy "my-hello", a copy of the above declaration.</p><p>As with the ritualistic "Hello World" taught with most programming languages,
this will possibly be the most "manual" approach. We will work out an ideal
setup later; for now we will go the simplest route.</p><p>Save the following to a file <code>my-hello.scm</code>.</p><pre><code class="language-scheme">(use-modules (guix packages)
(guix download)
(guix build-system gnu)
(guix licenses))
(package
(name "my-hello")
(version "2.10")
(source (origin
(method url-fetch)
(uri (string-append "mirror://gnu/hello/hello-" version
".tar.gz"))
(sha256
(base32
"0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
(build-system gnu-build-system)
(synopsis "Hello, Guix world: An example custom Guix package")
(description
"GNU Hello prints the message \"Hello, world!\" and then exits. It
serves as an example of standard GNU coding practices. As such, it supports
command-line arguments, multiple languages, and so on.")
(home-page "https://www.gnu.org/software/hello/")
(license gpl3+))</code></pre><p>We will explain the extra code in a moment.</p><p>Feel free to play with the different values of the various fields. If you
change the source, you'll need to update the checksum. Indeed, Guix refuses to
build anything if the given checksum does not match the computed checksum of the
source code. To obtain the correct checksum of the package declaration, we
need to download the source, compute the sha256 checksum and convert it to
base32.</p><p>Thankfully, Guix can automate this task for us; all we need is to provide the
URI:</p><pre><code class="language-sh">$ guix download mirror://gnu/hello/hello-2.10.tar.gz
Starting download of /tmp/guix-file.JLYgL7
From https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz...
following redirection to `https://mirror.ibcp.fr/pub/gnu/hello/hello-2.10.tar.gz'...
…10.tar.gz 709KiB 2.5MiB/s 00:00 [##################] 100.0%
/gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i</code></pre><p>In this specific case the output tells us which mirror was chosen.
If the result of the above command is not the same as in the above snippet,
update your <code>my-hello</code> declaration accordingly.</p><p>Note that GNU package tarballs come with an OpenPGP signature, so you
should definitely check the signature of this tarball with <code>gpg</code> to
authenticate it before going further:</p><pre><code class="language-sh">$ guix download mirror://gnu/hello/hello-2.10.tar.gz.sig
Starting download of /tmp/guix-file.03tFfb
From https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz.sig...
following redirection to `https://ftp.igh.cnrs.fr/pub/gnu/hello/hello-2.10.tar.gz.sig'...
….tar.gz.sig 819B 1.2MiB/s 00:00 [##################] 100.0%
/gnu/store/rzs8wba9ka7grrmgcpfyxvs58mly0sx6-hello-2.10.tar.gz.sig
0q0v86n3y38z17rl146gdakw9xc4mcscpk8dscs412j22glrv9jf
$ gpg --verify /gnu/store/rzs8wba9ka7grrmgcpfyxvs58mly0sx6-hello-2.10.tar.gz.sig /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
gpg: Signature made Sun 16 Nov 2014 01:08:37 PM CET
gpg: using RSA key A9553245FDE9B739
gpg: Good signature from "Sami Kerola <kerolasa@iki.fi>" [unknown]
gpg: aka "Sami Kerola (http://www.iki.fi/kerolasa/) <kerolasa@iki.fi>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 8ED3 96E3 7E38 D471 A005 30D3 A955 3245 FDE9 B739</code></pre><p>You can then happily run</p><pre><code class="language-sh">$ guix package --install-from-file=my-hello.scm</code></pre><p>You should now have <code>my-hello</code> in your profile!</p><pre><code class="language-sh">$ guix package --list-installed=my-hello
my-hello 2.10 out
/gnu/store/f1db2mfm8syb8qvc357c53slbvf1g9m9-my-hello-2.10</code></pre><p>We've gone as far as we could without any knowledge of Scheme. Now is the right
time to introduce the minimum we need from the language before we can proceed.</p><h1>A Scheme crash-course</h1><p>As we've seen above, basic packages don't require much Scheme knowledge, if
at all. But as you progress and your desire to write more and more complex
packages grows, it will become both necessary and empowering to hone your Lisper
skills.</p><p>Since an extensive Lisp course is very much out of the scope of this tutorial,
we will only cover some basics here.</p><blockquote><p><strong>Update</strong>: Check out <a href="https://guix.gnu.org/cookbook/en/html_node/A-Scheme-Crash-Course.html">the
Cookbook</a>
for an up-to-date and translated version of this introduction to Scheme.</p></blockquote><p>Guix uses the Guile implementation of Scheme. To start playing with the
language, install it with <code>guix package --install guile</code> and start a <a href="https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop">REPL</a> by
running <code>guile</code> from the command line.</p><p>Alternatively you can also run <code>guix environment --ad-hoc guile -- guile</code> if
you'd rather not have Guile installed in your user profile.</p><p>In the following examples we use the <code>></code> symbol to denote the REPL prompt, that
is, the line reserved for user input. See <a href="https://www.gnu.org/software/guile/manual/html_node/Using-Guile-Interactively.html">the Guile manual</a> for more details on
the REPL.</p><ul><li><p>Scheme syntax boils down to a tree of expressions (or <em>s-expression</em> in Lisp
lingo). An expression can be a literal such as numbers and strings, or a
compound which is a parenthesized list of compounds and literals. <code>#t</code> and
<code>#f</code> stand for the booleans "true" and "false", respectively.</p><p>Examples of valid expressions:</p><pre><code class="language-scheme">> "Hello World!"
"Hello World!"
> 17
17
> (display (string-append "Hello " "Guix" "\n"))
"Hello Guix!"</code></pre></li><li><p>This last example is a function call embedded in another function call. When
a parenthesized expression is evaluated, the first term is the function and
the rest are the arguments passed to the function. Every function returns the
last evaluated expression as value.</p></li><li><p>Anonymous functions are declared with the <code>lambda</code> term:</p><pre><code class="language-scheme">> (lambda (x) (* x x))
#<procedure 120e348 at <unknown port>:24:0 (x)></code></pre><p>The above lambda returns the square of its argument. Since everything is an
expression, the <code>lambda</code> expression returns an anonymous function, which can
in turn be applied to an argument:</p><pre><code class="language-scheme">> ((lambda (x) (* x x)) 3)
9</code></pre></li><li><p>Anything can be assigned a global name with <code>define</code>:</p><pre><code class="language-scheme">> (define a 3)
> (define square (lambda (x) (* x x)))
> (square a)
9</code></pre></li><li><p>Procedures can be defined more concisely with the following syntax:</p><pre><code class="language-scheme">(define (square x) (* x x))</code></pre></li><li><p>A list structure can be created with the <code>list</code> procedure:</p><pre><code class="language-scheme">> (list 2 a 5 7)
(2 3 5 7)</code></pre></li><li><p>The <em>quote</em> disables evaluation of a parenthesized expression: the first term
is not called over the other terms. Thus it effectively returns a list of
terms.</p><pre><code class="language-scheme">> '(display (string-append "Hello " "Guix" "\n"))
(display (string-append "Hello " "Guix" "\n"))
> '(2 a 5 7)
(2 a 5 7)</code></pre></li><li><p>The <em>quasiquote</em> disables evaluation of a parenthesized expression until a
comma re-enables it. Thus it provides us with fine-grained control over what
is evaluated and what is not.</p><pre><code class="language-scheme">> `(2 a 5 7 (2 ,a 5 ,(+ a 4)))
(2 a 5 7 (2 3 5 7))</code></pre><p>Note that the above result is a list of mixed elements: numbers, symbols (here
<code>a</code>) and the last element is a list itself.</p></li><li><p>Multiple variables can be named locally with <code>let</code>:</p><pre><code class="language-scheme">> (define x 10)
> (let ((x 2)
(y 3))
(list x y))
(2 3)
> x
10
> y
ERROR: In procedure module-lookup: Unbound variable: y</code></pre><p>Use <code>let*</code> to allow later variable declarations to refer to earlier
definitions.</p><pre><code class="language-scheme">> (let* ((x 2)
(y (* x 3)))
(list x y))
(2 6)</code></pre></li><li><p>The keyword syntax is <code>#:</code>, it is used to create unique identifiers. See also
the <a href="https://www.gnu.org/software/guile/manual/html_node/Keywords.html">Keywords section in the Guile manual</a>.</p></li><li><p>The percentage <code>%</code> is typically used for read-only global variables in the
build stage. Note that it is merely a convention, like <code>_</code> in C. Scheme Lisp
treats <code>%</code> exactly the same as any other letter.</p></li><li><p>Modules are created with <code>define-module</code>. For instance</p><pre><code class="language-scheme">(define-module (guix build-system ruby)
#:use-module (guix store)
#:export (ruby-build
ruby-build-system))</code></pre><p>defines the module <code>ruby</code> which must be located in
<code>guix/build-system/ruby.scm</code> somewhere in <code>GUILE_LOAD_PATH</code>. It depends on
the <code>(guix store)</code> module and it exports two symbols, <code>ruby-build</code> and
<code>ruby-build-system</code>.</p></li></ul><p>For a more detailed introduction, check out <a href="http://www.troubleshooters.com/codecorn/scheme_guile/hello.htm">Scheme at a Glance</a>, by Steve Litt.</p><p>One of the reference Scheme books is the seminal <em>Structure and Interpretation
of Computer Programs</em>, by Harold Abelson and Gerald Jay Sussman, with Julie
Sussman. You'll find a free copy <a href="https://mitpress.mit.edu/sites/default/files/sicp/index.html">online</a>, together with <a href="https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-001-structure-and-interpretation-of-computer-programs-spring-2005/video-lectures/">videos of the lectures
by the authors</a>. The book is available in Texinfo format as the <code>sicp</code> Guix
package. Go ahead, run <code>guix package --install sicp</code> and start reading with
<code>info sicp</code> (or with the Emacs Info reader). An unofficial ebook <a href="https://sarabander.github.io/sicp/">is also
available</a>.</p><p>You'll find more books, tutorials and other resources at <a href="https://schemers.org/">https://schemers.org/</a>.</p><h1>Setup</h1><p>Now that we know some Scheme basics we can detail the different possible setups
for working on Guix packages.</p><p>There are several ways to set up a Guix packaging environment.</p><p>We recommend you work directly on the Guix source checkout since it makes it
easier for everyone to contribute to the project.</p><p>But first, let's look at other possibilities.</p><h3>Local file</h3><p>This is what we previously did with <code>my-hello</code>. With the Scheme basics we've
covered, we are now able to explain the leading chunks. As stated in <code>guix package --help</code>:</p><pre><code class="language-sh">-f, --install-from-file=FILE
install the package that the code within FILE
evaluates to</code></pre><p>Thus the last expression <em>must</em> return a package, which is the case in our
earlier example.</p><p>The <code>use-modules</code> expression tells which of the modules we need in the file.
Modules are a collection of values and procedures. They are commonly called
"libraries" or "packages" in other programming languages.</p><h3>GUIX_PACKAGE_PATH</h3><p><em>Note: Starting from Guix 0.16, the more flexible Guix "channels" are the
preferred way and supersede <code>GUIX_PACKAGE_PATH</code>. See below.</em></p><p>It can be tedious to specify the file from the command line instead of simply
calling <code>guix package --install my-hello</code> as you would do with the official
packages.</p><p>Guix makes it possible to streamline the process by adding as many "package
declaration paths" as you want.</p><p>Create a directory, say <code>~./guix-packages</code> and add it to the <code>GUIX_PACKAGE_PATH</code>
environment variable:</p><pre><code class="language-sh">$ mkdir ~/guix-packages
$ export GUIX_PACKAGE_PATH=~/guix-packages</code></pre><p>To add several directories, separate them with a colon (<code>:</code>).</p><p>Our previous <code>my-hello</code> needs some adjustments though:</p><pre><code class="language-scheme">(define-module (my-hello)
#:use-module (guix licenses)
#:use-module (guix packages)
#:use-module (guix build-system gnu)
#:use-module (guix download))
(define-public my-hello
(package
(name "my-hello")
(version "2.10")
(source (origin
(method url-fetch)
(uri (string-append "mirror://gnu/hello/hello-" version
".tar.gz"))
(sha256
(base32
"0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
(build-system gnu-build-system)
(synopsis "Hello, Guix world: An example custom Guix package")
(description
"GNU Hello prints the message \"Hello, world!\" and then exits. It
serves as an example of standard GNU coding practices. As such, it supports
command-line arguments, multiple languages, and so on.")
(home-page "https://www.gnu.org/software/hello/")
(license gpl3+)))</code></pre><p>Note that we have assigned the package value to an exported variable name with
<code>define-public</code>. This is effectively assigning the package to the <code>my-hello</code>
variable so that it can be referenced, among other as dependency of other
packages.</p><p>If you use <code>guix package --install-from-file=my-hello.scm</code> on the above file, it
will fail because the last expression, <code>define-public</code>, does not return a
package. If you want to use <code>define-public</code> in this use-case nonetheless, make
sure the file ends with an evaluation of <code>my-hello</code>:</p><pre><code class="language-scheme">; ...
(define-public my-hello
; ...
)
my-hello</code></pre><p>This last example is not very typical.</p><p>Now <code>my-hello</code> should be part of the package collection like all other official
packages. You can verify this with:</p><pre><code class="language-sh">$ guix package --show=my-hello</code></pre><h3>Guix channels</h3><p>Guix 0.16 features channels, which is very similar to <code>GUIX_PACKAGE_PATH</code> but
provides better integration and provenance tracking. Channels are not
necessarily local, they can be maintained as a public Git repository for
instance. Of course, several channels can be used at the same time.</p><p>See the <a href="http://guix.info/manual/en/Channels.html">“Channels” section in the manual</a> for setup details.</p><h3>Direct checkout hacking</h3><p>Working directly on the Guix project is recommended: it reduces the friction
when the time comes to submit your changes upstream to let the community benefit
from your hard work!</p><p>Unlike most software distributions, the Guix repository holds in one place both
the tooling (including the package manager) and the package definitions. This
choice was made so that it would give developers the flexibility to modify the
API without breakage by updating all packages at the same time. This reduces
development inertia.</p><p>Check out the official <a href="https://git-scm.com/">Git</a> repository:</p><pre><code class="language-sh">$ git clone https://git.savannah.gnu.org/git/guix.git</code></pre><p>In the rest of this article, we use <code>$GUIX_CHECKOUT</code> to refer to the location of
the checkout.</p><p>Follow the instruction from the <a href="https://www.gnu.org/software/guix/manual/en/html_node/Contributing.html">"Contributing" chapter</a> in the manual to set up the
repository environment.</p><p>Once ready, you should be able to use the package definitions from the
repository environment.</p><p>Feel free to edit package definitions found in <code>$GUIX_CHECKOUT/gnu/packages</code>.</p><p>The <code>$GUIX_CHECKOUT/pre-inst-env</code> script lets you use <code>guix</code> over the package
collection of the repository.</p><ul><li><p>Search packages, such as Ruby:</p><pre><code class="language-sh">$ cd $GUIX_CHECKOUT
$ ./pre-inst-env guix package --list-available=ruby
ruby 1.8.7-p374 out gnu/packages/ruby.scm:119:2
ruby 2.1.6 out gnu/packages/ruby.scm:91:2
ruby 2.2.2 out gnu/packages/ruby.scm:39:2</code></pre></li><li><p>Build a package, here Ruby version 2.1:</p><pre><code class="language-sh">$ ./pre-inst-env guix build --keep-failed ruby@2.1
/gnu/store/c13v73jxmj2nir2xjqaz5259zywsa9zi-ruby-2.1.6</code></pre></li><li><p>Install it to your user profile:</p><pre><code class="language-sh">$ ./pre-inst-env guix package --install ruby@2.1</code></pre></li><li><p>Check for common mistakes:</p><pre><code class="language-sh">$ ./pre-inst-env guix lint ruby@2.1</code></pre></li></ul><p>Guix strives at maintaining a high packaging standard; when contributing to the
Guix project, remember to</p><ul><li>follow the <a href="https://www.gnu.org/software/guix/manual/en/html_node/Coding-Style.html">coding style</a>,</li><li>and review the <a href="https://www.gnu.org/software/guix/manual/en/html_node/Submitting-Patches.html">check list</a> from the manual.</li></ul><p>Once you are happy with the result, you are welcome to send your contribution to
make it part of Guix. This process is also detailed in the <a href="https://www.gnu.org/software/guix/manual/en/html_node/Contributing.html">manual</a>.</p><p>It's a community effort so the more join in, the better Guix becomes!</p><h1>Extended example</h1><p>The above "Hello World" example is as simple as it goes. Packages can be more
complex than that and Guix can handle more advanced scenarios. Let's look at
another, more sophisticated package (slightly modified from the source):</p><pre><code class="language-scheme">(define-module (gnu packages version-control)
#:use-module ((guix licenses) #:prefix license:)
#:use-module (guix utils)
#:use-module (guix packages)
#:use-module (guix git-download)
#:use-module (guix build-system cmake)
#:use-module (gnu packages ssh)
#:use-module (gnu packages web)
#:use-module (gnu packages pkg-config)
#:use-module (gnu packages python)
#:use-module (gnu packages compression)
#:use-module (gnu packages tls))
(define-public my-libgit2
(let ((commit "e98d0a37c93574d2c6107bf7f31140b548c6a7bf")
(revision "1"))
(package
(name "my-libgit2")
(version (git-version "0.26.6" revision commit))
(source (origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/libgit2/libgit2/")
(commit commit)))
(file-name (git-file-name name version))
(sha256
(base32
"17pjvprmdrx4h6bb1hhc98w9qi6ki7yl57f090n9kbhswxqfs7s3"))
(patches (search-patches "libgit2-mtime-0.patch"))
(modules '((guix build utils)))
(snippet '(begin
;; Remove bundled software.
(delete-file-recursively "deps")
#t))))
(build-system cmake-build-system)
(outputs '("out" "debug"))
(arguments
`(#:tests? #t ; Run the test suite (this is the default)
#:configure-flags '("-DUSE_SHA1DC=ON") ; SHA-1 collision detection
#:phases
(modify-phases %standard-phases
(add-after 'unpack 'fix-hardcoded-paths
(lambda _
(substitute* "tests/repo/init.c"
(("#!/bin/sh") (string-append "#!" (which "sh"))))
(substitute* "tests/clar/fs.h"
(("/bin/cp") (which "cp"))
(("/bin/rm") (which "rm")))
#t))
;; Run checks more verbosely.
(replace 'check
(lambda _ (invoke "./libgit2_clar" "-v" "-Q")))
(add-after 'unpack 'make-files-writable-for-tests
(lambda _ (for-each make-file-writable (find-files "." ".*")))))))
(inputs
`(("libssh2" ,libssh2)
("http-parser" ,http-parser)
("python" ,python-wrapper)))
(native-inputs
`(("pkg-config" ,pkg-config)))
(propagated-inputs
;; These two libraries are in 'Requires.private' in libgit2.pc.
`(("openssl" ,openssl)
("zlib" ,zlib)))
(home-page "https://libgit2.github.com/")
(synopsis "Library providing Git core methods")
(description
"Libgit2 is a portable, pure C implementation of the Git core methods
provided as a re-entrant linkable library with a solid API, allowing you to
write native speed custom Git applications in any language with bindings.")
;; GPLv2 with linking exception
(license license:gpl2))))</code></pre><p>(In those cases were you only want to tweak a few fields from a package
definition, you should rely on inheritance instead of copy-pasting everything.
See below.)</p><p>Let's discuss those fields in depth.</p><h3><code>git-fetch</code> method</h3><p>Unlike the <code>url-fetch</code> method, <code>git-fetch</code> expects a <code>git-reference</code> which takes
a Git repository and a commit. The commit can be any Git reference such as
tags, so if the <code>version</code> is tagged, then it can be used directly. Sometimes
the tag is prefixed with a <code>v</code>, in which case you'd use <code>(commit (string-append "v" version))</code>.</p><p>To ensure that the source code from the Git repository is stored in a unique
directory with a readable name we use <code>(file-name (git-file-name name version))</code>.</p><p>Note that there is also a <code>git-version</code> procedure that can be used to derive the
version when packaging programs for a specific commit.</p><h3>Snippets</h3><p>Snippets are quoted (i.e. non-evaluated) Scheme code that are a means of patching
the source. They are a Guix-y alternative to the traditional <code>.patch</code> files.
Because of the quote, the code in only evaluated when passed to the Guix daemon
for building.</p><p>There can be as many snippet as needed.</p><p>Snippets might need additional Guile modules which can be imported from the
<code>modules</code> field.</p><h3>Inputs</h3><p>First, a syntactic comment: See the quasi-quote / comma syntax?</p><pre><code class="language-scheme">(native-inputs
`(("pkg-config" ,pkg-config)))</code></pre><p>is equivalent to</p><pre><code class="language-scheme">(native-inputs
(list (list "pkg-config" pkg-config)))</code></pre><p>You'll mostly see the former because it's shorter.</p><p>There are 3 different input types. In short:</p><ul><li><strong>native-inputs:</strong> Required for building but not runtime – installing a package
through a substitute won't install these inputs.</li><li><strong>inputs:</strong> Installed in the store but not in the profile, as well as being
present at build time.</li><li><strong>propagated-inputs:</strong> Installed in the store and in the profile, as well as
being present at build time.</li></ul><p>See <a href="https://www.gnu.org/software/guix/manual/en/html_node/package-Reference.html">the package reference in the manual</a> for more details.</p><p>The distinction between the various inputs is important: if a dependency can be
handled as an <em>input</em> instead of a <em>propagated input</em>, it should be done so, or
else it "pollutes" the user profile for no good reason.</p><p>For instance, a user installing a graphical program that depends on a
command line tool might only be interested in the graphical part, so there is no
need to force the command line tool into the user profile. The dependency is a
concern to the package, not to the user. <em>Inputs</em> make it possible to handle
dependencies without bugging the user by adding undesired executable files (or
libraries) to their profile.</p><p>Same goes for <em>native-inputs</em>: once the program is installed, build-time
dependencies can be safely garbage-collected.
It also matters when a substitute is available, in which case only the <em>inputs</em>
and <em>propagated inputs</em> will be fetched: the <em>native inputs</em> are not required to
install a package from a substitute.</p><h3>Outputs</h3><p>Just like how a package can have multiple inputs, it can also produce multiple
outputs.</p><p>Each output corresponds to a separate directory in the store.</p><p>The user can choose which output to install; this is useful to save space or
to avoid polluting the user profile with unwanted executables or libraries.</p><p>Output separation is optional. When the <code>outputs</code> field is left out, the
default and only output (the complete package) is referred to as <code>"out"</code>.</p><p>Typical separate output names include <code>debug</code> and <code>doc</code>.</p><p>It's advised to separate outputs only when you've shown it's worth it: if the
output size is significant (compare with <code>guix size</code>) or in case the package is
modular.</p><h3>Build system arguments</h3><p>The <code>arguments</code> is a keyword-value list used to configure the build process.</p><p>The simplest argument <code>#:tests?</code> can be used to disable the test suite when
building the package. This is mostly useful when the package does not feature
any test suite. It's strongly recommended to keep the test suite on if there is
one.</p><p>Another common argument is <code>:make-flags</code>, which specifies a list of flags to
append when running make, as you would from the command line. For instance, the
following flags</p><pre><code class="language-scheme">#:make-flags (list (string-append "prefix=" (assoc-ref %outputs "out"))
"CC=gcc")</code></pre><p>translate into</p><pre><code class="language-sh">$ make CC=gcc prefix=/gnu/store/...-<out></code></pre><p>This sets the C compiler to <code>gcc</code> and the <code>prefix</code> variable (the installation
directory in Make parlance) to <code>(assoc-ref %outputs "out")</code>, which is a build-stage
global variable pointing to the destination directory in the store (something like
<code>/gnu/store/...-my-libgit2-20180408</code>).</p><p>Similarly, it's possible to set the "configure" flags.</p><pre><code class="language-scheme">#:configure-flags '("-DUSE_SHA1DC=ON")</code></pre><p>The <code>%build-inputs</code> variable is also generated in scope. It's an association
table that maps the input names to their store directories.</p><p>The <code>phases</code> keyword lists the sequential steps of the build system. Typically
phases include <code>unpack</code>, <code>configure</code>, <code>build</code>, <code>install</code> and <code>check</code>. To know
more about those phases, you need to work out the appropriate build system
definition in <code>$GUIX_CHECKOUT/guix/build/gnu-build-system.scm</code>:</p><pre><code class="language-scheme">(define %standard-phases
;; Standard build phases, as a list of symbol/procedure pairs.
(let-syntax ((phases (syntax-rules ()
((_ p ...) `((p . ,p) ...)))))
(phases set-SOURCE-DATE-EPOCH set-paths install-locale unpack
bootstrap
patch-usr-bin-file
patch-source-shebangs configure patch-generated-file-shebangs
build check install
patch-shebangs strip
validate-runpath
validate-documentation-location
delete-info-dir-file
patch-dot-desktop-files
install-license-files
reset-gzip-timestamps
compress-documentation)))</code></pre><p>Or from the REPL:</p><pre><code class="language-scheme">> (add-to-load-path "/path/to/guix/checkout")
> ,module (guix build gnu-build-system)
> (map first %standard-phases)
(set-SOURCE-DATE-EPOCH set-paths install-locale unpack bootstrap
patch-usr-bin-file patch-source-shebangs configure
patch-generated-file-shebangs build check install patch-shebangs strip
validate-runpath validate-documentation-location delete-info-dir-file
patch-dot-desktop-files install-license-files reset-gzip-timestamps
compress-documentation)</code></pre><p>If you want to know more about what happens during those phases, consult the
associated procedures.</p><p>For instance, as of this writing the definition of <code>unpack</code> for the GNU build
system is</p><pre><code class="language-scheme">(define* (unpack #:key source #:allow-other-keys)
"Unpack SOURCE in the working directory, and change directory within the
source. When SOURCE is a directory, copy it in a sub-directory of the current
working directory."
(if (file-is-directory? source)
(begin
(mkdir "source")
(chdir "source")
;; Preserve timestamps (set to the Epoch) on the copied tree so that
;; things work deterministically.
(copy-recursively source "."
#:keep-mtime? #t))
(begin
(if (string-suffix? ".zip" source)
(invoke "unzip" source)
(invoke "tar" "xvf" source))
(chdir (first-subdirectory "."))))
#t)</code></pre><p>Note the <code>chdir</code> call: it changes the working directory to where the source was
unpacked.
Thus every phase following the <code>unpack</code> will use the source as a working
directory, which is why we can directly work on the source files.
That is to say, unless a later phase changes the working directory to something
else.</p><p>We modify the list of <code>%standard-phases</code> of the build system with the
<code>modify-phases</code> macro as per the list of specified modifications, which may have
the following forms:</p><ul><li><code>(add-before PHASE NEW-PHASE PROCEDURE)</code>: Run <code>PROCEDURE</code> named <code>NEW-PHASE</code> before <code>PHASE</code>.</li><li><code>(add-after PHASE NEW-PHASE PROCEDURE)</code>: Same, but afterwards.</li><li><code>(replace PHASE PROCEDURE)</code>.</li><li><code>(delete PHASE)</code>.</li></ul><p>The <code>PROCEDURE</code> supports the keyword arguments <code>inputs</code> and <code>outputs</code>. Each
input (whether <em>native</em>, <em>propagated</em> or not) and output directory is referenced
by their name in those variables. Thus <code>(assoc-ref outputs "out")</code> is the store
directory of the main output of the package. A phase procedure may look like
this:</p><pre><code class="language-scheme">(lambda* (#:key inputs outputs #:allow-other-keys)
(let (((bash-directory (assoc-ref inputs "bash"))
(output-directory (assoc-ref outputs "out"))
(doc-directory (assoc-ref outputs "doc"))
; ...
#t)</code></pre><p>The procedure must return <code>#t</code> on success. It's brittle to rely on the return
value of the last expression used to tweak the phase because there is no
guarantee it would be a <code>#t</code>. Hence the trailing <code>#t</code> to ensure the right value
is returned on success.</p><h3>Code staging</h3><p>The astute reader may have noticed the quasi-quote and comma syntax in the
argument field. Indeed, the build code in the package declaration should not be
evaluated on the client side, but only when passed to the Guix daemon. This
mechanism of passing code around two running processes is called <a href="https://arxiv.org/abs/1709.00833">code staging</a>.</p><h3>"Utils" functions</h3><p>When customizing <code>phases</code>, we often need to write code that mimics the
equivalent system invocations (<code>make</code>, <code>mkdir</code>, <code>cp</code>, etc.) commonly used during
regular "Unix-style" installations.</p><p>Some like <code>chmod</code> are native to Guile. See the <a href="https://www.gnu.org/software/guile/manual/html_node">Guile reference manual</a> for a
complete list.</p><p>Guix provides additional helper functions which prove especially handy in the
context of package management.</p><p>Some of those functions can be found in
<code>$GUIX_CHECKOUT/guix/guix/build/utils.scm</code>. Most of them mirror the behaviour
of the traditional Unix system commands:</p><ul><li><strong>which:</strong> Like the <code>which</code> system command.</li><li><strong>find-files:</strong> Akin to the <code>find</code> system command.</li><li><strong>mkdir-p:</strong> Like <code>mkdir -p</code>, which creates all parents as needed.</li><li><strong>install-file:</strong> Similar to <code>install</code> when installing a file to a (possibly
non-existing) directory. Guile has <code>copy-file</code> which works
like <code>cp</code>.</li><li><strong>copy-recursively:</strong> Like <code>cp -r</code>.</li><li><strong>delete-file-recursively:</strong> Like <code>rm -rf</code>.</li><li><strong>invoke:</strong> Run an executable. This should be used instead of <code>system*</code>.</li><li><strong>with-directory-excursion:</strong> Run the body in a different working directory,
then restore the previous working directory.</li><li><strong>substitute*:</strong> A "sed-like" function.</li></ul><h3>Module prefix</h3><p>The license in our last example needs a prefix: this is because of how the
<code>license</code> module was imported in the package, as <code>#:use-module ((guix licenses) #:prefix license:)</code>. The <a href="https://www.gnu.org/software/guile/manual/html_node/Using-Guile-Modules.html">Guile module import mechanism</a> gives the user full
control over namespacing: this is needed to avoid clashes between, say, the
<code>zlib</code> variable from <code>licenses.scm</code> (a <em>license</em> value) and the <code>zlib</code> variable
from <code>compression.scm</code> (a <em>package</em> value).</p><h1>Other build systems</h1><p>What we've seen so far covers the majority of packages using a build system
other than the <code>trivial-build-system</code>. The latter does not automate anything
and leaves you to build everything manually. This can be more demanding and we
won't cover it here for now, but thankfully it is rarely necessary to fall back
on this system.</p><p>For the other build systems, such as ASDF, Emacs, Perl, Ruby and many more, the
process is very similar to the GNU build system except for a few specialized
arguments.</p><p>Learn more about build systems in</p><ul><li><a href="https://www.gnu.org/software/guix/manual/en/html_node/Build-Systems.html#Build-Systems">the manual, section 4.2 Build systems</a>,</li><li>the source code in the <code>$GUIX_CHECKOUT/guix/build</code> and
<code>$GUIX_CHECKOUT/guix/build-system</code> directories.</li></ul><h1>Programmable and automated package definition</h1><p>We can't repeat it enough: having a full-fledged programming language at hand
empowers us in ways that reach far beyond traditional package management.</p><p>Let's illustrate this with some awesome features of Guix!</p><h3>Recursive importers</h3><p>You might find some build systems good enough that there is little to do at all
to write a package, to the point that it becomes repetitive and tedious after a
while. A <em>raison d'être</em> of computers is to replace human beings at those
boring tasks. So let's tell Guix to do this for us and create the package
definition of an R package from CRAN (the output is trimmed for conciseness):</p><pre><code class="language-sh">$ guix import cran --recursive walrus
(define-public r-mc2d
; ...
(license gpl2+)))
(define-public r-jmvcore
; ...
(license gpl2+)))
(define-public r-wrs2
; ...
(license gpl3)))
(define-public r-walrus
(package
(name "r-walrus")
(version "1.0.3")
(source
(origin
(method url-fetch)
(uri (cran-uri "walrus" version))
(sha256
(base32
"1nk2glcvy4hyksl5ipq2mz8jy4fss90hx6cq98m3w96kzjni6jjj"))))
(build-system r-build-system)
(propagated-inputs
`(("r-ggplot2" ,r-ggplot2)
("r-jmvcore" ,r-jmvcore)
("r-r6" ,r-r6)
("r-wrs2" ,r-wrs2)))
(home-page "https://github.com/jamovi/walrus")
(synopsis "Robust Statistical Methods")
(description
"This package provides a toolbox of common robust statistical tests, including robust descriptives, robust t-tests, and robust ANOVA. It is also available as a module for 'jamovi' (see <https://www.jamovi.org> for more information). Walrus is based on the WRS2 package by Patrick Mair, which is in turn based on the scripts and work of Rand Wilcox. These analyses are described in depth in the book 'Introduction to Robust Estimation & Hypothesis Testing'.")
(license gpl3)))</code></pre><p>The recursive importer won't import packages for which Guix already has package
definitions, except for the very first.</p><p>Not all applications can be packaged this way, only those relying on a select
number of supported systems. Read about the full list of importers in the <a href="https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-import.html">guix
import section</a> of the manual.</p><h3>Automatic update</h3><p>Guix can be smart enough to check for updates on systems it knows. It can
report outdated package definitions with</p><pre><code class="language-sh">$ guix refresh hello</code></pre><p>In most cases, updating a package to a newer version requires little more than
changing the version number and the checksum. Guix can do that automatically as
well:</p><pre><code class="language-sh">$ guix refresh hello --update</code></pre><h3>Inheritance</h3><p>If you've started browsing the existing package definitions, you might have
noticed that a significant number of them have a <code>inherit</code> field:</p><pre><code class="language-scheme">(define-public adwaita-icon-theme
(package (inherit gnome-icon-theme)
(name "adwaita-icon-theme")
(version "3.26.1")
(source (origin
(method url-fetch)
(uri (string-append "mirror://gnome/sources/" name "/"
(version-major+minor version) "/"
name "-" version ".tar.xz"))
(sha256
(base32
"17fpahgh5dyckgz7rwqvzgnhx53cx9kr2xw0szprc6bnqy977fi8"))))
(native-inputs
`(("gtk-encode-symbolic-svg" ,gtk+ "bin")))))</code></pre><p>All unspecified fields are inherited from the parent package. This is very
convenient to create alternative packages, for instance with different source,
version or compilation options.</p><h1>Getting help</h1><p>Sadly, some applications can be tough to package. Sometimes they need a patch to
work with the non-standard filesystem hierarchy enforced by the store.
Sometimes the tests won't run properly. (They can be skipped but this is not
recommended.) Other times the resulting package won't be reproducible.</p><p>Should you be stuck, unable to figure out how to fix any sort of packaging
issue, don't hesitate to ask the community for help.</p><p>See the <a href="https://www.gnu.org/software/guix/contact/">Guix homepage</a> for information on the mailing lists, IRC, etc.</p><h1>Conclusion</h1><p>This tutorial was a showcase of the sophisticated package management that Guix
boasts. At this point we have mostly restricted this introduction to the
<code>gnu-build-system</code> which is a core abstraction layer on which more advanced
abstractions are based.</p><p>Where do we go from here? Next we ought to dissect the innards of the build
system by removing all abstractions, using the <code>trivial-build-system</code>: this
should give us a thorough understanding of the process before investigating some
more advanced packaging techniques and edge cases.</p><p>Other features worth exploring are the interactive editing and debugging
capabilities of Guix provided by the Guile REPL.</p><p>Those fancy features are completely optional and can wait; now is a good time
to take a well-deserved break. With what we've introduced here you should be
well armed to package lots of programs. You can get started right away and
hopefully we will see your contributions soon!</p><h1>References</h1><ul><li><p>The <a href="https://www.gnu.org/software/guix/manual/en/html_node/Defining-Packages.html">package reference in the manual</a></p></li><li><p><a href="https://gitlab.com/pjotrp/guix-notes/blob/master/HACKING.org">Pjotr’s hacking guide to GNU Guix</a></p></li><li><p><a href="https://www.gnu.org/software/guix/guix-ghm-andreas-20130823.pdf">"GNU Guix: Package without a
scheme!"</a>,
by Andreas Enge</p></li></ul><h1>Notices</h1><p><a href="http://creativecommons.org/licenses/by-sa/4.0/"><img src="https://i.creativecommons.org/l/by-sa/4.0/88x31.png" alt="CC-BY-SA" title="CC-BY-SA
4.0" /></a></p><p>This work is licensed under a Creative Commons Attribution-ShareAlike 4.0
International License.</p><h4>About GNU Guix</h4><p><a href="https://www.gnu.org/software/guix">GNU Guix</a> is a transactional package
manager for the GNU system. The Guix System Distribution or GuixSD is
an advanced distribution of the GNU system that relies on GNU Guix and
<a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects the user's
freedom</a>.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. Guix uses low-level
mechanisms from the Nix package manager, except that packages are
defined as native <a href="https://www.gnu.org/software/guile">Guile</a> modules,
using extensions to the <a href="http://schemers.org">Scheme</a> language. GuixSD
offers a declarative approach to operating system configuration
management, and is highly customizable and hackable.</p><p>GuixSD can be used on an i686, x86_64, ARMv7, and AArch64 machines. It
is also possible to use Guix on top of an already installed GNU/Linux
system, including on mips64el and aarch64.</p>https://guix.gnu.org/blog/2018/multi-dimensional-transactions-and-rollbacks-oh-my//Multi-dimensional transactions and rollbacks, oh my!Ludovic Courtès2018-07-24T14:30:00Z2018-07-24T14:30:00Z One of the highlights of version
0.15.0
was the overhaul of guix pull ,
the command that updates Guix and its package collection. In Debian
terms, you can think of guix pull as: apt-get update && apt-get install apt Let’s be frank, guix pull does not yet run as quickly as this
apt-get command—in the “best case”, when pre-built binaries are
available, it currently runs in about 1m30s on a recent laptop. More
about the performance story in a future post… One of the key features of…<p>One of the <a href="https://www.gnu.org/software/guix/blog/2018/gnu-guix-and-guixsd-0.15.0-released/">highlights of version
0.15.0</a>
was the overhaul of <a href="https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-pull.html"><code>guix pull</code></a>,
the command that updates Guix and its package collection. In Debian
terms, you can think of <code>guix pull</code> as:</p><pre><code>apt-get update && apt-get install apt</code></pre><p>Let’s be frank, <code>guix pull</code> does not yet run as quickly as this
<code>apt-get</code> command—in the “best case”, when pre-built binaries are
available, it currently runs in about 1m30s on a recent laptop. More
about the performance story in a future post…</p><p>One of the key features of the new <code>guix pull</code> is the ability to <em>roll
back</em> to previous versions of Guix. That’s a distinguishing feature
that opens up new possibilities.</p><h1>“Profile generations”</h1><p>Transactional upgrades and rollbacks have been a distinguishing feature
of Guix since Day 1. They come for free as a consequence of the
functional package management model inherited from the Nix package
manager. To many users, this alone is enough to justify using a
functional package manager: if an upgrade goes wrong, you can always
roll back. Let’s recap how this all works.</p><p>As a user, you install packages in your own <em>profile</em>, which defaults to
<code>~/.guix-profile</code>. Then from time to time you update Guix and its
package collection:</p><pre><code>$ guix pull</code></pre><p>This updates <code>~/.config/guix/current</code>, giving you an updated <code>guix</code>
executable along with an updated set of packages. You can now upgrade
the packages that are in your profile:</p><pre><code>$ guix package -u
The following packages will be upgraded:
diffoscope 93 → 96 /gnu/store/…-diffoscope-96
emacs 25.3 → 26.1 /gnu/store/…-emacs-26.1
gimp 2.8.22 → 2.10.4 /gnu/store/…-gimp-2.10.4
gnupg 2.2.7 → 2.2.9 /gnu/store/…-gnupg-2.2.9</code></pre><p>The upgrade creates a new <em>generation</em> of your profile—the previous
generation of your profile, with diffoscope 93, emacs 25.3, and so on is
still around. You can list profile generations:</p><pre><code>$ guix package --list-generations
Generation 1 Jun 08 2018 20:06:21
diffoscope 93 out /gnu/store/…-diffoscope-93
emacs 25.3 out /gnu/store/…-emacs-25.3
gimp 2.8.22 out /gnu/store/…-gimp-2.8.22
gnupg 2.2.7 out /gnu/store/…-gnupg-2.2.7
python 3.6.5 out /gnu/store/…-python-3.6.5
Generation 2 Jul 12 2018 12:42:08 (current)
- diffoscope 93 out /gnu/store/…-diffoscope-93
- emacs 25.3 out /gnu/store/…-emacs-25.3
- gimp 2.8.22 out /gnu/store/…-gimp-2.8.22
- gnupg 2.2.7 out /gnu/store/…-gnupg-2.2.7
+ diffoscope 96 out /gnu/store/…-diffoscope-96
+ emacs 26.1 out /gnu/store/…-emacs-26.1
+ gimp 2.10.4 out /gnu/store/…-gimp-2.10.4
+ gnupg 2.2.9 out /gnu/store/…-gnupg-2.2.9</code></pre><p>That shows our two generations with the diff between Generation 1 and
Generation 2. We can at any time run <code>guix package --roll-back</code> and get
our previous versions of gimp, emacs, and so on. Each generation is
just a bunch of symlinks to those packages, so what we have looks like
this:</p><p><img src="https://www.gnu.org/software/guix/static/blog/img/guix-pull1.png" alt="Image of the profile generations." /></p><p>Notice that python was not updated, so it’s shared between both
generations. And of course, all the dependencies that didn’t change in
between—e.g., the C library—are shared among all packages.</p><h1><code>guix pull</code> generations</h1><p>Like I wrote above, <code>guix pull</code> brings the latest set of package
definitions from Git <code>master</code>. The Guix package collection usually
contains only the latest version of each package; for example, current
<code>master</code> only has version 26.1 of Emacs and version 2.10.4 of the GIMP
(there are notable exceptions such as GCC or Python.) Thus, <code>guix package -i gimp</code>, from today’s master, can only install gimp 2.10.4.
Often, that’s not a problem: you can keep old profile generations
around, so if you really need that older version of Emacs, you can run
it from your previous generation.</p><p>Still, having <code>guix pull</code> keep track of the changes to Guix and its
package collection is useful. Starting from 0.15.0, <code>guix pull</code> creates
a new generation, just like <code>guix package</code> does. After you’ve run <code>guix pull</code>, you can now list Guix generations as well:</p><pre><code>$ guix pull -l
Generation 10 Jul 14 2018 00:02:03
guix 27f7cbc
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: origin/master
commit: 27f7cbc91d1963118e44b14d04fcc669c9618176
Generation 11 Jul 20 2018 10:44:46
guix 82549f2
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: origin/master
commit: 82549f2328c59525584b92565846217c288d8e85
14 new packages: bsdiff, electron-cash, emacs-adoc-mode,
emacs-markup-faces, emacs-rust-mode, inchi, luakit, monero-gui,
nethack, openbabel, qhull, r-txtplot, stb-image, stb-image-write
52 packages upgraded: angband@4.1.2, aspell-dict-en@2018.04.16-0,
assimp@4.1.0, bitcoin-core@0.16.1, botan@2.7.0, busybox@1.29.1,
…
Generation 12 Jul 23 2018 15:22:52 (current)
guix fef7bab
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: origin/master
commit: fef7baba786a96b7a3100c9c7adf8b45782ced37
20 new packages: ccrypt, demlo, emacs-dired-du,
emacs-helm-org-contacts, emacs-ztree, ffmpegthumbnailer,
go-github-com-aarzilli-golua, go-github-com-kr-text,
go-github-com-mattn-go-colorable, go-github-com-mattn-go-isatty,
go-github-com-mgutz-ansi, go-github-com-michiwend-golang-pretty,
go-github-com-michiwend-gomusicbrainz, go-github-com-stevedonovan-luar,
go-github-com-wtolson-go-taglib, go-github-com-yookoala-realpath,
go-gitlab-com-ambrevar-damerau, go-gitlab-com-ambrevar-golua-unicode,
guile-pfds, u-boot-cubietruck
27 packages upgraded: c-toxcore@0.2.4, calibre@3.28.0,
emacs-evil-collection@20180721-2.5d739f5,
…</code></pre><p>The nice thing here is that <code>guix pull</code> provides high-level information
about the differences between two subsequent generations of Guix.</p><p>In the end, Generation 1 of our profile was presumably built with Guix
Generation 11, while Generation 2 of our profile was built with Guix
Generation 12. We have a clear mapping between Guix generations as
created by <code>guix pull</code> and profile generations as created with <code>guix package</code>:</p><p><img src="https://www.gnu.org/software/guix/static/blog/img/guix-pull3.png" alt="Image of the Guix generations." /></p><p>Each generation created by <code>guix pull</code> corresponds to one commit in the
Guix repo. Thus, if I go to another machine and run:</p><pre><code>$ guix pull --commit=fef7bab</code></pre><p>then I know that I get the exact same Guix instance as my Generation 12
above. From there I can install diffoscope, emacs, etc. and I know I’ll
get the exact same binaries as those I have above, thanks to
<a href="https://reproducible-builds.org/docs/definition/">reproducible builds</a>.</p><p>These are very strong guarantees in terms of reproducibility and
provenance tracking—properties that are
<a href="https://github.com/canonical-websites/snapcraft.io/issues/651">typically</a>
<a href="https://lwn.net/Articles/752982/">missing</a> from “applications bundles”
à la Docker.</p><p>In addition, you can easily run an older Guix. For instance, this is
how you would install the version of gimp that was current as of
Generation 10:</p><pre><code>$ ~/.config/guix/current-10-link/bin/guix package -i gimp</code></pre><p>At this point your profile contains gimp coming from an old Guix along
with packages installed from the latest Guix. Past and present coexist
in the same profile. The historical dimension of the profile no longer
matches exactly the history of Guix itself.</p><h1>Composing Guix revisions</h1><p>Some people have expressed interest in being able to compose packages
coming from different revisions of Guix—say to create a profile
containing old versions of Python and NumPy, but also the latest and
greatest GCC. It may seem far-fetched but it has very real
applications: there are large collections of scientific packages and in
particular bioinformatics packages that don’t move as fast as our
beloved flagship free software packages, and users may require ancient
versions of some of the tools.</p><p>We could keep old versions of many packages but maintainability costs
would grow exponentially. Instead, Guix users can take advantage of the
version control history of Guix itself to mix and match packages coming
from different revisions of Guix. As shown above, it’s already possible
to achieve this by running the <code>guix</code> program off the generation of
interest. It does the job, but can we do better?</p><p>In the process of enhancing <code>guix pull</code> we developed a high-level API
that allows an instance of Guix to “talk” to a different instance of
Guix—<a href="https://debbugs.gnu.org/cgi/bugreport.cgi?bug=32115">an
“inferior”</a>. It’s
what allows <code>guix pull</code> to display the list of packages that were added
or upgraded between two revisions. The next logical step will be to
provide seamless integration of packages coming from an inferior. That
way, users would be able to refer to “past” package graphs right from <a href="https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#index-profile-manifest">a
profile
manifest</a>
or from the command-line. Future work!</p><h1>On coupling</h1><p>The time traveler in you might be wondering: Why are package definitions
coupled with the package manager, doesn’t it make it harder to compose
packages coming from different revisions? Good point!</p><p>Tight coupling certainly complicates this kind of composition: we can’t
just have any revision of Guix load package definitions from any other
revision; this could fail altogether, or it could provide a different
build result. Another potential issue is that <code>guix pull</code>ing an older
revision not only gives you an older set of packages, it also gives you
older tools, bug-for-bug.</p><p>The reason for this coupling is that a package definition <a href="https://www.gnu.org/software/guix/manual/en/html_node/Defining-Packages.html">like this
one</a>
doesn’t exist in a vacuum. Its meaning is defined by the implementation
of <a href="https://www.gnu.org/software/guix/manual/en/html_node/package-Reference.html">package
objects</a>,
by
<a href="https://www.gnu.org/software/guix/manual/en/html_node/Build-Systems.html"><code>gnu-build-system</code></a>,
by a number of <a href="https://arxiv.org/abs/1305.4584">lower-level</a>
<a href="https://hal.inria.fr/hal-01580582/en">abstractions</a> that are all
defined as extensions of the Scheme language in Guix itself, and
ultimately by <a href="https://www.gnu.org/software/guile/">Guile</a>, which
implements the language Guix is written in. Each instance created by
<code>guix pull</code> brings all these components. Because Guix is implemented as
a set of programming language extensions and libraries, that package
definitions depend on all these parts becomes manifest. Instead of
being frozen, the APIs and package definitions evolve together, which
gives us developers a lot of freedom on the changes we can make.</p><p><a href="https://nixos.org/nix/">Nix</a> results from a different design choice.
Nix-the-package-manager implements the Nix language, which acts as a
“frozen” interface. Package definitions in Nixpkgs are written in that
language, and a given version of Nix can <em>possibly</em> interpret both
current and past package definitions without further ado. The Nix
language does evolve though, so at one point an old Nix inevitably
<a href="https://github.com/NixOS/nixpkgs/blob/master/lib/minver.nix">becomes unable to evaluate a new
Nixpkgs</a>,
and <em>vice versa</em>.</p><p>These two approaches make different tradeoffs. Nix’ loose coupling
simplifies the implementation and makes it easy to compose old and new
package definitions, to some extent; Guix’ tight coupling makes such
composition more difficult to implement, but it leaves developers more
freedom and, we hope, may support “time travels” over longer period of
times. Time will tell!</p><h1>It’s like driving a DeLorean</h1><p><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/1/15/TeamTimeCar.com-BTTF_DeLorean_Time_Machine-OtoGodfrey.com-JMortonPhoto.com-05.jpg/800px-TeamTimeCar.com-BTTF_DeLorean_Time_Machine-OtoGodfrey.com-JMortonPhoto.com-05.jpg" alt="Inside the cabin of the DeLorean time machine in “Back to the Future.”" /></p><p><em><a href="https://commons.wikimedia.org/wiki/File:TeamTimeCar.com-BTTF_DeLorean_Time_Machine-OtoGodfrey.com-JMortonPhoto.com-05.jpg">Picture</a>
of a DeLorean cabin by Oto Godfrey and Justin Morton, under <a href="https://creativecommons.org/licenses/by-sa/4.0/deed.en">CC-BY-SA
4.0</a>.</em></p><p>That profile generations are kept around already gave users a time
machine of sorts—you can always roll back to a previous state of your
software environment. With the addition of roll-back support for <code>guix pull</code>, this adds another dimension to the time machine: you can
roll-back to a previous state of Guix itself and from there create
alternative futures or even mix bits from the past with bits from the
present. We hope you’ll enjoy it!</p>https://guix.gnu.org/blog/2018/customize-guixsd-use-stock-ssh-agent-everywhere//Customize GuixSD: Use Stock SSH Agent Everywhere!Chris Marusich2018-05-26T17:00:00Z2018-05-26T17:00:00Z I frequently use SSH. Since I don't like typing my password all the
time, I use an SSH agent. Originally I used the GNOME Keyring as my
SSH agent, but recently I've switched to using the ssh-agent from
OpenSSH. I accomplished this by doing the following two things: Replace the default GNOME Keyring with a custom-built version that
disables the SSH agent feature. Start my desktop session with OpenSSH's ssh-agent so that it's
always available to any applications in my desktop session. Below, I'll show you in…<p>I frequently use SSH. Since I don't like typing my password all the
time, I use an SSH agent. Originally I used the GNOME Keyring as my
SSH agent, but recently I've switched to using the <code>ssh-agent</code> from
OpenSSH. I accomplished this by doing the following two things:</p><ul><li><p>Replace the default GNOME Keyring with a custom-built version that
disables the SSH agent feature.</p></li><li><p>Start my desktop session with OpenSSH's <code>ssh-agent</code> so that it's
always available to any applications in my desktop session.</p></li></ul><p>Below, I'll show you in detail how I did this. In addition to being
useful for anyone who wants to use OpenSSH's <code>ssh-agent</code> in GuixSD, I
hope this example will help to illustrate how GuixSD enables you to
customize your entire system to be just the way you want it!</p><h1>The Problem: GNOME Keyring Can't Handle My SSH Keys</h1><p>On GuixSD, I like to use the <a href="https://www.gnome.org">GNOME desktop
environment</a>. GNOME is just one of <a href="https://www.gnu.org/software/guix/manual/en/html_node/Desktop-Services.html">the
various desktop environments that GuixSD
supports</a>.
By default, the GNOME desktop environment on GuixSD comes with a lot
of goodies, including the <a href="https://wiki.gnome.org/Projects/GnomeKeyring">GNOME
Keyring</a>, which is
GNOME's integrated solution for securely storing secrets, passwords,
keys, and certificates.</p><p>The GNOME Keyring has many useful features. One of those is <a href="https://wiki.gnome.org/Projects/GnomeKeyring/Ssh">its SSH
Agent feature</a>.
This feature allows you to use the GNOME Keyring as an SSH agent.
This means that when you invoke a command like <code>ssh-add</code>, it will add
the private key identities to the GNOME Keyring. Usually this is
quite convenient, since it means that GNOME users basically get an SSH
agent for free!</p><p>Unfortunately, up until <a href="https://www.gnome.org/news/2018/03/gnome-3-28-released/">GNOME 3.28 (the current
release)</a>,
the GNOME Keyring's SSH agent implementation was not as complete as
the stock SSH agent from OpenSSH. As a result, <a href="https://bugzilla.gnome.org/show_bug.cgi?id=775981">earlier versions of
GNOME Keyring did not support many use
cases</a>. This was a
problem for me, since GNOME Keyring couldn't read my modern SSH keys.
To make matters worse, by design the SSH agent for GNOME Keyring and
OpenSSH both use the same environment variables (e.g.,
<code>SSH_AUTH_SOCK</code>). This makes it difficult to use OpenSSH's
<code>ssh-agent</code> everywhere within my GNOME desktop environment.</p><p>Happily, starting with GNOME 3.28, <a href="https://bugzilla.gnome.org/show_bug.cgi?id=775981">GNOME Keyring delegates all SSH
agent functionality to the stock SSH agent from
OpenSSH</a>. They
have removed their custom implementation entirely. This means that
today, I could solve my problem simply by using the most recent
version of GNOME Keyring. I'll probably do just that when the new
release gets included in Guix. However, when I first encountered this
problem, GNOME 3.28 hadn't been released yet, so the only option
available to me was to customize GNOME Keyring or remove it entirely.</p><p>In any case, I'm going to show you how I solved this problem by
modifying the default GNOME Keyring from the Guix package collection.
The same ideas can be used to customize any package, so hopefully it
will be a useful example. And what if you don't use GNOME, but you do
want to use OpenSSH's <code>ssh-agent</code>? In that case, you may still need
to customize your GuixSD system a little bit. Let me show you how!</p><h1>The Solution: <code>~/.xsession</code> and a Custom GNOME Keyring</h1><p>The goal is to make OpenSSH's <code>ssh-agent</code> available everywhere when we
log into our GNOME desktop session. First, we must arrange for
<code>ssh-agent</code> to be running whenever we're logged in.</p><p>There are many ways to accomplish this. For example, I've seen people
implement shell code in their shell's start-up files which basically
manages their own <code>ssh-agent</code> process. However, I prefer to just
start <code>ssh-agent</code> once and not clutter up my shell's start-up files
with unnecessary code. So that's what we're going to do!</p><h1>Launch OpenSSH's <code>ssh-agent</code> in Your <code>~/.xsession</code></h1><p>By default, GuixSD uses the <a href="https://sourceforge.net/projects/slim.berlios">SLiM desktop
manager</a>. When you log
in, SLiM presents you with a menu of so-called "desktop sessions",
which correspond to the desktop environments you've declared in your
<a href="https://www.gnu.org/software/guix/manual/en/html_node/operating_002dsystem-Reference.html">operating system
declaration</a>.
For example, if you've added the
<a href="https://www.gnu.org/software/guix/manual/en/html_node/Desktop-Services.html">gnome-desktop-service</a>
to your operating system declaration, then you'll see an option for
GNOME at the SLiM login screen.</p><p>You can further customize your desktop session with the <code>~/.xsession</code>
file. The contract for this file in GuixSD is the same as it is for
many GNU/Linux distributions: <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/services/xorg.scm?id=263c9941a1e523b360ca9f42d1ed6b11e6e6e285#n392">if it exists, then it will be
executed</a>.
The arguments passed to it will be the command line invocation that
would normally be executed to start the desktop session that you
selected from the SLiM login screen. Your <code>~/.xsession</code> is expected
to do whatever is necessary to customize and then start the specified
desktop environment. For example, when you select GNOME from the SLiM
login screen, your <code>~/.xsession</code> file will basically be executed like
this (for the exact execution mechanism, please refer to the source
code linked above):</p><pre><code class="language-shell">$ ~/.xsession gnome-session</code></pre><p>The upshot of all this is that the <code>~/.xsession</code> is an <em>ideal</em> place
to set up your SSH agent! If you start an SSH agent in your
<code>~/.xsession</code> file, you can have the SSH agent available everywhere,
automatically! Check it out: Put this into your <code>~/.xsession</code> file,
and make the file executable:</p><pre><code class="language-shell">#!/run/current-system/profile/bin/bash
exec ssh-agent "$@"</code></pre><p>When you invoke <code>ssh-agent</code> in this way, it executes the specified
program in an environment where commands like <code>ssh-add</code> just work. It
does this by setting environment variables such as <code>SSH_AUTH_SOCK</code>,
which programs like <code>ssh-add</code> find and use automatically. Because
GuixSD allows you to customize your desktop session like this, you can
use any SSH agent you want in any desktop environments that you want,
automatically!</p><p>Of course, if you're using GNOME Keyring version 3.27 or earlier (like
I was), then this isn't quite enough. In that case, the SSH agent
feature of GNOME Keyring will override the environment variables set
by OpenSSH's <code>ssh-agent</code>, so commands like <code>ssh-add</code> will wind up
communicating with the GNOME Keyring instead of the <code>ssh-agent</code> you
launched in your <code>~/.xsession</code>. This is bad because, as previously
mentioned, GNOME Keyring version 3.27 or earlier doesn't support as
many uses cases as OpenSSH's <code>ssh-agent</code>.</p><p>How can we work around this problem?</p><h1>Customize the GNOME Keyring</h1><p>One heavy-handed solution would be to remove GNOME Keyring entirely.
That would work, but then you would lose out on all the other great
features that it has to offer. Surely we can do better!</p><p>The GNOME Keyring documentation
<a href="https://wiki.gnome.org/Projects/GnomeKeyring/Ssh">explains</a> that one
way to disable the SSH agent feature is to include the
<code>--disable-ssh-agent</code> configure flag when building it. Thankfully,
Guix provides some ways to customize software in <em>exactly</em> this way!</p><p>Conceptually, we "just" have to do the following two things:</p><ul><li><p>Customize the existing <code>gnome-keyring</code> package.</p></li><li><p>Make the <code>gnome-desktop-service</code> use our custom <code>gnome-keyring</code>
package.</p></li></ul><h1>Create a Custom GNOME Keyring Package</h1><p>Let's begin by defining a custom <code>gnome-keyring</code> package, which we'll
call <code>gnome-keyring-sans-ssh-agent</code>. With Guix, we can do this in
less than ten lines of code:</p><pre><code class="language-scheme">(define-public gnome-keyring-sans-ssh-agent
(package
(inherit gnome-keyring)
(name "gnome-keyring-sans-ssh-agent")
(arguments
(substitute-keyword-arguments
(package-arguments gnome-keyring)
((#:configure-flags flags)
`(cons "--disable-ssh-agent" ,flags))))))</code></pre><p>Don't worry if some of that code is unclear at first. I'll clarify it
now!</p><p>In Guix, a <code><package></code> record like the one above is defined by a macro
called <code>define-record-type*</code> (<a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/records.scm?id=263c9941a1e523b360ca9f42d1ed6b11e6e6e285#n178">defined in the file guix/records.scm in
the Guix
source</a>).
It's similar to an <a href="https://www.gnu.org/software/guile/manual/en/html_node/SRFI_002d9-Records.html#SRFI_002d9-Records">SRFI-9
record</a>.
The <code>inherit</code> feature of this macro is very useful: it creates a new
copy of an existing record, overriding specific fields in the new copy
as needed.</p><p>In the above, we define <code>gnome-keyring-sans-ssh-agent</code> to be a copy of
the <code>gnome-keyring</code> package, and we use <code>inherit</code> to change the <code>name</code>
and <code>arguments</code> fields in that new copy. We also use the
<code>substitute-keyword-arguments</code> macro (<a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/utils.scm?id=263c9941a1e523b360ca9f42d1ed6b11e6e6e285#n345">defined in the file
guix/utils.scm in the Guix
source</a>)
to add <code>--disable-ssh-agent</code> to the list of <a href="https://www.gnu.org/software/guix/manual/en/html_node/Build-Systems.html">configure
flags</a>
defined in the <code>gnome-keyring</code> package. The effect of this is to
define a new GNOME Keyring package that is built exactly the same as
the original, but in which the SSH agent is disabled.</p><p>I'll admit this code may seem a little opaque at first, but all code
does when you first learn it. Once you get the hang of things, you
can customize packages any way you can imagine. If you want to learn
more, you should read the docstrings for the <code>define-record-type*</code> and
<code>substitute-keyword-arguments</code> macros in the Guix source code. It's
also very helpful to <code>grep</code> the source code to see examples of how
these macros are used in practice. For example:</p><pre><code class="language-shell">$ # Search the currently installed Guix for the current user.
$ grep -r substitute-keyword-arguments ~/.config/guix/latest
$ # Search the Guix Git repository, assuming you've checked it out here.
$ grep -r substitute-keyword-arguments ~/guix</code></pre><h1>Use the Custom GNOME Keyring Package</h1><p>OK, we've created our own custom GNOME Keyring package. Great! Now,
how do we use it?</p><p>In GuixSD, the GNOME desktop environment is treated as a <a href="https://www.gnu.org/software/guix/manual/en/html_node/Services.html">system
service</a>. To
make GNOME use our custom GNOME Keyring package, we must somehow
customize the <code>gnome-desktop-service</code> (<a href="https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/services/desktop.scm?id=263c9941a1e523b360ca9f42d1ed6b11e6e6e285#n795">defined in the file
gnu/services/desktop.scm</a>)
to use our custom package. How do we customize a service? Generally,
the answer depends on the service. Thankfully, many of GuixSD's
services, including the <code>gnome-desktop-service</code>, follow a similar
pattern. In this case, we "just" need to pass a custom
<code><gnome-desktop-configuration></code> record to the <code>gnome-desktop-service</code>
procedure in our operating system declaration, like this:</p><pre><code class="language-scheme">(operating-system
...
(services (cons*
(gnome-desktop-service
#:config my-gnome-desktop-configuration)
%desktop-services)))</code></pre><p>Here, the <code>cons*</code> procedure just adds the GNOME desktop service to the
<code>%desktop-services</code> list, returning the new list. For details, please
refer to <a href="https://www.gnu.org/software/guile/manual/en/html_node/List-Constructors.html#index-cons_002a">the Guile
manual</a>.</p><p>Now the question is: what should <code>my-gnome-desktop-configuration</code> be?
Well, if we examine <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/services/desktop.scm?id=263c9941a1e523b360ca9f42d1ed6b11e6e6e285#n799">the definition of this record type in the Guix
source</a>,
we see the following:</p><pre><code class="language-scheme">(define-record-type* <gnome-desktop-configuration> gnome-desktop-configuration
make-gnome-desktop-configuration
gnome-desktop-configuration
(gnome-package gnome-package (default gnome)))</code></pre><p>The <code>gnome</code> package referenced here is a "meta" package: it exists
only to aggregate many GNOME packages together, including
<code>gnome-keyring</code>. To see <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/gnome.scm?id=263c9941a1e523b360ca9f42d1ed6b11e6e6e285#n5977">its
definition</a>,
we can simply invoke <code>guix edit gnome</code>, which <a href="https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-edit.html#Invoking-guix-edit">opens the file where
the package is
defined</a>:</p><pre><code class="language-scheme">(define-public gnome
(package
(name "gnome")
(version (package-version gnome-shell))
(source #f)
(build-system trivial-build-system)
(arguments '(#:builder (mkdir %output)))
(propagated-inputs
;; TODO: Add more packages according to:
;; <https://packages.debian.org/jessie/gnome-core>.
`(("adwaita-icon-theme" ,adwaita-icon-theme)
("baobab" ,baobab)
("font-cantarell" ,font-cantarell)
[... many packages omitted for brevity ...]
("gnome-keyring" ,gnome-keyring)
[... many packages omitted for brevity ...]
(synopsis "The GNU desktop environment")
(home-page "https://www.gnome.org/")
(description
"GNOME is the graphical desktop for GNU. It includes a wide variety of
applications for browsing the web, editing text and images, creating
documents and diagrams, playing media, scanning, and much more.")
(license license:gpl2+)))</code></pre><p>Apart from being a little long, this is <a href="https://www.gnu.org/software/guix/manual/en/html_node/Defining-Packages.html#Defining-Packages">just a normal package
definition</a>.
We can see that <code>gnome-keyring</code> is included in the list of
<code>propagated-inputs</code>. So, we need to create a replacement for the
<code>gnome</code> package that uses our <code>gnome-keyring-sans-ssh-agent</code> instead
of <code>gnome-keyring</code>. The following package definition accomplishes
that:</p><pre><code class="language-scheme">(define-public gnome-sans-ssh-agent
(package
(inherit gnome)
(name "gnome-sans-ssh-agent")
(propagated-inputs
(map (match-lambda
((name package)
(if (equal? name "gnome-keyring")
(list name gnome-keyring-sans-ssh-agent)
(list name package))))
(package-propagated-inputs gnome)))))</code></pre><p>As before, we use <code>inherit</code> to create a new copy of the <code>gnome</code>
package that overrides the original <code>name</code> and <code>propagated-inputs</code>
fields. Since Guix packages are just defined using good old scheme,
we can use existing language features like
<a href="https://www.gnu.org/software/guile/manual/en/html_node/List-Mapping.html#index-map"><code>map</code></a>
and
<a href="https://www.gnu.org/software/guile/manual/en/html_node/Pattern-Matching.html#Pattern-Matching"><code>match-lambda</code></a>
to manipulate the list of propagated inputs. The effect of the above
is to create a new package that is the same as the <code>gnome</code> package but
uses <code>gnome-keyring-sans-ssh-agent</code> instead of <code>gnome-keyring</code>.</p><p>Now that we have <code>gnome-sans-ssh-agent</code>, we can create a custom
<code><gnome-desktop-configuration></code> record and pass it to the
<code>gnome-desktop-service</code> procedure as follows:</p><pre><code class="language-scheme">(operating-system
...
(services (cons*
(gnome-desktop-service
#:config (gnome-desktop-configuration
(gnome-package gnome-sans-ssh-agent)))
%desktop-services)))</code></pre><h1>Wrapping It All Up</h1><p>Finally, you need to run the following commands as <code>root</code> to create
and boot into the new <a href="https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-system.html">system
generation</a>
(replace <code>MY-CONFIG</code> with the path to the customized operating system
configuration file):</p><pre><code class="language-shell"># guix system reconfigure MY-CONFIG
# reboot</code></pre><p>After you log into GNOME, any time you need to use SSH, the stock SSH
agent from OpenSSH that you started in your <code>~/.xsession</code> file will be
used instead of the GNOME Keyring's SSH agent. It just works! Note
that it still works even if you select a non-GNOME desktop session
(like XFCE) at the SLiM login screen, since the <code>~/.xsession</code> is not
tied to any particular desktop session,</p><p>In the unfortunate event that something went wrong and things just
aren't working when you reboot, don't worry: with GuixSD, you can
safely roll back to the previous system generation via <a href="https://www.gnu.org/software/guix/manual/en/html_node/Using-the-Configuration-System.html#index-roll_002dback_002c-of-the-operating-system">the usual
mechanisms</a>.
For example, you can run this from the command line to roll back:</p><pre><code class="language-shell"># guix system roll-back
# reboot</code></pre><p>This is one of the great benefits that comes from the fact that <a href="https://www.gnu.org/software/guix/manual/en/html_node/Introduction.html#Introduction">Guix
follows the functional software deployment
model</a>.
However, note that because the <code>~/.xsession</code> file (like many files in
your home directory) is not managed by Guix, you must manually undo
the changes that you made to it in order to roll back fully.</p><h1>Conclusion</h1><p>I hope this helps give you some ideas for how you can customize your
own GuixSD system to make it exactly what you want it to be. Not only
can you customize your desktop session via your <code>~/.xsession</code> file,
but Guix also provides tools for you to modify any of the default
packages or services to suit your specific needs.</p><p>Happy hacking!</p><h1>Notices</h1><p><a href="http://creativecommons.org/publicdomain/zero/1.0/"><img src="https://licensebuttons.net/p/zero/1.0/88x31.png" alt="CC0" title="CC0 1.0
Universal" /></a></p><p>To the extent possible under law, Chris Marusich has waived all
copyright and related or neighboring rights to this article,
"Customize GuixSD: Use Stock SSH Agent Everywhere!". This work is
published from: United States.</p><p>The views expressed in this article are those of Chris Marusich and do
not necessarily reflect the views of his past, present, or future
employers.</p><h4>About GNU Guix</h4><p><a href="https://www.gnu.org/software/guix">GNU Guix</a> is a transactional package
manager for the GNU system. The Guix System Distribution or GuixSD is
an advanced distribution of the GNU system that relies on GNU Guix and
<a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects the user's
freedom</a>.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. Guix uses low-level
mechanisms from the Nix package manager, except that packages are
defined as native <a href="https://www.gnu.org/software/guile">Guile</a> modules,
using extensions to the <a href="http://schemers.org">Scheme</a> language. GuixSD
offers a declarative approach to operating system configuration
management, and is highly customizable and hackable.</p><p>GuixSD can be used on an i686, x86_64 and armv7 machines. It is also
possible to use Guix on top of an already installed GNU/Linux system,
including on mips64el and aarch64.</p>https://guix.gnu.org/blog/2014/guix-at-the-2014-gnu-hackers-meeting//Guix at the 2014 GNU Hackers MeetingLudovic Courtès2014-10-11T00:00:00+02002014-10-11T00:00:00+0200 The Guix talk of this summer's GNU Hackers Meeting is now available (get the slides ). DOWNLOAD VIDEO (WebM, 60 minutes) It gives an introduction to Guix from a user's viewpoint, and covers topics such as features for GNU maintainers, programming interfaces, declarative operating system configuration, status of the GNU/Hurd port, and the new Emacs and Web interfaces---with a bunch of demos. Do not miss other fine talks from the GHM . Many thanks to everyone who took care of the video recordings. …<p>The Guix talk of this summer's GNU Hackers Meeting is now available (get the <a href="https://www.gnu.org/software/guix/guix-ghm-20140815.pdf">slides</a>).</p><video src="https://audio-video.gnu.org/video/ghm2014/2014-08--courtes--were-building-the-gnu-system--ghm.webm" controls="controls"><div class="action-box centered-text"><a class="button-big" href="https://audio-video.gnu.org/video/ghm2014/2014-08--courtes--were-building-the-gnu-system--ghm.webm">DOWNLOAD VIDEO</a><p>(WebM, 60 minutes)</p></div></video><p>It gives an introduction to Guix from a user's viewpoint, and covers topics such as features for GNU maintainers, programming interfaces, declarative operating system configuration, status of the GNU/Hurd port, and the new Emacs and Web interfaces---with a bunch of demos.</p><p>Do not miss <a href="http://audio-video.gnu.org/video/ghm2014/">other fine talks from the GHM</a>. Many thanks to everyone who took care of the video recordings.</p>https://guix.gnu.org/blog/2013/guix-at-the-european-lisp-symposium//Guix at the European Lisp SymposiumLudovic Courtès2013-05-21T00:00:00+02002013-05-21T00:00:00+0200 A paper presenting the design of Guix's Scheme API and packaging language has been accepted for the 2013 European Lisp Symposium (ELS) . ELS will take place in Madrid on June 3-4. …<div><p>A <a href="http://arxiv.org/abs/1305.4584">paper</a> presenting the design of Guix's Scheme API and packaging language has been accepted for the <a href="http://www-sop.inria.fr/members/Manuel.Serrano/conferences/els13.html">2013 European Lisp Symposium (ELS)</a>. ELS will take place in Madrid on June 3-4.<br /></p></div>