https://guix.gnu.org/feeds/blog/functional-programming.atomGNU Guix — Blog — Functional programmingfeed author nameGNU Guixhttps://guix.gnu.org/themes/initial/img/icon.png2024-03-20T10:57:56Zhttps://guix.gnu.org/blog/2020/grafts-continued//Grafts, continuedLudovic Courtès2020-05-06T15:00:00Z2020-05-06T15:00:00Z Guix includes a mechanism called grafts that allows us to provide
users with security
updates
in a timely fashion, even for core packages deep down in the dependency
graph. Most users value the benefits of grafts, but newcomers were also
unavoidably struck by what turned out to be the undesirable side effect
of our graft implementation on user experience. This had been a
well-known problem for a while, but 1.1.0 finally addressed these
issues . This article recaps how grafts are implemented, what problems that
caused, and how we solved it. It’s a deep dive into core Guix,…<p>Guix includes a mechanism called <em>grafts</em> that allows us to provide
users with <a href="https://guix.gnu.org/manual/en/html_node/Security-Updates.html">security
updates</a>
in a timely fashion, even for core packages deep down in the dependency
graph. Most users value the benefits of grafts, but newcomers were also
unavoidably struck by what turned out to be the undesirable side effect
of our graft implementation on user experience. This had been a
well-known problem for a while, but <a href="https://guix.gnu.org/blog/2020/gnu-guix-1.1.0-released/">1.1.0 finally addressed these
issues</a>.</p><p>This article recaps how grafts are implemented, what problems that
caused, and how we solved it. It’s a deep dive into core Guix, and I
hope it’ll be insightful to all and intriguing to the functional
programming geeks among us!</p><h1>What’s this “graft” thing anyway?</h1><p>Grafts were introduced in the early days of Guix to <a href="https://guix.gnu.org/blog/2016/timely-delivery-of-security-updates/">address probably
the main practical shortcomings of functional software
deployment</a>.
In a nutshell, functional deployment as implemented by Nix and Guix
means that, when a package changes, everything that depends on it must
be rebuilt (or re-downloaded). To deploy a security fix in the C
library or in Bash, you would thus need to rebuild everything. Even
with a huge build farm, that can significantly delay the deployment of
fixes; users would find themselves either rebuilding things locally or,
at best, re-downloading binaries of everything.</p><p>To address this, Guix developers can instead specify a <em>replacement</em> in
a <a href="https://guix.gnu.org/manual/en/html_node/Defining-Packages.html">package
definition</a>.
If we have a bug-fix for, say, libc, developers would (1) define a
package for the fixed libc, and (2) add a <code>replacement</code> field in the
original libc package pointing to that fixed package. The effect is
that <em>only the bug-fix libc needs to be built</em>. When building a
package, the bug-fix libc is automatically <em>grafted onto that package</em>,
such that the resulting package refers to the bug-fix libc. <a href="https://guix.gnu.org/manual/en/html_node/Security-Updates.html">See the
manual</a>
for more.</p><p>When “lowering” a high-level <a href="https://guix.gnu.org/manual/en/html_node/Defining-Packages.html">package
definition</a>
to a low-level
<a href="https://guix.gnu.org/manual/en/html_node/Derivations.html"><em>derivation</em></a>,
Guix traverses the package dependency graph and identifies a set of
potentially applicable grafts. Why “potentially applicable”? Consider
this scenario: assume <code>perl</code> has a <code>replacement</code>; <code>coreutils</code> has a
dependency on <code>perl</code>, but it’s a build-time dependency: <code>coreutils</code> does
not depend on <code>perl</code> at run time. Thus, <code>coreutils</code> can be used as is,
there is no need to graft it.</p><p>But how do we know whether a dependency is a built-time-only dependency?
The <a href="https://guix.gnu.org/manual/en/html_node/package-Reference.html#index-package"><code>native-inputs</code>
field</a>
of a package usually lists build-time dependencies, but it’s more of a
hint. Ultimately, the set of run-time dependencies, which we call the
<em>references</em>, is the subset of the build-time dependencies that the
garbage collector (GC) in the build daemon finds <em>in the build
result</em>—Section 5.5.1 of <a href="http://nixos.org/~eelco/pubs/phd-thesis.pdf">Eelco Dolstra’s PhD
thesis</a> describes how the
GC
scans for references. In our example, we first have to actually build
<code>coreutils</code> before we can tell whether it depends on <code>perl</code> at
run time.</p><p>Guix arranges to graft only when necessary. In this example, <code>guix build coreutils</code> would return the same as <code>guix build coreutils --no-grafts</code>. Conversely, since <code>inkscape</code> has a run-time dependency on
<code>perl</code>, <code>guix build inkscape</code> returns a derivation that grafts the
<code>perl</code> replacement onto the original <code>inkscape</code> build result, the one
returned by <code>guix build inkscape --no-grafts</code>. The (simplified)
dependency graph of the derivation for the grafted <code>inkscape</code> looks like
this:</p><p><img src="https://guix.gnu.org/static/blog/img/inkscape-graft.svg" alt="Dependency graph of the graft derivation of Inkscape." /></p><p>Grafts are a form of what <a href="https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf"><em>Build Systems à la
Carte</em></a>
by Mokhov <em>et al.</em> (a good read!) refers to as <em>dynamic dependencies</em>:
grafting depends on intermediate <em>build results</em>.</p><p>Still here? With the background in place, let’s look at the problems
that arose.</p><h1>Grafts, the user interface, and performance</h1><p>Conceptually, to decide whether to graft a package, we examine the
references of the build result of the ungrafted package. However, we
usually want <code>guix install</code> & co. to first display an overall build
plan, especially when invoked with <code>--dry-run</code>:</p><pre><code>$ guix install inkscape
The following package will be installed:
inkscape 1.0
71.3 MB will be downloaded:
/gnu/store/xq64iaxx2gmlcgnipj31wjxlf1yd2g2p-gsl-2.6
/gnu/store/zrmhnj3pwchn2msphgnwzwd3q89m46rn-aspell-0.60.8
/gnu/store/5g31zf21lk8nrfd2b8qrm19nwdm9gir9-potrace-1.16
/gnu/store/qpr7387bsjs7rpl6flrwdvn2zbzh5bch-ghostscript-with-cups-9.52
/gnu/store/7y3lvk3xf4im8n44337mc6y0ccysvfia-font-dejavu-2.37
/gnu/store/95n3zvzxzq2bxvdvz292cg04757ij30x-cups-minimal-2.3.1
…</code></pre><p>To accommodate that, the pre-1.1.0 implementation of grafts did the
following: when
<a href="https://guix.gnu.org/manual/en/html_node/Substitutes.html">substitutes</a>
were enabled, it would get the list of references of ungrafted packages
from substitutes; only when substitutes for an ungrafted package are
missing would it first try to build that package. Thus, when
substitutes are available, <code>guix install</code> and similar commands would be
able to display the build plan upfront. However, when a packages had no
substitutes, you’d see Guix start building it without saying a word
about the build plan, which was <a href="https://issues.guix.gnu.org/issue/28310">arguably
confusing</a>.</p><p>But it’s worse than that. Grafting is per-package, so every time you
would lower a package to a derivation, you would need to answer the
question “does <em>this</em> specific package have substitutes, and if so,
should it be grafted?” The end result was <a href="https://issues.guix.gnu.org/issue/22990">poor resource usage and
terrible user interface
feedback</a>. For every package
that is a graft candidate, the user would see that infamous line:</p><pre><code>updating substitutes from 'https://ci.guix.gnu.org'...</code></pre><p>The problem was particularly acute when building whole systems with
<code>guix system</code> because there would typically be a large number of such
packages. Furthermore, each of these lines would correspond to
(roughly) a single HTTP GET request on a fresh TLS connection. That can
be slow… and annoying. Perhaps to some users this <code>updating substitutes</code> stuttering was the proof of the developers’ incompetence
and perhaps, truth be told, to some of us developers it was a small
price to pay for the sophistication of grafts.</p><p>For users who disable substitutes and build everything locally, the
situation wasn’t much better: all the packages candidate for grafting
would be built one by one, thereby missing parallelization opportunities
as specified by
<a href="https://guix.gnu.org/manual/en/html_node/Invoking-guix_002ddaemon.html"><code>--max-jobs</code></a>.</p><h1>Gathering dynamic dependencies</h1><p>To address this, all these individual dynamic dependencies need to be
gathered somehow instead of being treated one by one. Conceptually, we
would like to, roughly, do a first pass lowering packages to derivations
as if grafting was disabled, build all these derivations, and then do a
second pass to determine which packages in the graph need to be grafted and
to compute the relevant grafting derivation. That would address the
performance issue: we’d now have as much parallelism as possible, so we
wouldn’t query substitutes or build packages one by one. If we reify
that second pass to the user interface code, it also addresses the user
interface issue by allowing it to display, possibly, two build plans:
the “ungrafted” one followed by the grafted one.</p><p>The problem is that our API is inherently serial: the
<code>package-derivation</code> function takes <em>one</em> package, lowers it, and
returns its derivation:</p><pre><code class="language-scheme">(use-modules (guix)
(gnu packages base)
(gnu packages inkscape))
(define s (open-connection))
(package-derivation s coreutils)
⇒ #<derivation /gnu/store/rpfdbax1py483m9ydhvp73s7dgmn6xh4-coreutils-8.31.drv => /gnu/store/jkj7wxybgcpdamkl6fz7wwbb1ak5wxvk-coreutils-8.31-debug /gnu/store/zibwkb5xavnv6z3gzknfqjsxb9b0izh0-coreutils-8.31 7f6c92e3a000>
(package-derivation s coreutils #:graft? #f)
⇒ #<derivation /gnu/store/rpfdbax1py483m9ydhvp73s7dgmn6xh4-coreutils-8.31.drv => /gnu/store/jkj7wxybgcpdamkl6fz7wwbb1ak5wxvk-coreutils-8.31-debug /gnu/store/zibwkb5xavnv6z3gzknfqjsxb9b0izh0-coreutils-8.31 7f6c92e3a000>
(package-derivation s inkscape)
⇒ #<derivation /gnu/store/jzm2zsq8m0rj8wdsmikc0p2ik0cprrcf-inkscape-0.92.4.drv => /gnu/store/clj8rjnsip8a35hyd9nf4l65w7ahn0gs-inkscape-0.92.4 7f6c9c15b730>
(package-derivation s inkscape #:graft? #f)
⇒ #<derivation /gnu/store/psd31x1fq0v2g594z217mh56xzg21dym-inkscape-0.92.4.drv => /gnu/store/zz28ckjwfxwkx3gsm8sc452kmvfiky6y-inkscape-0.92.4 7f6c90ad4f50></code></pre><p>Lowering includes dealing with grafts, and
that’s why we ended up with one-by-one inefficiencies. An option would
be to make all the API “plural”: have <code>package-derivation</code> and its
friends accept a <em>list</em> of packages instead of a single one. That would
be a huge amount of work and the end result would be unpleasant to use:
it’s easier to reason one-by-one.</p><p>The solution implemented in 1.1.0 instead starts from this observation:
the call graph of <code>package-derivation</code> mirrors the package graph. Thus,
we could gather dynamic dependencies using <a href="https://guix.gnu.org/manual/en/html_node/The-Store-Monad.html">monad
trickery</a>
or using “control effects”. We went for the latter, which didn’t have
the “contamination” problem of monads and led to simpler code.</p><p>The starting point is that, by definition, code with dynamic
dependencies necessarily calls
<a href="https://guix.gnu.org/manual/en/html_node/The-Store.html#index-build_002dderivations"><code>build-derivations</code></a>.
Taking advantage of <a href="https://www.gnu.org/software/guile/manual/html_node/Prompts.html">delimited continuations in
Guile</a>,
<code>build-derivations</code> is instrumented to <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=041b340da409078951267b6a8c43b27716e6b7ec">abort to a “build handler”
prompt</a>
when it’s called. The build handler receives the list of derivations to
build along with a continuation to invoke to resume the aborted
computation and start building things. User interface code can install
a build handler that displays what’s going to be built:</p><pre><code class="language-scheme">(with-build-handler (lambda (continue store things mode)
(show-what-to-build store things)
(continue #t))
…)</code></pre><p>To implement dry runs, simply omit the call to <code>continue</code> and nothing
will be built. (This is a slightly simplified artist view, see
<a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=07ce23e011d18460e7ff5553d4ff640f7073075b"><code>build-notifier</code></a>
for the real thing.)</p><p>Now, we need to take advantage of this mechanism to gather the
individual <code>build-derivations</code> calls so we can later emit a single
<code>build-derivations</code> call for all the gathered derivations. The goal is
to effectively gather all the calls for ungrafted packages, build them
all at once, and then resume graft computation.</p><p>To achieve that, we write a build handler that, when invoked, returns an
<code><unresolved></code> object that captures what to build and the continuation.
In addition, we provide a primitive to <em>introduce parallelism</em> such
that, if a dynamic dependency is encountered, we keep going and attempt
to compute as much as possible without resolving that dynamic
dependency. These are <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=c40bf5816cb3ffb59920a61f71bd34b53cac3637"><code>build-accumulator</code> and
<code>map/accumulate-builds</code></a>.
<code>map/accumulate-builds</code> is like <code>map</code>, except that it accumulates and
gathers <code>build-derivations</code> request.</p><p>By using <code>map/accumulate-builds</code> instead of <code>map</code> in a few
<a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=584dfdac3795541ff020aca3f488ceaf2ddd7fc3">key</a>
<a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=25af35fa32bf6c991510406a330d4a42bd5beba8">places</a>,
we obtain a good approximation of what we wanted, as illustrated in this
run:</p><pre><code>$ guix install zile-on-guile vim-full
The following packages will be installed:
zile-on-guile 2.4.14-0.fd09781
vim-full 8.2.0411
9.4 MB will be downloaded:
/gnu/store/vf7w4yiax38ra7x8aqqvbnc38c0ldgpm-zile-on-guile-2.4.14-0.fd09781
/gnu/store/dnj9wljcck9vdwgp7dwxk00qnnk1g3c5-vim-full-8.2.0411
downloading from https://ci.guix.gnu.org/nar/lzip/dnj9wljcck9vdwgp7dwxk00qnnk1g3c5-vim-full-8.2.0411...
vim-full-8.2.0411 8.9MiB 7.6MiB/s 00:01 [##################] 100.0%
downloading from https://ci.guix.gnu.org/nar/lzip/vf7w4yiax38ra7x8aqqvbnc38c0ldgpm-zile-on-guile-2.4.14-0.fd09781...
zile-on-guile-2.4.14-0.fd09781 140KiB 1.8MiB/s 00:00 [##################] 100.0%
The following derivation will be built:
/gnu/store/d9xms78917w67xq71pqsx5x9s6dmq6d7-profile.drv
The following graft will be made:
/gnu/store/4n6dmg6iwjg0adpcvqygr9wgsnclswss-vim-full-8.2.0411.drv
applying 8 grafts for /gnu/store/4n6dmg6iwjg0adpcvqygr9wgsnclswss-vim-full-8.2.0411.drv...
building /gnu/store/d9xms78917w67xq71pqsx5x9s6dmq6d7-profile.drv...</code></pre><p>What we see above is first a build plan that downloads binaries for the
two ungrafted packages, followed by a build plan for one grafting
derivations: we have successfully preserved parallelism.</p><p>The solution resembles the <code>suspending</code> scheduler discussed in the <em>à
la Carte</em> paper, though decomposition is not as principled as what the
paper describes. It remains an approximation and not the
optimal way to deal with dynamic dependencies. There are still
situations <a href="https://issues.guix.gnu.org/issue/40612">where that shows</a>,
but overall, it’s a significant improvement. Unlike <a href="https://issues.guix.gnu.org/issue/22990#7">other solutions
prototyped before</a>, this one
has the advantage of being orthogonal and simple: less than <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=c40bf5816cb3ffb59920a61f71bd34b53cac3637">100 new
lines of
code</a>,
and even <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=4b75a7060058bc2e959dcb4145067f6bba3e34e5">about 30 lines
removed</a>
from the graft implementation. That alone contributes a lot to the
author’s satisfaction. :-)</p><h1>Interlude: a scatter/gather pattern?</h1><p>In the end, we’re just gathering all the <code>build-derivations</code> calls,
turning them into a single call, and finally calling all the original
site continuations with the result. The same kind of issue shows up
when dealing with sequences of remote procedure calls (RPCs) and HTTP
requests, and it seems there’s a more general pattern lurking here.
Consider code like this:</p><pre><code class="language-scheme">(map (lambda (thing)
(http-get (thing->url thing)))
lst)</code></pre><p>Wouldn’t it be nice if we could somehow capture all the <code>http-get</code>
calls, turn them into a series of <a href="https://en.wikipedia.org/wiki/HTTP_pipelining">pipelined GET
requests</a>, and resume the
continuations with their result?</p><p>I haven’t found a standard functional pattern to address this and would
welcome ideas!</p><h1>Dynamic dependencies of all shapes</h1><p>We have seen how Guix deals with dynamic dependencies. Nix supports a
similar but limited form of dynamic dependencies through
the <code>import</code> primitive of the
Nix language, <a href="https://github.com/NixOS/nix/blob/master/src/libexpr/primops.cc#L74">which can take the result of a derivation
build</a>;
it does not attempt to gather the resulting <code>buildPaths</code> calls.</p><p>If we take a step back, we can see that Nix and Guix actually support
other forms of dynamic dependencies. For example, it’s possible to
write derivations whose result is a function of the reference graph of
another derivation’s build result. This is achieved in Guix by passing
the <a href="https://guix.gnu.org/manual/en/html_node/G_002dExpressions.html#index-gexp_002d_003ederivation"><code>#:references-graphs</code> argument of
<code>gexp->derivation</code></a>,
which leads the build daemon to <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/nix/libstore/build.cc?id=298fb2907e3f432cea7dee9f58e89ab8d9dbd56f#n1763">include the reference graph in the
build
environment</a>.</p><p>Another form of dynamic dependency is <em>derivation-building derivations</em>
or <em>recursive derivations</em>, which were <a href="https://github.com/NixOS/nix/pull/3205">recently implemented in
Nix</a>. It supports another form
of dynamic dependency where the build process of a derivation can itself
create and build derivations (these are <a href="https://en.wikipedia.org/wiki/Parallel_task_scheduling_problem"><em>moldable
tasks</em></a>
in scheduling parlance). It’s a great feature because in a nutshell, it
allows Nix to be used not only to compose packages, but also at a finer
grain as part of a package build process.</p><p>Guix supports yet another form of dynamic dependencies. The newfangled
<a href="https://guix.gnu.org/manual/en/html_node/Invoking-guix-deploy.html"><code>guix deploy</code>
tool</a>
works by <a href="https://guix.gnu.org/blog/2019/managing-servers-with-gnu-guix-a-tutorial/">evaluating g-expressions (gexps)
remotely</a>.
For example, before actually deploying an operating system, it first
runs code on the remote node to perform sanity checks: checking whether
the declared file system UUIDs or labels are valid, checking whether
additional kernel modules should be added to the initial RAM disk, and
so forth. To do that,
<a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/remote.scm#n109"><code>remote-eval</code></a>
first builds a derivation that produces a Scheme program, deploys it
along with all its dependencies on that target machine, runs it, and
retrieves the result. This form of dynamic dependency also benefits
from the gathering machinery discussed above.</p><h1>Conclusion</h1><p>This is a long article on what may look like a fine point of Guix design
and implementation, but there’s much to say about it! Grafts are key to
the use of functional deployment in production because they enable quick
security updates, and it’s a lot better if they don’t harm the user
experience.</p><p>The pre-1.1.0 implementation of grafts had a negative impact on the user
interface and on performance, which was due to the sequential handling
of grafts, one package at a time. In 1.1.0 we addressed it by using
delimited continuations to gather dynamic dependencies such as grafts,
perform builds in bulk, and resume each derivation computation.</p><p>As it turned out, the implementation of dynamic dependencies raises lots
of interesting design and implementation issues, and it’s probably not
the end of the road!</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package
manager and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects
user
freedom</a>.
Guix can be used on top of any system running the kernel Linux, or it
can be used as a standalone operating system distribution for i686,
x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2017/back-from-gpce//Back from GPCELudovic Courtès2017-11-01T12:00:00Z2017-11-01T12:00:00Z Last week, I was at GPCE
2017 ,
an academic conference focused on generative programming techniques. I
presented Code Staging in
GNU Guix , a paper that discusses
the motivation for and genesis of
G-expressions
as well as recent improvements. The slides are available
here . …<p>Last week, I was at <a href="https://conf.researchr.org/track/gpce-2017/gpce-2017-GPCE-2017">GPCE
2017</a>,
an academic conference focused on generative programming techniques. I
presented <a href="https://hal.inria.fr/hal-01580582/en"><em>Code Staging in
GNU Guix</em></a>, a paper that discusses
the motivation for and genesis of
<a href="https://www.gnu.org/software/guix/manual/en/html_node/G_002dExpressions.html">G-expressions</a>
as well as recent improvements. The slides are <a href="https://www.gnu.org/software/guix/guix-gpce-20171023.pdf">available
here</a>.</p>https://guix.gnu.org/blog/2016/back-from-cufp-2016//Back from CUFP 2016sirgazil2016-09-24T00:00:00+02002016-09-24T00:00:00+0200 The Commercial Users of Functional Programming 2016 Workshop concluded. There is no video recording this time, but you can download the slides from Ludovic Courtès' talk, "Guix: Scheme as a uniform OS admin and deployment interface". …<p>The <a href="http://cufp.org/2016/">Commercial Users of Functional Programming 2016 Workshop</a> concluded. There is no video recording this time, but you can download the <a href="https://www.gnu.org/software/guix/guix-cufp-20160924.pdf">slides</a> from Ludovic Courtès' talk, "Guix: Scheme as a uniform OS admin and deployment interface".</p>https://guix.gnu.org/blog/2016/back-from-the-scheme-workshop-2016//Back from the Scheme Workshop 2016sirgazil2016-09-18T00:00:00+02002016-09-18T00:00:00+0200 The 17th Annual Scheme and Functional Programming Workshop 2016 concluded. There is no video recording this time, but you can download the slides from Ludovic Courtès' invited talk, "GNU Guix: The Functional GNU/Linux Distro That’s a Scheme Library". …<p>The <a href="http://dconf.org/2016/index.html">17th Annual Scheme and Functional Programming Workshop 2016</a> concluded. There is no video recording this time, but you can download the <a href="https://www.gnu.org/software/guix/guix-scheme-workshop-20160918.pdf">slides</a> from Ludovic Courtès' invited talk, "GNU Guix: The Functional GNU/Linux Distro That’s a Scheme Library".</p>https://guix.gnu.org/blog/2016/timely-delivery-of-security-updates//Timely delivery of security updatesLudovic Courtès2016-03-02T00:00:00+01002016-03-02T00:00:00+0100 Yesterday, a new version of OpenSSL was released , addressing several serious vulnerabilities, some of which are nicknamed "DROWN" . Like all free software distributions, we were waiting to deploy the fixes as soon as possible. This time though, we are happy to report that we were able to deploy it to users faster than before: an hour or so after disclosure. This was made possible by fixing our fast update deployment mechanism, which is based on grafts . Updates in a functional package management framework GNU Guix…<div><p>Yesterday, a <a href="http://openssl.org/news/secadv/20160301.txt">new version of OpenSSL was released</a>, addressing several serious vulnerabilities, some of which are <a href="https://drownattack.com/">nicknamed "DROWN"</a>. Like all free software distributions, we were waiting to deploy the fixes as soon as possible. This time though, we are happy to report that we were able to <a href="http://git.savannah.gnu.org/cgit/guix.git/commit/?id=caeadfddb01d2cda19d2f761ba9906ef8f162173">deploy it</a> to users faster than before: an hour or so after disclosure. This was made possible by fixing our fast update deployment mechanism, which is based on <em>grafts</em>.<br /></p><h4>Updates in a functional package management framework</h4><p>GNU Guix implements the <a href="https://www.gnu.org/software/guix/manual/en/html_node/Introduction.html">functional package management discipline</a>. What this means is that the the package graph in Guix is an immutable, <a href="https://en.wikipedia.org/wiki/Persistent_data_structure">persistent data structure</a>—similar to a singly-linked list in a functional programming language, or to the <a href="http://eagain.net/articles/git-for-computer-scientists/">object graph in the Git version control system</a>.<br /></p><p>A common difficulty with persistent data structures is the algorithmic complexity of updates—the computational cost of updating an arbitrary element of the data structure. For instance, to update the nth element of a singly-linked list, you first need to traverse and copy the n ? 1 elements at the head of the list, then insert the new element and make it point to the tail of the list.<br /></p><p>With the functional package management paradigm, the cost of updating a package is simple to understand: you need to rebuild the package itself, <em>and all the packages that depend on it</em>. This is nice in many ways: all packages <em>must</em> build from source, there is no way we can be using binaries that cannot be <a href="/news/reproducible-builds-a-means-to-an-end.html">rebuilt from their Corresponding Source</a>, breakage due to incompatible application binary interfaces (ABIs) is foreign to our users, we have a precise trail of the tools that produced binaries—that is, builds are <a href="https://en.wikipedia.org/wiki/Referential_transparency">“referentially transparent”</a>, and as a bonus, we get <a href="https://www.gnu.org/software/guix/manual/en/html_node/Features.html">features</a> such as transactional upgrades and rollbacks, peaceful coexistence of different variants of the same package, and more.<br /></p><p>But obviously, this update cost is very high when all you want is to deliver an important security update in a core package. Regarding yesterday’s update, <a href="http://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-refresh.html">guix refresh -l openssl</a> shows that 2,115 packages depend on OpenSSL. On top of that, Guix supports 4 architectures, so needless to say, rebuilding everything that depends on OpenSSL would take time. Sure, users do not have to wait for <a href="http://www.gnu.org/software/guix/manual/en/html_node/Substitutes.html">pre-built binaries</a> and can instead build just what they need locally; in practice, they’d better have a powerful machine, though.<br /></p><h4>Grafting important updates</h4><p>A solution to this problem has been floating around for some time: the idea is to <em>graft</em> important package updates onto packages that depend on it. That way, we would rebuild OpenSSL, but all we need to do for packages that depend on OpenSSL is to substitute the reference to the “broken” OpenSSL with a reference to the security update, with the understanding that this substitution process is orders of magnitude cheaper than rebuilding packages, and faster than redownloading rebuilt packages.<br /></p><p>Shea Levy had implemented a form of grafting <a href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/replace-dependency.nix">in Nixpkgs</a> in 2013, and Guix itself has provided <a href="https://www.gnu.org/software/guix/manual/en/html_node/Security-Updates.html">the infrastructure for grafted updates</a> since <a href="/news/gnu-guix-08-released.html">version 0.8</a> in 2014. With Guix, package developers simply have to <a href="http://git.savannah.gnu.org/cgit/guix.git/commit/?id=caeadfddb01d2cda19d2f761ba9906ef8f162173">define a replacement</a> in the object representing the package that needs an update and the tools automatically pick the replacement and graft it onto packages as needed.<br /></p><p>The problem is that these implementations had a severe limitation, described in <a href="http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22139">this bug report</a>: grafting was not <em>recursive</em>. When we provided a patched OpenSSL to be grafted, any package that directly depended on OpenSSL, would be appropriately grafted to refer to the new OpenSSL. However, if a package depended on libfoo, which in turn depended on OpenSSL, then that package would keep referring to the old libfoo, which refered to the old OpenSSL. That made grafts useless in most situations.<br /></p><h4>Good news!</h4><p>This bug was finally addressed, <a href="https://lists.gnu.org/archive/html/guix-devel/2016-03/msg00009.html">just in time for yesterday’s OpenSSL update</a>. We have identified things to improve, but overall, it has worked pretty well. It has worked so well that we even experienced <a href="http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22876#8">our first ABI break</a> like all <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1313509">real</a><a href="https://bugs.gentoo.org/show_bug.cgi?id=576128">distros</a>!<br /></p><p>From now on, we have confidence that we can deliver important updates quickly using grafts, and happily rebuild the world in the background, whenever is convenient. This is an important improvement for functional package management to keep our users happy and safe.<br /></p><h4>About GNU Guix</h4><p><a href="http://www.gnu.org/software/guix">GNU Guix</a> is a functional package manager for the GNU system. The Guix System Distribution or GuixSD is an advanced distribution of the GNU system that relies on GNU Guix and <a href="http://www.gnu.org/distros/free-system-distribution-guidelines.html">respects the user's freedom</a>.<br /></p><p>In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. Guix uses low-level mechanisms from the Nix package manager, except that packages are defined as native <a href="http://www.gnu.org/software/guile">Guile</a> modules, using extensions to the <a href="http://schemers.org">Scheme</a> language. GuixSD offers a declarative approach to operating system configuration management, and is highly customizable and hackable.<br /></p><p>GuixSD can be used on an i686 or x86_64 machine. It is also possible to use Guix on top of an already installed GNU/Linux system, including on mips64el and armv7.<br /></p></div>https://guix.gnu.org/blog/2015/service-composition-in-guixsd//Service composition in GuixSDLudovic Courtès2015-11-19T00:00:00+01002015-11-19T00:00:00+0100 GuixSD provides a declarative, stateless approach to operating system configuration management. In this context, the mechanism offered to select and compose system services is a crucial one. This post presents the new service framework introduced in the 0.9.0 version of GNU Guix. Declarative Configuration Management GuixSD is not like your parents’ distro. Instead of fiddling with configuration files all around, or running commands that do so as a side effect, the system administrator declares what the system will be like. This takes the form of an …<div><p>GuixSD provides a declarative, stateless approach to operating system configuration management. In this context, the mechanism offered to select and compose system services is a crucial one. This post presents the new service framework introduced in the <a href="/news/gnu-guix-090-released.html">0.9.0 version</a> of GNU Guix.<br /></p><h4>Declarative Configuration Management</h4><p>GuixSD is not like your parents’ distro. Instead of fiddling with configuration files all around, or running commands that do so as a side effect, the system administrator <em>declares</em> what the system will be like. This takes the form of an <a href="https://www.gnu.org/software/guix/manual/en/html_node/Using-the-Configuration-System.html">operating-system declaration</a>, which specifies all the details: file systems, user accounts, locale, timezone, system services, etc.<br /></p><p>If you’re familiar with it, this may remind you of what deployment tools like Ansible and Puppet provide. There is an important difference though: GuixSD takes a stateless—or “purely functional”—approach. This means that instantiating the system with <a href="https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-system.html">guix system</a> always produces the same result, without modifying the current system state. This is what makes it possible to test new system configurations, roll-back to previous ones, and so on. The <a href="https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-system.html">guix system</a> command allows system configurations to be instantiated on the bare metal, in virtual machines, or in <a href="/news/container-provisioning-with-guix.html">containers</a>, which makes it easy to test them.<br /></p><p>In GuixSD, operating-system declarations are first-class objects in the <a href="https://www.gnu.org/software/guile/">host language</a>. They can be inspected at the REPL:<br /></p><div class="example"><pre>scheme@(guile-user)> ,use (gnu)
scheme@(guile-user)> (define os (load "os-config.scm"))
scheme@(guile-user)> (operating-system-kernel os)
$1 = #<package linux-libre-4.2.6 gnu/packages/linux.scm:279 2ea90c0>
scheme@(guile-user)> (length (operating-system-user-services os))
$2 = 30
scheme@(guile-user)> (map user-account-name (operating-system-users os))
$3 = ("alice" "nobody" "root")
</pre></div><p>It is also possible to write functions that take or return OS configurations. For instance, the <a href="http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/system/vm.scm#n382">virtualized-operating-system function</a> returns a variant of the given OS where the set of file systems and the initrd are changed so that the resulting OS can be used in a lightweight virtual machine environment. Likewise for <a href="http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/system/linux-container.scm#n50">containerized-operating-system</a>.<br /></p><h4>Services Beyond Daemons</h4><p>System services are specified in the services field of operating-system declarations, which is a list of service objects. As a user, we want to be able to ideally add one line specifying the <a href="https://www.gnu.org/software/guix/manual/en/html_node/Services.html">system service</a> we want to add, possibly with several instances of a service, and have GuixSD do the right thing.<br /></p><p>Before 0.9.0, GuixSD had a narrow definition of what a “system service” is. Each service in the operating-system configuration had to map to exactly one dmd service—<a href="https://www.gnu.org/software/dmd">GNU dmd</a> is the init system of GuixSD. This would work well in many cases: an SSH server or a log-in daemon is indeed a service that dmd has to take care of, even a file system mount is an operation that can be usefully inserted into dmd’s service dependency graph.<br /></p><p>However, this simple mapping failed to capture more complex service composition patterns. A striking example is “super-daemons”—daemons that can spawn other daemons, such as dbus-daemon or inetd. From the user viewpoint, it does not matter whether a daemon is started by dmd, or by dbus-daemon, or by inetd; this should be transparent. If it’s a D-Bus service, then dbus-daemon’s configuration file should be told about the service; if it’s an inetd service, then inetd.conf should be augmented accordingly; if it’s a dmd service, information on how to start and stop it should go to dmd’s configuration file. Unfortunately, the pre-0.9.0 services could not express such things.<br /></p><p>Worse, this approach did not capture the more general pattern of <em>service extension</em>. In the examples above, the super-daemons are effectively <em>extended</em> by other services that rely on them. But there are many cases where services are similarly extended: <a href="https://wiki.gentoo.org/wiki/Project:Eudev">eudev</a> can be passed new device rules, <a href="http://www.freedesktop.org/wiki/Software/polkit/">polkit</a> can be extended with new rules and actions, the <a href="http://www.linux-pam.org/">Pluggable authentication module system (PAM)</a> can be extended with new services, and so on. At that point it was clear that GuixSD’s naive approach wouldn’t scale.<br /></p><h4>Composing System Services</h4><p>The lesson learned from these observations is that system services <em>extend</em> each other in various way. The new <a href="https://www.gnu.org/software/guix/manual/en/html_node/Service-Composition.html">service composition framework</a> is built around this model: “system services”, broadly defined, can extend each other, and services and their “extends” relationships form a graph. The root of the graph is the operating system itself.<br /></p><p>We can see that this pattern applies to services that are not daemons. PAM is one such example. Accounts are another example: GuixSD provides an “account service” that can be extended with new user accounts or groups; for example, the <a href="https://www.gnu.org/software/guix/manual/en/html_node/Networking-Services.html#index-ntp_002dservice">Network time protocol (NTP) daemon</a> needs to run under the unprivileged “ntp” user, so the NTP service extends the account service with an “ntp” user account. Likewise, the “/etc” service can be extended with new files to be added to /etc; the “setuid” service can be extended with new programs to be made setuid-root. <a href="https://www.gnu.org/software/guix/manual/en/html_node/Service-Reference.html">See the manual</a> for more examples.<br /></p><p>The nice thing is that composition of services is made <em>explicit</em>: extensions can only happen where explicit extension relationships have been <a href="https://www.gnu.org/software/guix/manual/en/html_node/Service-Types-and-Services.html">declared</a>. By looking at the extension graph, users can see how services fit together. The <a href="https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-system.html#system_002dextension_002dgraph">guix system extension-graph</a> command, for instance, takes an operating-system declaration and renders the extension graph in the Graphviz format, making it easy to inspect the OS configuration structure.<br /></p><p>The API makes it easy to see how services contributed to a specific service’s configuration. For instance, the following expression shows the PAM service as extended by other declared services:<br /></p><pre><code class="language-scheme">(fold-services (operating-system-services os)
#:target-type pam-root-service-type)
</code></pre><p>The result is a service object whose value is a list of pam-service objects. Likewise, the following expression returns the /etc service, whose value is a list of entries to be added to /etc:<br /></p><pre><code class="language-scheme">(fold-services (operating-system-services os)
#:target-type etc-service-type)
</code></pre><p>This contrasts with the approach taken by <a href="http://nixos.org/">NixOS</a>, GuixSD’s cousin, and described in this <a href="https://nixos.org/~eelco/pubs/nixos-jfp-final.pdf">2010 paper</a>. In NixOS, the whole system configuration is described in an “attribute set”—a list of key/value associations, similar to JavaScript objects or Python dictionaries. Each NixOS service is passed the whole system configuration, allowing it to inspect and change any part of it.<br /></p><p>This form of <a href="https://en.wikipedia.org/wiki/Ambient_authority">ambient authority</a> gives a lot of flexibility, but it makes it harder to reason about service composition—all a service implementation does is inspect, add, or modify attributes of the global configuration, which may or may not affect other services. The use of a loose key/value dictionary also prevents good error reporting; for instance, a typo in a service name may go undetected. Lastly, NixOS services are enabled by writing service.enable = true stanzas, which leads to complications for services that may have several instances, each with its own configuration.<br /></p><h4>Wrapping Up</h4><p>The new <a href="https://www.gnu.org/software/guix/manual/en/html_node/Service-Composition.html">service composition framework</a> in GuixSD 0.9.0 addresses shortcomings found in previous versions of GuixSD. It simplifies operating-system declarations for users, and provides a highly extensible framework that clearly exposes the way services are composed.<br /></p><p>This new framework has already allowed us to integrate <a href="https://www.gnu.org/software/guix/manual/en/html_node/Desktop-Services.html">Freedesktop and GNOME services</a> in a convenient way. We hope it will prove fruitful as we address other types of services, such as Web services.<br /></p><h4>About GNU Guix</h4><p><a href="http://www.gnu.org/software/guix">GNU Guix</a> is a functional package manager for the GNU system. The Guix System Distribution or GuixSD is an advanced distribution of the GNU system that relies on GNU Guix and <a href="http://www.gnu.org/distros/free-system-distribution-guidelines.html">respects the user's freedom</a>.<br /></p><p>In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. Guix uses low-level mechanisms from the Nix package manager, except that packages are defined as native <a href="http://www.gnu.org/software/guile">Guile</a> modules, using extensions to the <a href="http://schemers.org">Scheme</a> language. GuixSD offers a declarative approach to operating system configuration management, and is highly customizable and hackable.<br /></p><p>GuixSD can be used on an i686 or x86_64 machine. It is also possible to use Guix on top of an already installed GNU/Linux system, including on mips64el and armv7.<br /></p></div>https://guix.gnu.org/blog/2006/purely-functional-software-deployment-model//The Purely Functional Software Deployment Modelsirgazil2006-01-18T12:45:00Z2006-01-18T12:45:00Z Eelco Dolstra's seminal PhD thesis about the Nix package manager,
which Guix is based on . Download (PDF). …<p>Eelco Dolstra's seminal PhD thesis about the Nix package manager,
which <a href="https://www.gnu.org/software/guix/manual/en/html_node/Acknowledgments.html">Guix is based on</a>. <a href="http://nixos.org/~eelco/pubs/phd-thesis.pdf">Download</a> (PDF).</p>