Previous: , Up: Programming Interface   [Contents][Index]

5.6 G-Expressions

So we have “derivations”, which represent a sequence of build actions to be performed to produce an item in the store (see Derivations). These build actions are performed when asking the daemon to actually build the derivations; they are run by the daemon in a container (see Invoking guix-daemon).

It should come as no surprise that we like to write these build actions in Scheme. When we do that, we end up with two strata of Scheme code9: the “host code”—code that defines packages, talks to the daemon, etc.—and the “build code”—code that actually performs build actions, such as making directories, invoking make, etc.

To describe a derivation and its build actions, one typically needs to embed build code inside host code. It boils down to manipulating build code as data, and the homoiconicity of Scheme—code has a direct representation as data—comes in handy for that. But we need more than the normal quasiquote mechanism in Scheme to construct build expressions.

The (guix gexp) module implements G-expressions, a form of S-expressions adapted to build expressions. G-expressions, or gexps, consist essentially of three syntactic forms: gexp, ungexp, and ungexp-splicing (or simply: #~, #$, and #$@), which are comparable to quasiquote, unquote, and unquote-splicing, respectively (see quasiquote in GNU Guile Reference Manual). However, there are major differences:

This mechanism is not limited to package and derivation objects: compilers able to “lower” other high-level objects to derivations or files in the store can be defined, such that these objects can also be inserted into gexps. For example, a useful type of high-level objects that can be inserted in a gexp is “file-like objects”, which make it easy to add files to the store and to refer to them in derivations and such (see local-file and plain-file below.)

To illustrate the idea, here is an example of a gexp:

(define build-exp
      (mkdir #$output)
      (chdir #$output)
      (symlink (string-append #$coreutils "/bin/ls")

This gexp can be passed to gexp->derivation; we obtain a derivation that builds a directory containing exactly one symlink to /gnu/store/…-coreutils-8.22/bin/ls:

(gexp->derivation "the-thing" build-exp)

As one would expect, the "/gnu/store/…-coreutils-8.22" string is substituted to the reference to the coreutils package in the actual build code, and coreutils is automatically made an input to the derivation. Likewise, #$output (equivalent to (ungexp output)) is replaced by a string containing the directory name of the output of the derivation.

In a cross-compilation context, it is useful to distinguish between references to the native build of a package—that can run on the host—versus references to cross builds of a package. To that end, the #+ plays the same role as #$, but is a reference to a native package build:

(gexp->derivation "vi"
       (mkdir #$output)
       (system* (string-append #+coreutils "/bin/ln")
                (string-append #$emacs "/bin/emacs")
                (string-append #$output "/bin/vi")))
   #:target "mips64el-linux-gnu")

In the example above, the native build of coreutils is used, so that ln can actually run on the host; but then the cross-compiled build of emacs is referenced.

Another gexp feature is imported modules: sometimes you want to be able to use certain Guile modules from the “host environment” in the gexp, so those modules should be imported in the “build environment”. The with-imported-modules form allows you to express that:

(let ((build (with-imported-modules '((guix build utils))
                   (use-modules (guix build utils))
                   (mkdir-p (string-append #$output "/bin"))))))
  (gexp->derivation "empty-dir"
                        (display "success!\n")

In this example, the (guix build utils) module is automatically pulled into the isolated build environment of our gexp, such that (use-modules (guix build utils)) works as expected.

Usually you want the closure of the module to be imported—i.e., the module itself and all the modules it depends on—rather than just the module; failing to do that, attempts to use the module will fail because of missing dependent modules. The source-module-closure procedure computes the closure of a module by looking at its source file headers, which comes in handy in this case:

(use-modules (guix modules))   ;for 'source-module-closure'

(with-imported-modules (source-module-closure
                         '((guix build utils)
                           (gnu build vm)))
  (gexp->derivation "something-with-vms"
                        (use-modules (guix build utils)
                                     (gnu build vm))

The syntactic form to construct gexps is summarized below.

Scheme Syntax: #~exp
Scheme Syntax: (gexp exp)

Return a G-expression containing exp. exp may contain one or more of the following forms:

(ungexp obj)

Introduce a reference to obj. obj may have one of the supported types, for example a package or a derivation, in which case the ungexp form is replaced by its output file name—e.g., "/gnu/store/…-coreutils-8.22.

If obj is a list, it is traversed and references to supported objects are substituted similarly.

If obj is another gexp, its contents are inserted and its dependencies are added to those of the containing gexp.

If obj is another kind of object, it is inserted as is.

(ungexp obj output)

This is like the form above, but referring explicitly to the output of obj—this is useful when obj produces multiple outputs (see Packages with Multiple Outputs).

(ungexp-native obj)
(ungexp-native obj output)

Same as ungexp, but produces a reference to the native build of obj when used in a cross compilation context.

(ungexp output [output])

Insert a reference to derivation output output, or to the main output when output is omitted.

This only makes sense for gexps passed to gexp->derivation.

(ungexp-splicing lst)

Like the above, but splices the contents of lst inside the containing list.

(ungexp-native-splicing lst)

Like the above, but refers to native builds of the objects listed in lst.

G-expressions created by gexp or #~ are run-time objects of the gexp? type (see below.)

Scheme Syntax: with-imported-modules modules body

Mark the gexps defined in body… as requiring modules in their execution environment. modules must be a list of Guile module names, such as '((guix build utils) (guix build gremlin)).

This form has lexical scope: it has an effect on the gexps directly defined in body…, but not on those defined, say, in procedures called from body….

Scheme Procedure: gexp? obj

Return #t if obj is a G-expression.

G-expressions are meant to be written to disk, either as code building some derivation, or as plain files in the store. The monadic procedures below allow you to do that (see The Store Monad, for more information about monads.)

Monadic Procedure: gexp->derivation name exp [#:system (%current-system)] [#:target #f] [#:graft? #t] [#:hash #f] [#:hash-algo #f] [#:recursive? #f] [#:env-vars '()] [#:modules '()] [#:module-path %load-path] [#:references-graphs #f] [#:allowed-references #f] [#:disallowed-references #f] [#:leaked-env-vars #f] [#:script-name (string-append name "-builder")] [#:local-build? #f] [#:substitutable? #t] [#:guile-for-build #f]

Return a derivation name that runs exp (a gexp) with guile-for-build (a derivation) on system; exp is stored in a file called script-name. When target is true, it is used as the cross-compilation target triplet for packages referred to by exp.

modules is deprecated in favor of with-imported-modules. Its meaning is to make modules available in the evaluation context of exp; modules is a list of names of Guile modules searched in module-path to be copied in the store, compiled, and made available in the load path during the execution of exp—e.g., ((guix build utils) (guix build gnu-build-system)).

graft? determines whether packages referred to by exp should be grafted when applicable.

When references-graphs is true, it must be a list of tuples of one of the following forms:

(file-name package)
(file-name package output)
(file-name derivation)
(file-name derivation output)
(file-name store-item)

The right-hand-side of each element of references-graphs is automatically made an input of the build process of exp. In the build environment, each file-name contains the reference graph of the corresponding item, in a simple text format.

allowed-references must be either #f or a list of output names and packages. In the latter case, the list denotes store items that the result is allowed to refer to. Any reference to another store item will lead to a build error. Similarly for disallowed-references, which can list items that must not be referenced by the outputs.

The other arguments are as for derivation (see Derivations).

The local-file, plain-file, computed-file, program-file, and scheme-file procedures below return file-like objects. That is, when unquoted in a G-expression, these objects lead to a file in the store. Consider this G-expression:

#~(system* #$(file-append glibc "/sbin/nscd") "-f"
           #$(local-file "/tmp/my-nscd.conf"))

The effect here is to “intern” /tmp/my-nscd.conf by copying it to the store. Once expanded, for instance via gexp->derivation, the G-expression refers to that copy under /gnu/store; thus, modifying or removing the file in /tmp does not have any effect on what the G-expression does. plain-file can be used similarly; it differs in that the file content is directly passed as a string.

Scheme Procedure: local-file file [name] [#:recursive? #f] [#:select? (const #t)]

Return an object representing local file file to add to the store; this object can be used in a gexp. If file is a relative file name, it is looked up relative to the source file where this form appears. file will be added to the store under name–by default the base name of file.

When recursive? is true, the contents of file are added recursively; if file designates a flat file and recursive? is true, its contents are added, and its permission bits are kept.

When recursive? is true, call (select? file stat) for each directory entry, where file is the entry’s absolute file name and stat is the result of lstat; exclude entries for which select? does not return true.

This is the declarative counterpart of the interned-file monadic procedure (see interned-file).

Scheme Procedure: plain-file name content

Return an object representing a text file called name with the given content (a string) to be added to the store.

This is the declarative counterpart of text-file.

Scheme Procedure: computed-file name gexp [#:options '(#:local-build? #t)]

Return an object representing the store item name, a file or directory computed by gexp. options is a list of additional arguments to pass to gexp->derivation.

This is the declarative counterpart of gexp->derivation.

Monadic Procedure: gexp->script name exp

Return an executable script name that runs exp using guile, with exp’s imported modules in its search path.

The example below builds a script that simply invokes the ls command:

(use-modules (guix gexp) (gnu packages base))

(gexp->script "list-files"
              #~(execl #$(file-append coreutils "/bin/ls")

When “running” it through the store (see run-with-store), we obtain a derivation that produces an executable file /gnu/store/…-list-files along these lines:

#!/gnu/store/…-guile-2.0.11/bin/guile -ds
(execl "/gnu/store/…-coreutils-8.22"/bin/ls" "ls")
Scheme Procedure: program-file name exp [#:guile #f]

Return an object representing the executable store item name that runs gexp. guile is the Guile package used to execute that script.

This is the declarative counterpart of gexp->script.

Monadic Procedure: gexp->file name exp [#:set-load-path? #t]

Return a derivation that builds a file name containing exp. When set-load-path? is true, emit code in the resulting file to set %load-path and %load-compiled-path to honor exp’s imported modules.

The resulting file holds references to all the dependencies of exp or a subset thereof.

Scheme Procedure: scheme-file name exp

Return an object representing the Scheme file name that contains exp.

This is the declarative counterpart of gexp->file.

Monadic Procedure: text-file* name text

Return as a monadic value a derivation that builds a text file containing all of text. text may list, in addition to strings, objects of any type that can be used in a gexp: packages, derivations, local file objects, etc. The resulting store file holds references to all these.

This variant should be preferred over text-file anytime the file to create will reference items from the store. This is typically the case when building a configuration file that embeds store file names, like this:

(define (
  ;; Return the name of a shell script in the store that
  ;; initializes the 'PATH' environment variable.
  (text-file* ""
              "export PATH=" coreutils "/bin:"
              grep "/bin:" sed "/bin\n"))

In this example, the resulting /gnu/store/… file will reference coreutils, grep, and sed, thereby preventing them from being garbage-collected during its lifetime.

Scheme Procedure: mixed-text-file name text

Return an object representing store file name containing text. text is a sequence of strings and file-like objects, as in:

(mixed-text-file "profile"
                 "export PATH=" coreutils "/bin:" grep "/bin")

This is the declarative counterpart of text-file*.

Scheme Procedure: file-append obj suffix

Return a file-like object that expands to the concatenation of obj and suffix, where obj is a lowerable object and each suffix is a string.

As an example, consider this gexp:

(gexp->script "run-uname"
              #~(system* #$(file-append coreutils

The same effect could be achieved with:

(gexp->script "run-uname"
              #~(system* (string-append #$coreutils

There is one difference though: in the file-append case, the resulting script contains the absolute file name as a string, whereas in the second case, the resulting script contains a (string-append …) expression to construct the file name at run time.

Of course, in addition to gexps embedded in “host” code, there are also modules containing build tools. To make it clear that they are meant to be used in the build stratum, these modules are kept in the (guix build …) name space.

Internally, high-level objects are lowered, using their compiler, to either derivations or store items. For instance, lowering a package yields a derivation, and lowering a plain-file yields a store item. This is achieved using the lower-object monadic procedure.

Monadic Procedure: lower-object obj [system] [#:target #f]

Return as a value in %store-monad the derivation or store item corresponding to obj for system, cross-compiling for target if target is true. obj must be an object that has an associated gexp compiler, such as a <package>.



The term stratum in this context was coined by Manuel Serrano et al. in the context of their work on Hop. Oleg Kiselyov, who has written insightful essays and code on this topic, refers to this kind of code generation as staging.

Previous: , Up: Programming Interface   [Contents][Index]