Writing easyconfig files in YAML syntax (.yeb format) [IN DEVELOPMENT]

Note

Because support for easyconfig files in YAML syntax (a.k.a. .yeb files) is still in development, using them currently requires enabling the use of experimental features (--experimental), see also Experimental features .

An up-to-date overview of current progress on support for .yeb easyconfigs is available at https://github.com/hpcugent/easybuild-framework/issues/1407.

Useful links:

Requirements

To use .yeb easyconfigs, you need to have:

  • an EasyBuild (development) version which is aware of the .yeb format (i.e., version 2.3.0dev or higher)
  • PyYAML installed and available in your Python search path (via $PYTHONPATH for example), such that import yaml works

Syntax

YAML header (optional)

Easyconfig files in YAML syntax can start with a standard YAML header.

It consists of two lines:

  • a line with a ‘%YAMLdirective which also indicates the YAML version being used (the latest YAML version is 1.2, and dates from 2009);
  • followed by a document marker line ‘---‘ (which is used to separate directives from content)

For example:

%YAML 1.2
---

This header is optional, but we recommend including it; one advantage is that editor will use proper syntax highlighting for YAML when the %YAML directory is included.

Comments

Comments can be included anywhere, and are prefixed with a hash character #:

# this is a comment

Internal variables

To define and use temporary/internal variables in easyconfig files, which can be useful to avoid hardcoding (partial) easyconfig parameter values, the YAML anchor/alias functionality can be used (see also http://www.yaml.org/spec/1.2/spec.html#id2765878).

A value can be marked for future reference via an anchor, using the ampersand character ‘&‘. Referring to it later is done using an asterisk character ‘*‘.

Typically, internal variables are defined at the top of the .yeb easyconfig file using a list named _internal_variables_, but this is just a matter of style; anchors can be defined throughout the entire file if desired.

For example, referring to the Python version being used in both the versionsuffix and list of dependencies can be done as follows:

_internal_variables_:
    - &pyver 2.7.10

versionsuffix: !join [-Python-, *pyver]
dependencies:
    - [Python, *pyver]

In this example, the !join is used to concatenate two string values, see also Concatenating strings and/or variables.

A more elaborate example of this is the goolf-1.4.10.yeb example easyconfig.

Concatenating strings and/or variables

The standard YAML format does not support the notion of string concatenation.

Since concatenating string values is a common pattern in easyconfig files, the EasyBuild framework defines the !join operator to support this.

For example, defining a versionsuffix that contains the Python version being used (which may be referred to elsewhere too) can be done as follows:

_internal_variables_:
    - &pyver 2.7.10

versionsuffix: !join [-Python-, *pyver]

Escaping string values with single or double quotes

Strings in YAML can be unquoted. However, when they contain special characters they need to be escaped by either single- or double-quoting the string.

Special characters that require single quotes are: : { } [ ] , & * # ? | - < > = ! % @ and `. When using single-quoted strings, any single quote inside the string must be doubled to escape it.

If the string contains control characters such as n, it must be escaped with double quotes.

Easyconfig parameter values

To define an easyconfig parameter, simply use <key>: <value> (i.e., use a colon : as a separator).

In YAML terminology, an easyconfig file is expressed as a mapping, with easyconfig parameters as keys.

Three types of values (nodes) are supported: scalars (strings, integers), sequences (lists) and mappings (dictionaries).

Scalar values

Using scalar values is straight-forward, no special syntax is required.

For string values, no quotes must be used (in general). However, quotes are sometimes required to escape characters that have special meaning in YAML (like ‘:‘). (Also see: Escaping string values with single or double quotes) It’s worth noting that there’s a subtle difference between using single and double quotes, see Flow Scalar Styles.

Examples:

name: gzip
version: 1.6

# single quotes are required for string values representing URLs, to escape the ':'
homepage: 'http://www.gnu.org/software/gzip/'

parallel: 1

Multiline strings can be expressed using indentation:

description:
    gzip is a popular data compression program
    as a replacement for compress

Sequences

Sequence values (a.k.a. lists) can be expressed in different ways, depending on their size.

If there are a limited number of (short) entries the value can be expressed on a single line, using square brackets ‘[‘ ‘]‘ and with comma ‘,‘ as separator.

Example:

# quotes are required to escape the ':'
source_urls: ['http://ftpmirror.gnu.org/gzip/', 'ftp://ftp.gnu.org/gnu/gzip/']

Alternatively indentation can be used for scope, with each entry on its own line, indicated with a dash and a space ``- ``.

Example:

# no quotes required here, since there's no ambiguity w.r.t. ':'
source_urls:
    - http://ftpmirror.gnu.org/gzip/
    - http://ftp.gnu.org/gnu/gzip/
    - ftp://ftp.gnu.org/gnu/gzip/

Mappings

Mapping values (a.k.a. dictionaries) are expressed using a colon ‘:‘ and space as key-value separator, a comma ‘,‘ to key-value pairs, and curly braces ‘{‘ ‘}‘ to mark the start/end.

For example:

toolchain: {name: intel, version: 2015b}

Nesting

Different types of values can be nested.

For example, sequence values can be used in a mapping:

sanity_check_paths: {
    files: [bin/gunzip, bin/gzip, bin/uncompress],
    dirs: [],
}

And sequences of sequences are also supported:

osdependencies
    - zlib
    - [openssl-devel, libssl-dev, libopenssl-devel]

Templates values and constants

Template values can be specified as a part of string values, using %(template_name).

Template constants are injected by the easyconfig .yeb parser as node anchors, and can be referred to with an alias node, i.e. using an asterisk *.

For example:

source_urls: [*GNU_SOURCE]
sources: ['%(name)s-%(version)s.tar.gz']  # equivalent with [*SOURCE_TAR_GZ]

See also Dynamic values for easyconfig parameters.

Dependencies

We updated the way dependencies are specified to match with the new toolchain format (OS dependencies and sanity check paths) The format is a bit more verbose than before, but easier to read. Each dependency is a list entry, indicated by a dash and space (- `). Each entry can specify a ``name: version` key-value pair, and a versionsuffix and toolchain. Only the name: version pair is required.

Dependencies can also be external modules. In this case, the dependency has to be specified with a name and the marker external_module: True. The boolean value is not case-sensitive.

A straightforward example:

dependencies:
    - libreadline: 6.3
    - Tcl: 8.6.4
    - name: fftw/3.3.4.4
      external_module: True

builddependencies:
    # empty versionsuffix, different toolchain (GCC/4.9.2)
    - CMake: 3.2.2
      toolchain: GCC, 4.9.2

A more complicated example from a toolchain easyconfig, where also the !join operator (see Concatenating strings and/or variables) and internal variables (see Internal variables) are used:

_internal_variables_:
    - &comp_name GCC
    - &comp_version 4.7.2
    - &comp [*comp_name, *comp_version]

    - &blaslib OpenBLAS
    - &blasver 0.2.6
    - &blas !join [*blaslib, -, *blasver]
    - &blas_suff -LAPACK-3.4.2

    - &comp_mpi_tc [gompi, 1.4.10]

dependencies:
    - *comp_name: *comp_version
    - OpenMPI: 1.6.4
      toolchain: *comp
    - *blaslib: *blasver
      versionsuffix: *blas_suff
      toolchain: *comp_mpi_tc
    - FFTW: 3.3.3
      toolchain: *comp_mpi_tc
    - ScaLAPACK: 2.0.2
      versionsuffix: !join [-, *blas, *blas_suff]
      toolchain: *comp_mpi_tc

For the full version of this easyconfig file, see the example .yeb easyconfig goolf-1.4.10.yeb.

OS dependencies and sanity check paths

To specify parameters that used to contain tuples such as osdependencies and sanity_check_paths, simply use lists (sequences) instead of tuples.

For example:

# note: this is eb syntax, will not work in .yeb files
osdependencies = [('openssl-devel', 'libssl-dev', 'libopenssl-devel')]

Becomes:

osdependencies: [[openssl-devel, libssl-dev, libopenssl-devel]]

And:

# note: this is eb syntax, will not work in .yeb files
sanity_check_paths = {
    'files': ['fileA', ('fileB', 'fileC')],
    'dirs' : ['dirA', 'dirB'],
}

Becomes:

sanity_check_paths: {
    files: [fileA, [fileB, fileC]],
    dirs: [dirA, dirB]
}

Shorthands for common easyconfig parameters

Toolchain format

The easyconfig parameter toolchain in .eb files is defined as a dictionary {'name':'foo', 'version':'bar'}. In the .yeb format, this can be done much easier by just using name, version. E.g:

# note: this is eb syntax, will not work in yeb files
toolchain = {'name':'intel', 'version':'2015b'}

becomes:

toolchain: intel, 2015b

Examples

gzip-1.6-GCC-4.9.2.yeb

Example easyconfig for gzip v1.6 using the GCC/4.9.2 toolchain.

%YAML 1.2
---
easyblock: ConfigureMake

name: gzip
version: 1.6

homepage: 'http://www.gnu.org/software/gzip/'
description:
    gzip is a popular data compression program
    as a replacement for compress

toolchain: GCC, 4.9.2

# http://ftp.gnu.org/gnu/gzip/gzip-1.6.tar.gz
source_urls: [*GNU_SOURCE]
sources: [%(name)s-%(version)s.tar.gz]

# make sure the gzip, gunzip and compress binaries are available after installation
sanity_check_paths: {
    files: [bin/gunzip, bin/gzip, bin/uncompress],
    dirs: [],
}

moduleclass: tools

goolf-1.4.10.yeb

Easyconfig file in YAML syntax for the goolf v1.4.10 toolchain.

_internal_variables_:
    - &version 1.4.10

    - &comp_name GCC
    - &comp_version 4.7.2
    - &comp [*comp_name, *comp_version]

    - &blaslib OpenBLAS
    - &blasver 0.2.6
    - &blas !join [*blaslib, -, *blasver]
    - &blas_suff -LAPACK-3.4.2

    - &comp_mpi_tc [gompi, *version]


easyblock: Toolchain

name: goolf
version: *version

homepage: (none)
description: |
    GNU Compiler Collection (GCC) based compiler toolchain, including
    OpenMPI for MPI support, OpenBLAS (BLAS and LAPACK support), FFTW and ScaLAPACK.

toolchain: {name: dummy, version: dummy}

# compiler toolchain dependencies
# we need GCC and OpenMPI as explicit dependencies instead of gompi toolchain
# because of toolchain preperation functions
    dependencies:
        - *comp_name: *comp_version
        - OpenMPI: 1.6.4
          toolchain: *comp
        - *blaslib: *blasver
          versionsuffix: *blas_suff
          toolchain: *comp_mpi_tc
        - FFTW: 3.3.3
          toolchain: *comp_mpi_tc
        - ScaLAPACK: 2.0.2
          versionsuffix: !join [-, *blas, *blas_suff]
          toolchain: *comp_mpi_tc

moduleclass: toolchain

Python-2.7.10-intel-2015b.yeb

_internal_variables_:
    - &numpyversion 1.9.2
    - &scipyversion 0.15.1

easyblock: ConfigureMake

name: Python
version: 2.7.10

homepage: http://python.org/
description: |
    Python is a programming language that lets you work more quickly and integrate your systems
    more effectively.

toolchain: intel, 2015b
toolchainopts: {pic: True, opt: True, optarch: True}

source_urls: ['http://www.python.org/ftp/python/%(version)s/']
sources: [*SOURCE_TGZ]

# python needs bzip2 to build the bz2 package
dependencies: [
    - bzip2: 1.0.6
    - zlib: 1.2.8
    - libreadline: 6.3
    - ncurses: 5.9
    - SQLite: 3.8.10.2
    - Tk: 8.4.6
      versionsuffix: -no-X11
  # - OpenSSL: 1.0.1m
  #   OS dependency should be preferred if the os version is more recent then this version, its
  #   nice to have an up to date openssl for security reasons
]

osdependencies: [[openssl-devel, libssl-dev, libopenssl-devel]]

# order is important!
# package versions updated May 28th 2015
exts_list: [
    [setuptools, '16.0', {
        source_urls: ["https://pypi.python.org/packages/source/s/setuptools/"],
    }],
    [pip, 7.0.1, {
        source_urls: ["https://pypi.python.org/packages/source/p/pip/"],
    }],
    [nose, 1.3.6, {
        source_urls: ["https://pypi.python.org/packages/source/n/nose/"],
    }],
    [numpy, *numpyversion, {
        source_urls: [
            [!join ["http://sourceforge.net/projects/numpy/files/NumPy/", *numpyversion], download]
        ],
        patches: [
            numpy-1.8.0-mkl.patch, # % numpyversion,
        ],
    }],
    [scipy, *scipyversion, {
        source_urls: [
            [!join ["http://sourceforge.net/projects/scipy/files/scipy/", *scipyversion], download]],
    }],
    [blist, 1.3.6, {
        source_urls: ["https://pypi.python.org/packages/source/b/blist/"],
    }],
    [mpi4py, 1.3.1, {
        source_urls: ["http://bitbucket.org/mpi4py/mpi4py/downloads/"],
    }],
    [paycheck, 1.0.2, {
        source_urls: ["https://pypi.python.org/packages/source/p/paycheck/"],
    }],
    [argparse, 1.3.0, {
        source_urls: ["https://pypi.python.org/packages/source/a/argparse/"],
    }],
    [pbr, 1.0.1, {
        source_urls: ["https://pypi.python.org/packages/source/p/pbr/"],
    }],
    [lockfile, 0.10.2, {
        source_urls: ["https://pypi.python.org/packages/source/l/lockfile/"],
    }],
    [Cython, '0.22', {
        source_urls: ["http://www.cython.org/release/"],
    }],
    [six, 1.9.0, {
        source_urls: ["https://pypi.python.org/packages/source/s/six/"],
    }],
    [dateutil, 2.4.2, {
        source_tmpl: python-%(name)s-%(version)s.tar.gz,
        source_urls: ["https://pypi.python.org/packages/source/p/python-dateutil/"],
    }],
    [deap, 1.0.2, {
        # escaped with quotes because yaml values can't start with %
        source_tmpl: "%(name)s-%(version)s.post2.tar.gz",
        source_urls: ["https://pypi.python.org/packages/source/d/deap/"],
    }],
    [decorator, 3.4.2, {
        source_urls: ["https://pypi.python.org/packages/source/d/decorator/"],
    }],
    [arff, 2.0.2, {
        source_tmpl: liac-%(name)s-%(version)s.zip,
        source_urls: ["https://pypi.python.org/packages/source/l/liac-arff/"],
    }],
    [pycrypto, 2.6.1, {
        modulename: Crypto,
        source_urls: ["http://ftp.dlitz.net/pub/dlitz/crypto/pycrypto/"],
    }],
    [ecdsa, '0.13', {
        source_urls: ["https://pypi.python.org/packages/source/e/ecdsa/"],
    }],
    [paramiko, 1.15.2, {
        source_urls: ["https://pypi.python.org/packages/source/p/paramiko/"],
    }],
    [pyparsing, 2.0.3, {
        source_urls: ["https://pypi.python.org/packages/source/p/pyparsing/"],
    }],
    [netifaces, 0.10.4, {
        source_urls: ["https://pypi.python.org/packages/source/n/netifaces"],
    }],
    [netaddr, 0.7.14, {
        source_urls: ["https://pypi.python.org/packages/source/n/netaddr"],
    }],
    [mock, 1.0.1, {
        source_urls: ["https://pypi.python.org/packages/source/m/mock"],
    }],
    [pytz, '2015.4', {
        source_urls: ["https://pypi.python.org/packages/source/p/pytz"],
    }],
    [pandas, 0.16.1, {
        source_urls: ["https://pypi.python.org/packages/source/p/pandas"],
    }],
]

moduleclass: lang