easybuild.tools.modules module

This python module implements the environment modules functionality:
  • loading modules
  • checking for available modules
  • ...
author:Stijn De Weirdt (Ghent University)
author:Dries Verdegem (Ghent University)
author:Kenneth Hoste (Ghent University)
author:Pieter De Baets (Ghent University)
author:Jens Timmerman (Ghent University)
author:David Brown (Pacific Northwest National Laboratory)
class easybuild.tools.modules.EnvironmentModulesC(mod_paths=None, testing=False)

Bases: easybuild.tools.modules.ModulesTool

Interface to (C) environment modules (modulecmd).

COMMAND = 'modulecmd'
REQ_VERSION = '3.2.10'
VERSION_REGEXP = '^\\s*(VERSION\\s*=\\s*)?(?P<version>\\d\\S*)\\s*'
update()

Update after new modules were added.

class easybuild.tools.modules.EnvironmentModulesTcl(mod_paths=None, testing=False)

Bases: easybuild.tools.modules.EnvironmentModulesC

Interface to (Tcl) environment modules (modulecmd.tcl).

COMMAND = 'modulecmd.tcl'
COMMAND_SHELL = ['tclsh']
REQ_VERSION = None
TERSE_OPTION = (1, '-t')
VERSION_OPTION = ''
VERSION_REGEXP = '^Modules\\s+Release\\s+Tcl\\s+(?P<version>\\d\\S*)\\s'
available(mod_name=None)

Return a list of available modules for the given (partial) module name; use None to obtain a list of all available modules.

Parameters:name – a (partial) module name for filtering (default: None)
remove_module_path(path, set_mod_paths=True)

Remove specified module path (using ‘module unuse’).

Parameters:
  • path – path to remove from $MODULEPATH via ‘unuse’
  • set_mod_paths – (re)set self.mod_paths
run_module(*args, **kwargs)

Run module command, tweak output that is exec’ed if necessary.

set_path_env_var(key, paths)

Set environment variable with given name to the given list of paths.

class easybuild.tools.modules.Lmod(*args, **kwargs)

Bases: easybuild.tools.modules.ModulesTool

Interface to Lmod.

COMMAND = 'lmod'
COMMAND_ENVIRONMENT = 'LMOD_CMD'
REQ_VERSION = '5.8'
SHOW_HIDDEN_OPTION = '--show-hidden'
USER_CACHE_DIR = '/home/docs/.lmod.d/.cache'
VERSION_REGEXP = '^Modules\\s+based\\s+on\\s+Lua:\\s+Version\\s+(?P<version>\\d\\S*)\\s'
available(mod_name=None)

Return a list of available modules for the given (partial) module name; use None to obtain a list of all available modules.

Parameters:name – a (partial) module name for filtering (default: None)
check_module_function(*args, **kwargs)

Check whether selected module tool matches ‘module’ function definition.

check_module_output(cmd, stdout, stderr)

Check output of ‘module’ command, see if if is potentially invalid.

compose_cmd_list(args, opts=None)

Compose full module command to run, based on provided arguments

Parameters:args – list of arguments for module command
Returns:list of strings representing the full module command to run
exist(mod_names, skip_avail=False)

Check if modules with specified names exists.

Parameters:
  • mod_names – list of module names
  • skip_avail – skip checking through ‘module avail’, only check via ‘module show’
prepend_module_path(path, set_mod_paths=True)

Prepend given module path to list of module paths, or bump it to 1st place.

Parameters:
  • path – path to prepend to $MODULEPATH
  • set_mod_paths – (re)set self.mod_paths
update()

Update after new modules were added.

class easybuild.tools.modules.Modules(*args, **kwargs)

Bases: easybuild.tools.modules.EnvironmentModulesC

NO LONGER SUPPORTED: interface to modules tool, use modules_tool from easybuild.tools.modules instead

class easybuild.tools.modules.ModulesTool(mod_paths=None, testing=False)

Bases: object

An abstract interface to a tool that deals with modules.

COMMAND = None
COMMAND_ENVIRONMENT = None
COMMAND_SHELL = None
REQ_VERSION = None
TERSE_OPTION = (0, '--terse')
USER_CACHE_DIR = None
VERSION_OPTION = '--version'
VERSION_REGEXP = None
add_module_path(path, set_mod_paths=True)

Add specified module path (using ‘module use’) if it’s not there yet.

Parameters:
  • path – path to add to $MODULEPATH via ‘use’
  • set_mod_paths – (re)set self.mod_paths
available(mod_name=None, extra_args=None)

Return a list of available modules for the given (partial) module name; use None to obtain a list of all available modules.

Parameters:mod_name – a (partial) module name for filtering (default: None)
buildstats()

Return tuple with data to be included in buildstats

check_cmd_avail()

Check whether modules tool command is available.

check_loaded_modules()

Check whether any (EasyBuild-generated) modules are loaded already in the current session

check_module_function(allow_mismatch=False, regex=None)

Check whether selected module tool matches ‘module’ function definition.

check_module_output(cmd, stdout, stderr)

Check output of ‘module’ command, see if if is potentially invalid.

check_module_path()

Check if MODULEPATH is set and change it if necessary.

compose_cmd_list(args, opts=None)

Compose full module command to run, based on provided arguments

Parameters:args – list of arguments for module command
Returns:list of strings representing the full module command to run
exist(mod_names, mod_exists_regex_template='^\\s*\\S*/%s.*:\\s*$', skip_avail=False)

Check if modules with specified names exists.

Parameters:
  • mod_names – list of module names
  • mod_exists_regex_template – template regular expression to search ‘module show’ output with
  • skip_avail – skip checking through ‘module avail’, only check via ‘module show’
exists(mod_name)

NO LONGER SUPPORTED: use exist method instead

get_value_from_modulefile(mod_name, regex)

Get info from the module file for the specified module.

Parameters:
  • mod_name – module name
  • regex – (compiled) regular expression, with one group
interpret_raw_path_lua(txt)

Interpret raw path (Lua syntax): resolve environment variables, join paths where pathJoin is specified

interpret_raw_path_tcl(txt)

Interpret raw path (TCL syntax): resolve environment variables

list()

Return result of ‘module list’.

load(modules, mod_paths=None, purge=False, init_env=None, allow_reload=True)

Load all requested modules.

Parameters:
  • modules – list of modules to load
  • mod_paths – list of module paths to activate before loading
  • purge – whether or not a ‘module purge’ should be run before loading
  • init_env – original environment to restore after running ‘module purge’
  • allow_reload – allow reloading an already loaded module
loaded_modules()

Return a list of loaded modules.

mk_module_cache_key(partial_key)

Create a module cache key, using the specified partial key, by combining it with the current $MODULEPATH.

modpath_extensions_for(mod_names)

Determine dictionary with $MODULEPATH extensions for specified modules. All potential $MODULEPATH extensions are included, even the ones guarded by a condition (which is not checked). Only direct $MODULEPATH extensions are found, no recursion if performed for modules that load other modules. Modules with an empty list of $MODULEPATH extensions are included in the result.

Parameters:mod_names – list of module names for which to determine the list of $MODULEPATH extensions
Returns:dictionary with module names as keys and lists of $MODULEPATH extensions as values
modulefile_path(mod_name, strip_ext=False)

Get the path of the module file for the specified module

Parameters:
  • mod_name – module name
  • strip_ext – strip (.lua) extension from module fileame (if present)
modules

(NO LONGER SUPPORTED!) Property providing access to ‘modules’ class variable

path_to_top_of_module_tree(top_paths, mod_name, full_mod_subdir, deps, modpath_exts=None)

Recursively determine path to the top of the module tree, for given module, module subdir and list of $MODULEPATH extensions per dependency module.

For example, when to determine the path to the top of the module tree for the HPL/2.1 module being installed with a goolf/1.5.14 toolchain in a Core/Compiler/MPI hierarchy (HierarchicalMNS):

  • starting point:
    top_paths = [‘<prefix>’, ‘<prefix>/Core’] mod_name = ‘HPL/2.1’ full_mod_subdir = ‘<prefix>/MPI/Compiler/GCC/4.8.2/OpenMPI/1.6.5’ deps = [‘GCC/4.8.2’, ‘OpenMPI/1.6.5’, ‘OpenBLAS/0.2.8-LAPACK-3.5.0’, ‘FFTW/3.3.4’, ‘ScaLAPACK/...’]
  • 1st iteration: find module that extends $MODULEPATH with ‘<prefix>/MPI/Compiler/GCC/4.8.2/OpenMPI/1.6.5’,
    => OpenMPI/1.6.5 (in ‘<prefix>/Compiler/GCC/4.8.2’ subdir); recurse with mod_name = ‘OpenMPI/1.6.5’ and full_mod_subdir = ‘<prefix>/Compiler/GCC/4.8.2’
  • 2nd iteration: find module that extends $MODULEPATH with ‘<prefix>/Compiler/GCC/4.8.2’
    => GCC/4.8.2 (in ‘<prefix>/Core’ subdir); recurse with mod_name = ‘GCC/4.8.2’ and full_mod_subdir = ‘<prefix>/Core’
  • 3rd iteration: try to find module that extends $MODULEPATH with ‘<prefix>/Core’
    => ‘<prefix>/Core’ is in top_paths, so stop recursion
Parameters:
  • top_paths – list of potentation ‘top of module tree’ (absolute) paths
  • mod_name – (short) module name for starting point (only used in log messages)
  • full_mod_subdir – absolute path to module subdirectory for starting point
  • deps – list of dependency modules for module at starting point
  • modpath_exts – list of module path extensions for each of the dependency modules
prepend_module_path(path, set_mod_paths=True)

Prepend given module path to list of module paths, or bump it to 1st place.

Parameters:
  • path – path to prepend to $MODULEPATH
  • set_mod_paths – (re)set self.mod_paths
purge()

Purge loaded modules.

read_module_file(mod_name)

Read module file with specified name.

remove_module_path(path, set_mod_paths=True)

Remove specified module path (using ‘module unuse’).

Parameters:
  • path – path to remove from $MODULEPATH via ‘unuse’
  • set_mod_paths – (re)set self.mod_paths
run_module(*args, **kwargs)

Run module command.

Parameters:
  • args – list of arguments for module command; first argument should be the subcommand to run
  • kwargs – dictionary with options that control certain aspects of how to run the module command
set_and_check_version()

Get the module version, and check any requirements

set_mod_paths(mod_paths=None)

Set mod_paths, based on $MODULEPATH unless a list of module paths is specified.

Parameters:mod_paths – list of entries for $MODULEPATH to use
set_path_env_var(key, paths)

Set path environment variable to the given list of paths.

show(mod_name)

Run ‘module show’ for the specified module.

unload(modules=None)

Unload all requested modules.

unuse(path)

Remove module path via ‘module unuse’.

update()

Update after new modules were added.

use(path)

Add module path via ‘module use’.

easybuild.tools.modules.avail_modules_tools()

Return all known modules tools.

easybuild.tools.modules.curr_module_paths()

Return a list of current module paths.

easybuild.tools.modules.get_software_libdir(name, only_one=True, fs=None)

Find library subdirectories for the specified software package.

Returns the library subdirectory, relative to software root. It fails if multiple library subdirs are found, unless only_one is False which yields a list of all library subdirs.

Parameters:
  • name – name of the software package
  • only_one – indicates whether only one lib path is expected to be found
  • fs – only retain library subdirs that contain one of the files in this list
easybuild.tools.modules.get_software_root(name, with_env_var=False)

Return the software root set for a particular software name.

easybuild.tools.modules.get_software_root_env_var_name(name)

Return name of environment variable for software root.

easybuild.tools.modules.get_software_version(name)

Return the software version set for a particular software name.

easybuild.tools.modules.get_software_version_env_var_name(name)

Return name of environment variable for software root.

easybuild.tools.modules.invalidate_module_caches_for(path)

Invalidate cache entries related to specified path.

easybuild.tools.modules.mk_module_path(paths)

Create a string representing the list of module paths.

easybuild.tools.modules.modules_tool(mod_paths=None, testing=False)

Return interface to modules tool (environment modules (C, Tcl), or Lmod)

easybuild.tools.modules.reset_module_caches()

Reset module caches.