modules
This python module implements the environment modules functionality: - loading modules - checking for available modules - ...
Authors:
- Stijn De Weirdt (Ghent University)
- Dries Verdegem (Ghent University)
- Kenneth Hoste (Ghent University)
- Pieter De Baets (Ghent University)
- Jens Timmerman (Ghent University)
- David Brown (Pacific Northwest National Laboratory)
EnvironmentModules
¶
Bases: EnvironmentModulesTcl
Interface to environment modules 4.0+
__init__(*args, **kwargs)
¶
Constructor, set Environment Modules-specific class variable values.
available(mod_name=None, extra_args=None)
¶
Return a list of available modules for the given (partial) module name; use None to obtain a list of all available modules.
| PARAMETER | DESCRIPTION |
|---|---|
mod_name |
a (partial) module name for filtering (default: None)
DEFAULT:
|
check_module_function(allow_mismatch=False, regex=None)
¶
Check whether selected module tool matches 'module' function definition.
check_module_output(cmd, stdout, stderr)
¶
Check output of 'module' command, see if if is potentially invalid.
get_setenv_value_from_modulefile(mod_name, var_name)
¶
Get value for specific 'setenv' statement from module file for the specified module.
| PARAMETER | DESCRIPTION |
|---|---|
mod_name |
module name
|
var_name |
name of the variable being set for which value should be returned
|
EnvironmentModulesC
¶
Bases: ModulesTool
Interface to (C) environment modules (modulecmd).
get_setenv_value_from_modulefile(mod_name, var_name)
¶
Get value for specific 'setenv' statement from module file for the specified module.
| PARAMETER | DESCRIPTION |
|---|---|
mod_name |
module name
|
var_name |
name of the variable being set for which value should be returned
|
run_module(*args, **kwargs)
¶
Run module command, tweak output that is exec'ed if necessary.
update()
¶
Update after new modules were added.
EnvironmentModulesTcl
¶
Bases: EnvironmentModulesC
Interface to (Tcl) environment modules (modulecmd.tcl).
available(mod_name=None, extra_args=None)
¶
Return a list of available modules for the given (partial) module name; use None to obtain a list of all available modules.
| PARAMETER | DESCRIPTION |
|---|---|
mod_name |
a (partial) module name for filtering (default: None)
DEFAULT:
|
remove_module_path(path, set_mod_paths=True)
¶
Remove specified module path (using 'module unuse').
| PARAMETER | DESCRIPTION |
|---|---|
path |
path to remove from $MODULEPATH via 'unuse'
|
set_mod_paths |
(re)set self.mod_paths
DEFAULT:
|
run_module(*args, **kwargs)
¶
Run module command, tweak output that is exec'ed if necessary.
set_path_env_var(key, paths)
¶
Set environment variable with given name to the given list of paths.
Lmod
¶
Bases: ModulesTool
Interface to Lmod.
__init__(*args, **kwargs)
¶
Constructor, set lmod-specific class variable values.
available(mod_name=None)
¶
Return a list of available modules for the given (partial) module name; use None to obtain a list of all available modules.
| PARAMETER | DESCRIPTION |
|---|---|
mod_name |
a (partial) module name for filtering (default: None)
DEFAULT:
|
check_module_function(*args, **kwargs)
¶
Check whether selected module tool matches 'module' function definition.
check_module_output(cmd, stdout, stderr)
¶
Check output of 'module' command, see if if is potentially invalid.
compose_cmd_list(args, opts=None)
¶
Compose full module command to run, based on provided arguments
| PARAMETER | DESCRIPTION |
|---|---|
args |
list of arguments for module command
|
| RETURNS | DESCRIPTION |
|---|---|
|
list of strings representing the full module command to run |
get_setenv_value_from_modulefile(mod_name, var_name)
¶
Get value for specific 'setenv' statement from module file for the specified module.
| PARAMETER | DESCRIPTION |
|---|---|
mod_name |
module name
|
var_name |
name of the variable being set for which value should be returned
|
module_wrapper_exists(mod_name)
¶
Determine whether a module wrapper with specified name exists. First check for wrapper defined in .modulerc.lua, fall back to also checking .modulerc (Tcl syntax).
prepend_module_path(path, set_mod_paths=True, priority=None)
¶
Prepend given module path to list of module paths, or bump it to 1st place.
| PARAMETER | DESCRIPTION |
|---|---|
path |
path to prepend to $MODULEPATH
|
set_mod_paths |
(re)set self.mod_paths
DEFAULT:
|
priority |
priority for this path in $MODULEPATH (Lmod-specific)
DEFAULT:
|
unuse(path)
¶
Remove a module path
update()
¶
Update after new modules were added.
use(path, priority=None)
¶
Add path to $MODULEPATH via 'module use'.
| PARAMETER | DESCRIPTION |
|---|---|
path |
path to add to $MODULEPATH
|
priority |
priority for this path in $MODULEPATH (Lmod-specific)
DEFAULT:
|
Modules
¶
Bases: EnvironmentModulesC
NO LONGER SUPPORTED: interface to modules tool, use modules_tool from easybuild.tools.modules instead
ModulesTool
¶
Bases: object
An abstract interface to a tool that deals with modules.
__init__(mod_paths=None, testing=False)
¶
Create a ModulesTool object
| PARAMETER | DESCRIPTION |
|---|---|
mod_paths |
A list of paths where the modules can be located @type mod_paths: list
DEFAULT:
|
__str__()
¶
String representation of this ModulesTool instance.
add_module_path(path, set_mod_paths=True)
¶
Add specified module path (using 'module use') if it's not there yet.
| PARAMETER | DESCRIPTION |
|---|---|
path |
path to add to $MODULEPATH via 'use'
|
set_mod_paths |
(re)set self.mod_paths
DEFAULT:
|
available(mod_name=None, extra_args=None)
¶
Return a list of available modules for the given (partial) module name; use None to obtain a list of all available modules.
| PARAMETER | DESCRIPTION |
|---|---|
mod_name |
a (partial) module name for filtering (default: None)
DEFAULT:
|
buildstats()
¶
Return tuple with data to be included in buildstats
check_cmd_avail()
¶
Check whether modules tool command is available.
check_loaded_modules()
¶
Check whether any (EasyBuild-generated) modules are loaded already in the current session
check_module_function(allow_mismatch=False, regex=None)
¶
Check whether selected module tool matches 'module' function definition.
check_module_output(cmd, stdout, stderr)
¶
Check output of 'module' command, see if if is potentially invalid.
check_module_path()
¶
Check if MODULEPATH is set and change it if necessary.
compose_cmd_list(args, opts=None)
¶
Compose full module command to run, based on provided arguments
| PARAMETER | DESCRIPTION |
|---|---|
args |
list of arguments for module command
|
| RETURNS | DESCRIPTION |
|---|---|
|
list of strings representing the full module command to run |
exist(mod_names, mod_exists_regex_template=None, skip_avail=False, maybe_partial=True)
¶
Check if modules with specified names exists.
| PARAMETER | DESCRIPTION |
|---|---|
mod_names |
list of module names
|
mod_exists_regex_template |
DEPRECATED and unused
DEFAULT:
|
skip_avail |
skip checking through 'module avail', only check via 'module show'
DEFAULT:
|
maybe_partial |
indicates if the module name may be a partial module name
DEFAULT:
|
get_setenv_value_from_modulefile(mod_name, var_name)
¶
Get value for specific 'setenv' statement from module file for the specified module.
| PARAMETER | DESCRIPTION |
|---|---|
mod_name |
module name
|
var_name |
name of the variable being set for which value should be returned
|
get_value_from_modulefile(mod_name, regex, strict=True)
¶
Get info from the module file for the specified module.
| PARAMETER | DESCRIPTION |
|---|---|
mod_name |
module name
|
regex |
(compiled) regular expression, with one group
|
interpret_raw_path_lua(txt)
¶
Interpret raw path (Lua syntax): resolve environment variables, join paths where pathJoin is specified
interpret_raw_path_tcl(txt)
¶
Interpret raw path (TCL syntax): resolve environment variables
list()
¶
Return result of 'module list'.
load(modules, mod_paths=None, purge=False, init_env=None, allow_reload=True)
¶
Load all requested modules.
| PARAMETER | DESCRIPTION |
|---|---|
modules |
list of modules to load
|
mod_paths |
list of module paths to activate before loading
DEFAULT:
|
purge |
whether or not a 'module purge' should be run before loading
DEFAULT:
|
init_env |
original environment to restore after running 'module purge'
DEFAULT:
|
allow_reload |
allow reloading an already loaded module
DEFAULT:
|
loaded_modules()
¶
Return a list of loaded modules.
mk_module_cache_key(partial_key)
¶
Create a module cache key, using the specified partial key, by combining it with the current $MODULEPATH.
modpath_extensions_for(mod_names)
¶
Determine dictionary with $MODULEPATH extensions for specified modules. All potential $MODULEPATH extensions are included, even the ones guarded by a condition (which is not checked). Only direct $MODULEPATH extensions are found, no recursion if performed for modules that load other modules. Modules with an empty list of $MODULEPATH extensions are included in the result.
| PARAMETER | DESCRIPTION |
|---|---|
mod_names |
list of module names for which to determine the list of $MODULEPATH extensions
|
| RETURNS | DESCRIPTION |
|---|---|
|
dictionary with module names as keys and lists of $MODULEPATH extensions as values |
module_wrapper_exists(mod_name, modulerc_fn='.modulerc', mod_wrapper_regex_template=None)
¶
Determine whether a module wrapper with specified name exists. Only .modulerc file in Tcl syntax is considered here.
modulefile_path(mod_name, strip_ext=False)
¶
Get the path of the module file for the specified module
| PARAMETER | DESCRIPTION |
|---|---|
mod_name |
module name
|
strip_ext |
strip (.lua) extension from module fileame (if present)
DEFAULT:
|
path_to_top_of_module_tree(top_paths, mod_name, full_mod_subdir, deps, modpath_exts=None)
¶
Recursively determine path to the top of the module tree, for given module, module subdir and list of $MODULEPATH extensions per dependency module.
For example, when to determine the path to the top of the module tree for the HPL/2.1 module being installed with a goolf/1.5.14 toolchain in a Core/Compiler/MPI hierarchy (HierarchicalMNS):
-
starting point: top_paths = ['
', ' /Core'] mod_name = 'HPL/2.1' full_mod_subdir = ' /MPI/Compiler/GCC/4.8.2/OpenMPI/1.6.5' deps = ['GCC/4.8.2', 'OpenMPI/1.6.5', 'OpenBLAS/0.2.8-LAPACK-3.5.0', 'FFTW/3.3.4', 'ScaLAPACK/...'] -
1st iteration: find module that extends $MODULEPATH with '
/MPI/Compiler/GCC/4.8.2/OpenMPI/1.6.5', => OpenMPI/1.6.5 (in ' /Compiler/GCC/4.8.2' subdir); recurse with mod_name = 'OpenMPI/1.6.5' and full_mod_subdir = ' /Compiler/GCC/4.8.2' -
2nd iteration: find module that extends $MODULEPATH with '
/Compiler/GCC/4.8.2' => GCC/4.8.2 (in ' /Core' subdir); recurse with mod_name = 'GCC/4.8.2' and full_mod_subdir = ' /Core' -
3rd iteration: try to find module that extends $MODULEPATH with '
/Core' => ' /Core' is in top_paths, so stop recursion
| PARAMETER | DESCRIPTION |
|---|---|
top_paths |
list of potentation 'top of module tree' (absolute) paths
|
mod_name |
(short) module name for starting point (only used in log messages)
|
full_mod_subdir |
absolute path to module subdirectory for starting point
|
deps |
list of dependency modules for module at starting point
|
modpath_exts |
list of module path extensions for each of the dependency modules
DEFAULT:
|
prepend_module_path(path, set_mod_paths=True, priority=None)
¶
Prepend given module path to list of module paths, or bump it to 1st place.
| PARAMETER | DESCRIPTION |
|---|---|
path |
path to prepend to $MODULEPATH
|
set_mod_paths |
(re)set self.mod_paths
DEFAULT:
|
priority |
priority for this path in $MODULEPATH (Lmod-specific)
DEFAULT:
|
purge()
¶
Purge loaded modules.
read_module_file(mod_name)
¶
Read module file with specified name.
remove_module_path(path, set_mod_paths=True)
¶
Remove specified module path (using 'module unuse').
| PARAMETER | DESCRIPTION |
|---|---|
path |
path to remove from $MODULEPATH via 'unuse'
|
set_mod_paths |
(re)set self.mod_paths
DEFAULT:
|
run_module(*args, **kwargs)
¶
Run module command.
| PARAMETER | DESCRIPTION |
|---|---|
args |
list of arguments for module command; first argument should be the subcommand to run
DEFAULT:
|
kwargs |
dictionary with options that control certain aspects of how to run the module command
DEFAULT:
|
set_and_check_version()
¶
Get the module version, and check any requirements
set_mod_paths(mod_paths=None)
¶
Set mod_paths, based on $MODULEPATH unless a list of module paths is specified.
| PARAMETER | DESCRIPTION |
|---|---|
mod_paths |
list of entries for $MODULEPATH to use
DEFAULT:
|
set_path_env_var(key, paths)
¶
Set path environment variable to the given list of paths.
show(mod_name)
¶
Run 'module show' for the specified module.
unload(modules=None)
¶
Unload all requested modules.
unuse(path)
¶
Remove module path via 'module unuse'.
update()
¶
Update after new modules were added.
use(path, priority=None)
¶
Add path to $MODULEPATH via 'module use'.
| PARAMETER | DESCRIPTION |
|---|---|
path |
path to add to $MODULEPATH
|
priority |
priority for this path in $MODULEPATH (Lmod-specific)
DEFAULT:
|
NoModulesTool
¶
Bases: ModulesTool
Class that mock the module behaviour, used for operation not requiring modules. Eg. tests, fetch only
avail_modules_tools()
¶
Return all known modules tools.
curr_module_paths(normalize=False)
¶
Return a list of current module paths.
| PARAMETER | DESCRIPTION |
|---|---|
normalize |
Normalize the paths
DEFAULT:
|
get_software_libdir(name, only_one=True, fs=None)
¶
Find library subdirectories for the specified software package.
Returns the library subdirectory, relative to software root. It fails if multiple library subdirs are found, unless only_one is False which yields a list of all library subdirs.
| PARAMETER | DESCRIPTION |
|---|---|
name |
name of the software package
|
only_one |
indicates whether only one lib path is expected to be found
DEFAULT:
|
fs |
only retain library subdirs that contain one of the files in this list
DEFAULT:
|
get_software_root(name, with_env_var=False)
¶
Return the software root set for a particular software name.
get_software_root_env_var_name(name)
¶
Return name of environment variable for software root.
get_software_version(name)
¶
Return the software version set for a particular software name.
get_software_version_env_var_name(name)
¶
Return name of environment variable for software root.
invalidate_module_caches_for(path)
¶
Invalidate cache entries related to specified path.
mk_module_path(paths)
¶
Create a string representing the list of module paths.
modules_tool(mod_paths=None, testing=False)
¶
Return interface to modules tool (environment modules (C, Tcl), or Lmod)
reset_module_caches()
¶
Reset module caches.