easybuild.tools.parallelbuild module

Module for doing parallel builds. This uses a PBS-like cluster. You should be able to submit jobs (which can have dependencies)

Support for PBS is provided via the PbsJob class. If you want you could create other job classes and use them here.

author:Toon Willems (Ghent University)
author:Kenneth Hoste (Ghent University)
author:Stijn De Weirdt (Ghent University)
easybuild.tools.parallelbuild.build_easyconfigs_in_parallel(build_command, easyconfigs, output_dir='easybuild-build', prepare_first=True)

Build easyconfigs in parallel by submitting jobs to a batch-queuing system. Return list of jobs submitted.

Argument easyconfigs is a list of easyconfigs which can be built: e.g. they have no unresolved dependencies. This function will build them in parallel by submitting jobs.

Parameters:
  • build_command – build command to use
  • easyconfigs – list of easyconfig files
  • output_dir – output directory
  • prepare_first – prepare by runnning fetch step first for each easyconfig
easybuild.tools.parallelbuild.create_job(job_backend, build_command, easyconfig, output_dir='easybuild-build')

Creates a job to build a single easyconfig.

Parameters:
  • job_backend – A factory object for querying server parameters and creating actual job objects
  • build_command – format string for command, full path to an easyconfig file will be substituted in it
  • easyconfig – easyconfig as processed by process_easyconfig
  • output_dir – optional output path; –regtest-output-dir will be used inside the job with this variable

returns the job

easybuild.tools.parallelbuild.prepare_easyconfig(ec)

Prepare for building specified easyconfig (fetch sources) :param ec: parsed easyconfig (EasyConfig instance)

easybuild.tools.parallelbuild.submit_jobs(ordered_ecs, cmd_line_opts, testing=False, prepare_first=True)

Submit jobs. :param ordered_ecs: list of easyconfigs, in the order they should be processed :param cmd_line_opts: list of command line options (in ‘longopt=value’ form) :param testing: If True, skip actual job submission :param prepare_first: prepare by runnning fetch step first for each easyconfig