ESM Environment

The package esm_environment takes care of generating the environments for the different HPCs supported by ESM-Tools. This is done through the use of the EnvironmentInfos class inside the different ESM-Tools packages.

For the correct definition of an environment for an HPC a yaml file for that system needs to be included inside the esm_tools package inside the configs/machines/ folder (e.g. ollie.yaml). This file should contain all the required preset variables for that system and the environment variables module_actions and export_vars.

Environment variables

module_actions (list)

A list of module actions to be included in the compilation and run scripts generated by esm_master and esm_runscripts respectively, such as module load netcdf, module unload netcdf, module purge, etc. The syntax of this list is such as that of the command that would be normally used in shell, but omitting the module word, for example:

module_actions:
    - "purge"
    - "load netcdf"

This variable also allows for sourcing files by adding a member to the list such as source <file_to_be_sourced>.

export_vars (dict)

A dictionary containing all the variables (and their values) to be exported. The syntax is as follows:

export_vars:
    A_VAR_TO_BE_EXPORTED: the_value

The previous example will result in the following export in the script produced by esm_master or esm_runscripts:

export A_VAR_TO_BE_EXPORTED=the_value

As a dictionary, export_vars is not allowed to have repeated keys. This could be a problem when environments are required to redefine a variable at different points of the script. To overcome this limitation, repetitions of the same variable are allowed if the key is followed by an integer contained inside [(int)]:

export_vars:
    A_VAR_TO_BE_EXPORTED: the_value
    A_VAR_TO_BE_EXPORTED[(1)]: $A_VAR_TO_BE_EXPORTED:another_value

The resulting script will contain the following exports:

export A_VAR_TO_BE_EXPORTED=the_value
export A_VAR_TO_BE_EXPORTED=$A_VAR_TO_BE_EXPORTED:another_value

Note that the index is removed once the exports are transferred into the script.

Modification of the environment through the model/setup files

As previously mentioned, the default environment for a HPC system is defined inside its machine file (in esm_tools/machines/<machine_name>.yaml). However, it is possible to modify this environment through the model and/or coupled setup files (or even inside the runscript) to adjust to the model/setup requirements. For this purpose, the variables environment_changes, compiletime_environment_changes and runtime_environment_changes can be used.

environment_changes (dict)

Allows for modifications of the machine module_actions and export_vars, both during compilation and runtime.

compiletime_environment_changes (dict)

Allows for modifications of the machine module_actions and export_vars, only applied during compilation time.

compiletime_environment_changes (dict)

Allows for modifications of the machine module_actions and export_vars, only applied during run time.

The syntax for this dictionary is the same as that defined in Environment variables, but using add_ in front of the environment variables (add_module_actions and add_export_vars). Furthermore, the environment variables can be nested inside choose_ blocks:

environment_changes:
        choose_computer:
                ollie:
                        add_export_vars:
                                COMPUTER_VAR: 'ollie'
                juwels:
                        add_export_vars:
                                COMPUTER_VAR: 'mistral'

Note

This changes are model-specific for compilation, meaning that the changes will only occur for the compilation script of the model containing those changes. For runtime, all the environments of the components will be added together into the same .run script. Please, refer to Coupled setup environment control for an explanation on how to control environments for a whole setup.

Coupled setup environment control

There are two ways in which the environments for the coupled setups can be modified: defining environment_changes for each component or defining a general environment_changes for the whole setup:

Component-by-component

The environment_changes are taken from the standalone component files. It is possible to modify these environment_changes through the setup file by including environment_changes inside the chapter of that component.

Warning

Handling environment_changes in this fashion implies that compilation scripts can potentially end up containing different environments.

General environment for setups

To define a general environment_changes for all the components of a setup, include the environment_changes inside the general section of the setup file. This will ignore all the environment_changes defined by the standalone files. It is still possible to add component-specific environment_changes from the component chapter inside the setup file.