esm_runscripts package¶
Top-level package for ESM Runscripts.
Submodules¶
esm_runscripts.batch_system module¶
-
exception
esm_runscripts.batch_system.
UnknownBatchSystemError
[source]¶ Bases:
Exception
Raise this exception when an unknown batch system is encountered
esm_runscripts.cli module¶
A small wrapper that combines the shell interface and the Python interface
esm_runscripts.compute module¶
-
esm_runscripts.compute.
all_files_to_copy_append
(config, model, filetype, categ, file_source, file_interm, file_target)[source]¶
-
esm_runscripts.compute.
compile_model
(config)[source]¶ Compiles the desired model before the run starts
-
esm_runscripts.compute.
copy_tools_to_thisrun
(config)[source]¶ Copies the tools, namelists and runscripts to the experiment directory, making sure that they don’t overwrite previously existing files unless the
-U
flag is used. :param config: Dictionary containing the configuration information. :type config: dict
-
esm_runscripts.compute.
initialize_experiment_logfile
(config)[source]¶ Initializes the log file for the entire experiment.
Creates a file
${BASE_DIR}/${EXPID}/log/${EXPID}_${setup_name}.log
to keep track of start/stop times, job id numbers, and so on. Use the functionwrite_to_log
to put information in this file afterwards.The user can specify
experiment_log_file
under thegeneral
section of the configuration to override the default name. Timestamps for each message are given by the sectionexperiment_log_file_dateformat
, or defaults toTue Mar 17 09:36:38 2020
, i.e."%c"
. Please usestftime
compatable formats, as described here: https://strftime.org- Parameters
dict – The experiment configuration
- Returns
As per convention for the plug-in system; this gives back the entire config.
- Return type
dict
Attention
Calling this has some filesystem side effects. If the run number in the general configuration is set to 1, and a file exists for
general.exp_log_file
; this file is removed; and re-initialized.
-
esm_runscripts.compute.
update_runscript
(fromdir, scriptsdir, tfile, gconfig, file_type)[source]¶ Updates the script
tfile
in the directoryscriptdir
with the file in the directoryfromdir
if the update flag (-U
) is used during the call ofesm_runscripts
. If that flag is not used and the source and target are different then raises a user-friendly error recommending to use the-U
flag with the warning that the files will be overwritten. :param cls: Compute object. :type cls: obj :param fromdir: Path of the source. :type fromdir: str :param scriptsdir: Path of the target. :type scriptsdir: str :param tfile: Name of the script to be updated. :type tfile: str :param gconfig: Dictionary containing the general information about the compute task. :type gconfig: dict :param file_type: String specifying the nature of the file, only necessary for printing informationand for the error description.
- Parameters
Exceptions –
---------- –
UserError – If the target and source are different and the
-U
flag is not used when callingesm_runscripts
, returns an error.
esm_runscripts.coupler module¶
esm_runscripts.database module¶
esm_runscripts.database_actions module¶
esm_runscripts.filelists module¶
esm_runscripts.helpers module¶
-
esm_runscripts.helpers.
assemble_log_message
(config, message, message_sep=None, timestampStr_from_Unix=False)[source]¶ Assembles message for log file. See doc for write_to_log
-
esm_runscripts.helpers.
write_to_log
(config, message, message_sep=None)[source]¶ Puts a message into the experiment log file
- Parameters
message (list) – A list of the message elements; which is joined by either (highest to lowest): 1) the message_sep argument passed to the method, 2) The user’s chosen seperator, as written in
config["general"]["experiment_log_file_message_sep"]
, 3) An empty space" "
.message_sep (None) – The hard-coded message seperator to use; which ignores user choices.
Note
The user can control two things regarding the logfile format:
The datestamp formatting, whjich is taken from the config section
general.experiment_log_file_dateformat
.The message seperators; taken from
general.experiment_log_file_message_sep
. Note that if the programmer passes amessage_sep
argument; this one wins over the user choice.
esm_runscripts.inspect module¶
esm_runscripts.last_minute module¶
esm_runscripts.methods module¶
esm_runscripts.namelists module¶
esm-runscripts
Core Plugins for dealing with Fortran Namelists.
Provides plugins for loading, modifying, deleting, and writing Fortran
Namelists as part of the esm-runscripts
recipe. All plugins are found under
the class Namelist as static methods. A deprecated class namelist
(small “n”) is
provided, which warns you when it is used.
-
class
esm_runscripts.namelists.
Namelist
[source]¶ Bases:
object
Methods for dealing with FORTRAN namelists
-
static
apply_echam_disturbance
(config)[source]¶ Applies a disturbance to the DYNCTL chapter of the echam namelist via the enstdif
Relevant configuration entries: * disturbance_years (list of int): Which year to apply the disturbance * distrubance (float): Value to apply. Default can be found in echam.yaml
-
static
nmls_finalize
(mconfig, verbose)[source]¶ Writes namelists to disk after all modifications have finished.
User Information
Part of the main log output will be a section specifing the actual namelists that have been used for your simulation, including all relevant additions, removals, or changes.
Programmer Information
A copy of the f90nml object representations of the namelists is stored under the dictionary key “namelist_objs”, as a dictionary of (“namelist_name”, f90nml_objfect) key/value pairs.
Warning
Removing this step from your recipe might result in a broken run, as the namelists will not be present in their desired form! Even if your model runs, it might not contain all user-required changes.
- Parameters
mconfig (dict) – The model (e.g. ECHAM, FESOM, NEMO or OIFS) configuration
- Returns
mconfig – The modified configuration.
- Return type
dict
-
static
nmls_load
(mconfig)[source]¶ Loads Fortran namelists into the configuration dictionary.
User Information
To associate namelists with a specific model, you should have a section in your configuration that lists the namelists:
fesom: namelists: - "namelist.config" - "namelist.oce" - "namelist.ice" - "namelist.diag"
Programmer Information
The namelists are represented by f90nml Namelist objects, and are stored under:
mconfig["namelists"]["namelist.echam"]``
This would point to the ECHAM namelist as a f90nml object, which closely resembles a dictionary.
The actual namelists to load are listed in the raw configuration as a list of strings:
mconfig['namelists'] = ['nml1', 'nml2', 'nml3', ...]
Namelists are assumed to have been copied to
mconfig["thisrun_config_dir"]
, and are loaded from there.If the
mconfig
has a key"namelist_case"
equal to “uppercase”, the uppercase attribute of the f90nml representation of the namelist is set toTrue
.- Parameters
mconfig (dict) – The model (e.g. ECHAM, FESOM, NEMO or OIFS) configuration
- Returns
mconfig – The modified configuration.
- Return type
dict
-
static
nmls_modify
(mconfig)[source]¶ Performs namelist changes.
User Information
In the configuration file, you should have a section as:
echam: namelist_changes: namelist.echam: radctl: co2vmr: 1200e-6
This would change the value of the echam namelist (namelist.echam), subsection radctl, entry co2vmr to the value 1200e-6.
Programmer Information
IDEA(PG): Maybe we can provide examples of how these functions are used in the code?
Note
Actual changes are performed by the f90nml package patch fuction. See here: https://tinyurl.com/y4ydz363
- Parameters
mconfig (dict) – The model (e.g. ECHAM, FESOM, NEMO or OIFS) configuration
- Returns
mconfig – The modified configuration.
- Return type
dict
-
static
nmls_remove
(mconfig)[source]¶ Removes an element from a namelist chapter.
User Information
In the configuration file, assume you have:
echam: namelist_changes: namelist.echam: radctl: co2vmr: "remove_from_namelist"
In this case, the entry co2vmr would be deleted from the radctl section of namelist.echam.
Programmer Information
IDEA(PG): Maybe we can provide examples of how these functions are used in the code?
- Parameters
mconfig (dict) – The model (e.g. ECHAM, FESOM, NEMO or OIFS) configuration
- Returns
mconfig – The modified configuration.
- Return type
dict
-
static
-
class
esm_runscripts.namelists.
namelist
(*args, **kwargs)[source]¶ Bases:
esm_runscripts.namelists.Namelist
Legacy class name. Please use Namelist instead!
esm_runscripts.oasis module¶
-
class
esm_runscripts.oasis.
oasis
(nb_of_couplings=1, coupled_execs=['echam', 'fesom'], runtime=1, debug_level=1, nnorest='F', mct_version='4.0', lucia=False)[source]¶ Bases:
object
esm_runscripts.prepare module¶
-
esm_runscripts.prepare.
model_env_into_computer
(config)[source]¶ This function allows to store in the
computer
dictionary, variables that were defined insideenvironment_changes
orcompile/runtime_environment_changes
in the components.It excludes
module_actions
andexport_vars
dictionaries as those are resolved later.This function is necessary for controlling
choose_
blocks in the computer file from the component configuration file (i.e. adduseMPI
case to the component to control whichuseMPI
case is selected in the computer file).This function works both for compilation time and run time, and the result is that all components work under the same environment. The only exception is for the compilation of components, where
add_export_vars
andadd_module_actions
are excluded from the merging intocomputer
, and are included individually in respective compilation scripts.Later on, it might be desirable to always split the environments both for compiling (done by Paul Gierz, but this function would need to be adapted) and running (not done yet).
If this script gives you problems contact Miguel Andres-Martinez (miguel.andres-martinez@awi.de).
- Parameters
config (dict) – Dictionary containing the simulation/compilation information
- Raises
User Note/Error – If the same variable is found in two or more different component environments. Asks the user how to proceed.
esm_runscripts.sim_objects module¶
Documentation goes here
-
class
esm_runscripts.sim_objects.
SimulationSetup
(command_line_config=None, user_config=None)[source]¶ Bases:
object
-
compute
(kill_after_submit=True)[source]¶ All steps needed for a model computation.
- Parameters
kill_after_submit (bool) – Default
True
. If set, the entire Python instance is killed with asys.exit()
as the very last after job submission.
-
tidy
()[source]¶ Performs steps for tidying up a simulation after a job has finished and submission of following jobs.
This method uses two lists,
all_files_to_copy
andall_listed_filetypes
to sort finished data from the current run folder back to the main experiment folder and submit new compute and post-process jobs. Files forlog
,mon
,outdata
, andrestart_out
are gathered. The program waits until the job completes or an error is found (See ~self.wait_and_observe). Then, if necessary, the coupler cleans up it’s files (unless it’s a standalone run), and the files in the lists are copied from the work folder to the current run folder. A check for unknown files is performed (see ~self.check_for_unknown_files), files are moved from the the current run folder to the main experiment folder, and new compute and post process jobs are started.Warning
The date is changed during this routine! Be careful where you put any calls that may depend on date information!
Note
This method is also responsible for calling the next compute job as well as the post processing job!
-
esm_runscripts.slurm module¶
Contains functions for dealing with SLURM-based batch systems
-
class
esm_runscripts.slurm.
Slurm
(config)[source]¶ Bases:
object
Deals with SLURM, allowing you to check if a job is submitted, get the current job ID, generate a srun hostfile, get the current job state, and check if a job is still running.
-
filename
¶ The filename for srun commands, defaults to
hostfile_srun
- Type
str
-
path
¶ Full path to this file, defaults to
thisrun_scripts_dir / filename
- Type
str
- Parameters
config (dict) – The run configuration, needed to determine where the script directory for this particular run is.
-
static
check_if_submitted
()[source]¶ Determines if a job is submitted in the currently running shell by checking for
SLURM_JOB_ID
in the environment- Returns
- Return type
bool
-
esm_runscripts.tidy module¶
-
class
esm_runscripts.tidy.
RunFolders
(config)[source]¶ Bases:
list
Logs the
run_
directories in<experiment_id>/log/run_folders.log
, updating it with new folders. The resulting object is a list ofrun_
paths that exist or existed during the run time (even if they got deleted). This is useful for indexing operations such as<object_name>[::<interval>]
used when removingrun_
folders. .. rubric:: NotesIt keeps the folder names sorted so there is no need of sorting out of the object, and it also prevents the existence of duplicates.
-
esm_runscripts.tidy.
clean_run_dir
(config)[source]¶ This plugin allows you to clean up the
run_${DATE}
folders. To do that you can use the following variables under thegeneral
section of your runscript (documentation follows order of code as it is executed):clean_runs
: This is the most important variable for most users. It can take the following values:True
: removes therun_
directory after each run (overrides every otherclean_
option).False
: does not remove anyrun_
directory (default) if noclean_
variable is defined.<int>
: giving an integer as a value results in deleting therun_
folders except for the last <int> runs (recommended option as it allows for debugging of crashed simulations).
Note
clean_runs: (bool)
is incompatible withclean_this_rundir
andclean_runs: (int)
is incompatible withclean_old_rundirs_except
(an error will be raised after the end of the first simulation). The functionality ofclean_runs
variable alone will suffice most of the standard user requirements. If finer tunning for the removal ofrun_
directories is required you can used the following variables instead ofclean_runs
.clean_this_rundir
: (bool) Removes the entire run directory (equivalent toclean_runs: (bool)
).clean_this_rundir: True
overrides every otherclean_
option.clean_old_rundirs_except
: (int) Removes the entire run directory except for the last <x> runs (equivalent toclean_runs: (int)
).clean_old_rundirs_keep_every
: (int) Removes the entire run directory except every <x>th run. Compatible withclean_old_rundirs_except
orclean_runs: (int)
.clean_<filetype>_dir
: (bool) Erases the run directory for a specific filetype. Compatible with all the other options.clean_size
: (int or float) Erases all files with size greater thanclean_size
, must be specified in bytes! Compatible with all the other options.
Example
To delete all the
run_
directories in your experiment include this into your runscript:general: clean_runs: True
To keep the last 2
run_
directories:general: clean_runs: 2
To keep the last 2 runs and every 5 runs:
general: clean_old_rundirs_except: 2 clean_old_rundirs_keep_every: 5