esm_runscripts package¶
Top-level package for ESM Runscripts.
Submodules¶
esm_runscripts.batch_system module¶
-
exception
esm_runscripts.batch_system.UnknownBatchSystemError[source]¶ Bases:
ExceptionRaise this exception when an unknown batch system is encountered
esm_runscripts.cli module¶
A small wrapper that combines the shell interface and the Python interface
esm_runscripts.compute module¶
-
esm_runscripts.compute.all_files_to_copy_append(config, model, filetype, categ, file_source, file_interm, file_target)[source]¶
-
esm_runscripts.compute.compile_model(config)[source]¶ Compiles the desired model before the run starts
-
esm_runscripts.compute.copy_tools_to_thisrun(config)[source]¶ Copies the tools, namelists and runscripts to the experiment directory, making sure that they don’t overwrite previously existing files unless the
-Uflag is used. :param config: Dictionary containing the configuration information. :type config: dict
-
esm_runscripts.compute.initialize_experiment_logfile(config)[source]¶ Initializes the log file for the entire experiment.
Creates a file
${BASE_DIR}/${EXPID}/log/${EXPID}_${setup_name}.logto keep track of start/stop times, job id numbers, and so on. Use the functionwrite_to_logto put information in this file afterwards.The user can specify
experiment_log_fileunder thegeneralsection of the configuration to override the default name. Timestamps for each message are given by the sectionexperiment_log_file_dateformat, or defaults toTue Mar 17 09:36:38 2020, i.e."%c". Please usestftimecompatable formats, as described here: https://strftime.org- Parameters
dict – The experiment configuration
- Returns
As per convention for the plug-in system; this gives back the entire config.
- Return type
dict
Attention
Calling this has some filesystem side effects. If the run number in the general configuration is set to 1, and a file exists for
general.exp_log_file; this file is removed; and re-initialized.
-
esm_runscripts.compute.update_runscript(fromdir, scriptsdir, tfile, gconfig, file_type)[source]¶ Updates the script
tfilein the directoryscriptdirwith the file in the directoryfromdirif the update flag (-U) is used during the call ofesm_runscripts. If that flag is not used and the source and target are different then raises a user-friendly error recommending to use the-Uflag with the warning that the files will be overwritten. :param cls: Compute object. :type cls: obj :param fromdir: Path of the source. :type fromdir: str :param scriptsdir: Path of the target. :type scriptsdir: str :param tfile: Name of the script to be updated. :type tfile: str :param gconfig: Dictionary containing the general information about the compute task. :type gconfig: dict :param file_type: String specifying the nature of the file, only necessary for printing informationand for the error description.
- Parameters
Exceptions –
---------- –
UserError – If the target and source are different and the
-Uflag is not used when callingesm_runscripts, returns an error.
esm_runscripts.coupler module¶
esm_runscripts.database module¶
esm_runscripts.database_actions module¶
esm_runscripts.filelists module¶
esm_runscripts.helpers module¶
-
class
esm_runscripts.helpers.SmartSink[source]¶ Bases:
objectA class for smart sinks that allow for logging (using
loggerfrom loguru), even if the file path of the log file is not yet defined. The actual sink is not the instanced object itself, but the methodsinkof the instance. The log record is saved inself.log_recordand the log file is written using the path specified inself.path. If the path is not specified, the log is stored only in theself.log_record. When the path is finally specified,self.log_recordis dumped into the log file and from that moment, any timeloggerlogs something it will also be written into the file. To specify the path the methoddef_pathneeds to be used.-
def_path(path)[source]¶ Method to define the path of the file. Once the path is defined, the log record is written into the file.
- Parameters
path (str) – Path of the logging file.
-
sink(message)[source]¶ The actual sink for loguru’s
logger. Once you define a logger level a sink needs to be provided. Standard sinks include file paths, methods, etc. Providing this method as a sink (logger.add(<name_of_the_instance>.sink, level="<your_level>", ...)) enables the functionality of the SmartSink object.- Parameters
message (str) – String containing the logging message.
-
-
esm_runscripts.helpers.assemble_log_message(config, message, message_sep=None, timestampStr_from_Unix=False)[source]¶ Assembles message for log file. See doc for write_to_log
-
esm_runscripts.helpers.write_to_log(config, message, message_sep=None)[source]¶ Puts a message into the experiment log file
- Parameters
message (list) – A list of the message elements; which is joined by either (highest to lowest): 1) the message_sep argument passed to the method, 2) The user’s chosen seperator, as written in
config["general"]["experiment_log_file_message_sep"], 3) An empty space" ".message_sep (None) – The hard-coded message seperator to use; which ignores user choices.
Note
The user can control two things regarding the logfile format:
The datestamp formatting, whjich is taken from the config section
general.experiment_log_file_dateformat.The message seperators; taken from
general.experiment_log_file_message_sep. Note that if the programmer passes amessage_separgument; this one wins over the user choice.
esm_runscripts.inspect module¶
esm_runscripts.last_minute module¶
esm_runscripts.methods module¶
esm_runscripts.namelists module¶
esm-runscripts Core Plugins for dealing with Fortran Namelists.
Provides plugins for loading, modifying, deleting, and writing Fortran
Namelists as part of the esm-runscripts recipe. All plugins are found under
the class Namelist as static methods. A deprecated class namelist (small “n”) is
provided, which warns you when it is used.
-
class
esm_runscripts.namelists.Namelist[source]¶ Bases:
objectMethods for dealing with FORTRAN namelists
-
static
apply_echam_disturbance(config)[source]¶ Applies a disturbance to the DYNCTL chapter of the echam namelist via the enstdif
Relevant configuration entries: * disturbance_years (list of int): Which year to apply the disturbance * distrubance (float): Value to apply. Default can be found in echam.yaml
-
static
nmls_finalize(mconfig, verbose)[source]¶ Writes namelists to disk after all modifications have finished.
User Information
Part of the main log output will be a section specifing the actual namelists that have been used for your simulation, including all relevant additions, removals, or changes.
Programmer Information
A copy of the f90nml object representations of the namelists is stored under the dictionary key “namelist_objs”, as a dictionary of (“namelist_name”, f90nml_objfect) key/value pairs.
Warning
Removing this step from your recipe might result in a broken run, as the namelists will not be present in their desired form! Even if your model runs, it might not contain all user-required changes.
- Parameters
mconfig (dict) – The model (e.g. ECHAM, FESOM, NEMO or OIFS) configuration
- Returns
mconfig – The modified configuration.
- Return type
dict
-
static
nmls_load(mconfig)[source]¶ Loads Fortran namelists into the configuration dictionary.
User Information
To associate namelists with a specific model, you should have a section in your configuration that lists the namelists:
fesom: namelists: - "namelist.config" - "namelist.oce" - "namelist.ice" - "namelist.diag"
Programmer Information
The namelists are represented by f90nml Namelist objects, and are stored under:
mconfig["namelists"]["namelist.echam"]``
This would point to the ECHAM namelist as a f90nml object, which closely resembles a dictionary.
The actual namelists to load are listed in the raw configuration as a list of strings:
mconfig['namelists'] = ['nml1', 'nml2', 'nml3', ...]
Namelists are assumed to have been copied to
mconfig["thisrun_config_dir"], and are loaded from there.If the
mconfighas a key"namelist_case"equal to “uppercase”, the uppercase attribute of the f90nml representation of the namelist is set toTrue.- Parameters
mconfig (dict) – The model (e.g. ECHAM, FESOM, NEMO or OIFS) configuration
- Returns
mconfig – The modified configuration.
- Return type
dict
-
static
nmls_modify(mconfig)[source]¶ Performs namelist changes.
User Information
In the configuration file, you should have a section as:
echam: namelist_changes: namelist.echam: radctl: co2vmr: 1200e-6
This would change the value of the echam namelist (namelist.echam), subsection radctl, entry co2vmr to the value 1200e-6.
Programmer Information
IDEA(PG): Maybe we can provide examples of how these functions are used in the code?
Note
Actual changes are performed by the f90nml package patch fuction. See here: https://tinyurl.com/y4ydz363
- Parameters
mconfig (dict) – The model (e.g. ECHAM, FESOM, NEMO or OIFS) configuration
- Returns
mconfig – The modified configuration.
- Return type
dict
-
static
nmls_remove(mconfig)[source]¶ Removes an element from a namelist chapter.
User Information
In the configuration file, assume you have:
echam: namelist_changes: namelist.echam: radctl: co2vmr: "remove_from_namelist"
In this case, the entry co2vmr would be deleted from the radctl section of namelist.echam.
Programmer Information
IDEA(PG): Maybe we can provide examples of how these functions are used in the code?
- Parameters
mconfig (dict) – The model (e.g. ECHAM, FESOM, NEMO or OIFS) configuration
- Returns
mconfig – The modified configuration.
- Return type
dict
-
static
-
class
esm_runscripts.namelists.namelist(*args, **kwargs)[source]¶ Bases:
esm_runscripts.namelists.NamelistLegacy class name. Please use Namelist instead!
esm_runscripts.oasis module¶
-
class
esm_runscripts.oasis.oasis(nb_of_couplings=1, coupled_execs=['echam', 'fesom'], runtime=1, debug_level=1, nnorest='F', mct_version='4.0', lucia=False)[source]¶ Bases:
object
esm_runscripts.prepare module¶
-
esm_runscripts.prepare.model_env_into_computer(config)[source]¶ This function allows to store in the
computerdictionary, variables that were defined insideenvironment_changesorcompile/runtime_environment_changesin the components.It excludes
module_actionsandexport_varsdictionaries as those are resolved later.This function is necessary for controlling
choose_blocks in the computer file from the component configuration file (i.e. adduseMPIcase to the component to control whichuseMPIcase is selected in the computer file).This function works both for compilation time and run time, and the result is that all components work under the same environment. The only exception is for the compilation of components, where
add_export_varsandadd_module_actionsare excluded from the merging intocomputer, and are included individually in respective compilation scripts.Later on, it might be desirable to always split the environments both for compiling (done by Paul Gierz, but this function would need to be adapted) and running (not done yet).
If this script gives you problems contact Miguel Andres-Martinez (miguel.andres-martinez@awi.de).
- Parameters
config (dict) – Dictionary containing the simulation/compilation information
- Raises
User Note/Error – If the same variable is found in two or more different component environments. Asks the user how to proceed.
esm_runscripts.sim_objects module¶
Documentation goes here
-
class
esm_runscripts.sim_objects.SimulationSetup(command_line_config=None, user_config=None)[source]¶ Bases:
object-
compute(kill_after_submit=True)[source]¶ All steps needed for a model computation.
- Parameters
kill_after_submit (bool) – Default
True. If set, the entire Python instance is killed with asys.exit()as the very last after job submission.
-
tidy()[source]¶ Performs steps for tidying up a simulation after a job has finished and submission of following jobs.
This method uses two lists,
all_files_to_copyandall_listed_filetypesto sort finished data from the current run folder back to the main experiment folder and submit new compute and post-process jobs. Files forlog,mon,outdata, andrestart_outare gathered. The program waits until the job completes or an error is found (See ~self.wait_and_observe). Then, if necessary, the coupler cleans up it’s files (unless it’s a standalone run), and the files in the lists are copied from the work folder to the current run folder. A check for unknown files is performed (see ~self.check_for_unknown_files), files are moved from the the current run folder to the main experiment folder, and new compute and post process jobs are started.Warning
The date is changed during this routine! Be careful where you put any calls that may depend on date information!
Note
This method is also responsible for calling the next compute job as well as the post processing job!
-
esm_runscripts.slurm module¶
Contains functions for dealing with SLURM-based batch systems
-
class
esm_runscripts.slurm.Slurm(config)[source]¶ Bases:
objectDeals with SLURM, allowing you to check if a job is submitted, get the current job ID, generate a srun hostfile, get the current job state, and check if a job is still running.
-
filename¶ The filename for srun commands, defaults to
hostfile_srun- Type
str
-
path¶ Full path to this file, defaults to
thisrun_scripts_dir / filename- Type
str
- Parameters
config (dict) – The run configuration, needed to determine where the script directory for this particular run is.
-
static
check_if_submitted()[source]¶ Determines if a job is submitted in the currently running shell by checking for
SLURM_JOB_IDin the environment- Returns
- Return type
bool
-
esm_runscripts.tidy module¶
-
class
esm_runscripts.tidy.RunFolders(config)[source]¶ Bases:
listLogs the
run_directories in<experiment_id>/log/run_folders.log, updating it with new folders. The resulting object is a list ofrun_paths that exist or existed during the run time (even if they got deleted). This is useful for indexing operations such as<object_name>[::<interval>]used when removingrun_folders. .. rubric:: NotesIt keeps the folder names sorted so there is no need of sorting out of the object, and it also prevents the existence of duplicates.
-
esm_runscripts.tidy.clean_run_dir(config)[source]¶ This plugin allows you to clean up the
run_${DATE}folders. To do that you can use the following variables under thegeneralsection of your runscript (documentation follows order of code as it is executed):clean_runs: This is the most important variable for most users. It can take the following values:True: removes therun_directory after each run (overrides every otherclean_option).False: does not remove anyrun_directory (default) if noclean_variable is defined.<int>: giving an integer as a value results in deleting therun_folders except for the last <int> runs (recommended option as it allows for debugging of crashed simulations).
Note
clean_runs: (bool)is incompatible withclean_this_rundirandclean_runs: (int)is incompatible withclean_old_rundirs_except(an error will be raised after the end of the first simulation). The functionality ofclean_runsvariable alone will suffice most of the standard user requirements. If finer tunning for the removal ofrun_directories is required you can used the following variables instead ofclean_runs.clean_this_rundir: (bool) Removes the entire run directory (equivalent toclean_runs: (bool)).clean_this_rundir: Trueoverrides every otherclean_option.clean_old_rundirs_except: (int) Removes the entire run directory except for the last <x> runs (equivalent toclean_runs: (int)).clean_old_rundirs_keep_every: (int) Removes the entire run directory except every <x>th run. Compatible withclean_old_rundirs_exceptorclean_runs: (int).clean_<filetype>_dir: (bool) Erases the run directory for a specific filetype. Compatible with all the other options.clean_size: (int or float) Erases all files with size greater thanclean_size, must be specified in bytes! Compatible with all the other options.
Example
To delete all the
run_directories in your experiment include this into your runscript:general: clean_runs: True
To keep the last 2
run_directories:general: clean_runs: 2
To keep the last 2 runs and every 5 runs:
general: clean_old_rundirs_except: 2 clean_old_rundirs_keep_every: 5