esm_runscripts package

Top-level package for ESM Runscripts.

Submodules

esm_runscripts.batch_system module

exception esm_runscripts.batch_system.UnknownBatchSystemError[source]

Bases: Exception

Raise this exception when an unknown batch system is encountered

class esm_runscripts.batch_system.batch_system(config, name)[source]

Bases: object

calc_requirements(config)[source]
static calculate_requirements(config)[source]
check_if_submitted()[source]
static determine_nodelist(config)[source]
static get_batch_header(config)[source]
static get_environment(config)[source]
static get_extra(config)[source]
get_job_state(jobid)[source]
get_jobid()[source]
static get_run_commands(config)[source]
static get_sad_filename(config)[source]
static get_submit_command(config, sadfilename)[source]
job_is_still_running(jobid)[source]
static submit(config)[source]
static write_simple_runscript(config)[source]
esm_runscripts.batch_system.get_run_commands_multisrun(config, commands)[source]

esm_runscripts.cli module

A small wrapper that combines the shell interface and the Python interface

esm_runscripts.cli.main()[source]
esm_runscripts.cli.parse_shargs()[source]

The arg parser for interactive use

esm_runscripts.compute module

esm_runscripts.compute.add_batch_hostfile(config)[source]
esm_runscripts.compute.all_files_to_copy_append(config, model, filetype, categ, file_source, file_interm, file_target)[source]
esm_runscripts.compute.color_diff(diff)[source]
esm_runscripts.compute.compile_model(config)[source]

Compiles the desired model before the run starts

esm_runscripts.compute.copy_files_to_thisrun(config)[source]
esm_runscripts.compute.copy_files_to_work(config)[source]
esm_runscripts.compute.copy_tools_to_thisrun(config)[source]

Copies the tools, namelists and runscripts to the experiment directory, making sure that they don’t overwrite previously existing files unless the -U flag is used. :param config: Dictionary containing the configuration information. :type config: dict

esm_runscripts.compute.create_new_files(config)[source]
esm_runscripts.compute.initialize_experiment_logfile(config)[source]

Initializes the log file for the entire experiment.

Creates a file ${BASE_DIR}/${EXPID}/log/${EXPID}_${setup_name}.log to keep track of start/stop times, job id numbers, and so on. Use the function write_to_log to put information in this file afterwards.

The user can specify experiment_log_file under the general section of the configuration to override the default name. Timestamps for each message are given by the section experiment_log_file_dateformat, or defaults to Tue Mar 17 09:36:38 2020, i.e. "%c". Please use stftime compatable formats, as described here: https://strftime.org

Parameters

dict – The experiment configuration

Returns

As per convention for the plug-in system; this gives back the entire config.

Return type

dict

Attention

Calling this has some filesystem side effects. If the run number in the general configuration is set to 1, and a file exists for general.exp_log_file; this file is removed; and re-initialized.

esm_runscripts.compute.modify_files(config)[source]
esm_runscripts.compute.modify_namelists(config)[source]
esm_runscripts.compute.prepare_coupler_files(config)[source]
esm_runscripts.compute.run_job(config)[source]
esm_runscripts.compute.update_runscript(fromdir, scriptsdir, tfile, gconfig, file_type)[source]

Updates the script tfile in the directory scriptdir with the file in the directory fromdir if the update flag (-U) is used during the call of esm_runscripts. If that flag is not used and the source and target are different then raises a user-friendly error recommending to use the -U flag with the warning that the files will be overwritten. :param cls: Compute object. :type cls: obj :param fromdir: Path of the source. :type fromdir: str :param scriptsdir: Path of the target. :type scriptsdir: str :param tfile: Name of the script to be updated. :type tfile: str :param gconfig: Dictionary containing the general information about the compute task. :type gconfig: dict :param file_type: String specifying the nature of the file, only necessary for printing information

and for the error description.

Parameters
  • Exceptions

  • ----------

  • UserError – If the target and source are different and the -U flag is not used when calling esm_runscripts, returns an error.

esm_runscripts.coupler module

class esm_runscripts.coupler.coupler_class(full_config, name)[source]

Bases: object

add_couplings(full_config)[source]
add_files(full_config)[source]
finalize(destination_dir)[source]
prepare(full_config, destination_dir)[source]
prepare_restarts(full_config)[source]
print_config_files()[source]
tidy(full_config)[source]

esm_runscripts.database module

class esm_runscripts.database.experiment(**kwargs)[source]

Bases: sqlalchemy.ext.declarative.api.Base

archive_folder
cpuh
exp_folder
expid
gb
id
static nicer_output(run)[source]
outcome
run_timestamp
runtime
setup_name
timestamp
static topline()[source]

esm_runscripts.database_actions module

esm_runscripts.database_actions.database_basic_entry(config)[source]
esm_runscripts.database_actions.database_entry(config)[source]
esm_runscripts.database_actions.database_entry_check(config)[source]
esm_runscripts.database_actions.database_entry_crashed(config)[source]
esm_runscripts.database_actions.database_entry_start(config)[source]
esm_runscripts.database_actions.database_entry_success(config)[source]

esm_runscripts.filelists module

esm_runscripts.filelists.assemble(config)[source]
esm_runscripts.filelists.assemble_intermediate_files_and_finalize_targets(config)[source]
esm_runscripts.filelists.check_for_unknown_files(config)[source]
esm_runscripts.filelists.choose_needed_files(config)[source]
esm_runscripts.filelists.complete_all_file_movements(config)[source]
esm_runscripts.filelists.complete_one_file_movement(config, model, filetype, movement, movetype)[source]
esm_runscripts.filelists.complete_restart_in(config)[source]
esm_runscripts.filelists.complete_sources(config)[source]
esm_runscripts.filelists.complete_targets(config)[source]
esm_runscripts.filelists.copy_files(config, filetypes, source, target)[source]
esm_runscripts.filelists.create_missing_file_movement_entries(config)[source]
esm_runscripts.filelists.get_method(movement)[source]
esm_runscripts.filelists.get_movement(config, model, filetype, source, target)[source]
esm_runscripts.filelists.globbing(config)[source]
esm_runscripts.filelists.log_used_files(config)[source]
esm_runscripts.filelists.rename_sources_to_targets(config)[source]
esm_runscripts.filelists.replace_year_placeholder(config)[source]
esm_runscripts.filelists.report_missing_files(config)[source]
esm_runscripts.filelists.reuse_sources(config)[source]
esm_runscripts.filelists.target_subfolders(config)[source]

esm_runscripts.helpers module

class esm_runscripts.helpers.SmartSink[source]

Bases: object

A class for smart sinks that allow for logging (using logger from loguru), even if the file path of the log file is not yet defined. The actual sink is not the instanced object itself, but the method sink of the instance. The log record is saved in self.log_record and the log file is written using the path specified in self.path. If the path is not specified, the log is stored only in the self.log_record. When the path is finally specified, self.log_record is dumped into the log file and from that moment, any time logger logs something it will also be written into the file. To specify the path the method def_path needs to be used.

def_path(path)[source]

Method to define the path of the file. Once the path is defined, the log record is written into the file.

Parameters

path (str) – Path of the logging file.

sink(message)[source]

The actual sink for loguru’s logger. Once you define a logger level a sink needs to be provided. Standard sinks include file paths, methods, etc. Providing this method as a sink (logger.add(<name_of_the_instance>.sink, level="<your_level>", ...)) enables the functionality of the SmartSink object.

Parameters

message (str) – String containing the logging message.

write_log(message, wmode)[source]

Method to write the logs into the disk.

Parameters
  • message (str, list) – String containing the logging message or list containing more than one logging message, to be written in the file.

  • wmode (str) – Writing mode to choose among "w" or "a".

esm_runscripts.helpers.assemble_log_message(config, message, message_sep=None, timestampStr_from_Unix=False)[source]

Assembles message for log file. See doc for write_to_log

esm_runscripts.helpers.end_it_all(config)[source]
esm_runscripts.helpers.evaluate(config, job_type, recipe_name)[source]
esm_runscripts.helpers.vprint(message, config)[source]
esm_runscripts.helpers.write_to_log(config, message, message_sep=None)[source]

Puts a message into the experiment log file

Parameters
  • message (list) – A list of the message elements; which is joined by either (highest to lowest): 1) the message_sep argument passed to the method, 2) The user’s chosen seperator, as written in config["general"]["experiment_log_file_message_sep"], 3) An empty space " ".

  • message_sep (None) – The hard-coded message seperator to use; which ignores user choices.

Note

The user can control two things regarding the logfile format:

  1. The datestamp formatting, whjich is taken from the config section general.experiment_log_file_dateformat.

  2. The message seperators; taken from general.experiment_log_file_message_sep. Note that if the programmer passes a message_sep argument; this one wins over the user choice.

esm_runscripts.inspect module

esm_runscripts.inspect.cat_file(full_filepath)[source]
esm_runscripts.inspect.dir_size(somepath)[source]
esm_runscripts.inspect.inspect_config(config)[source]
esm_runscripts.inspect.inspect_file(config)[source]
esm_runscripts.inspect.inspect_folder(config)[source]
esm_runscripts.inspect.inspect_namelists(config)[source]
esm_runscripts.inspect.inspect_overview(config)[source]
esm_runscripts.inspect.inspect_size(config)[source]
esm_runscripts.inspect.run_job(config)[source]

esm_runscripts.last_minute module

esm_runscripts.last_minute.apply_last_minute_changes(config)[source]
class esm_runscripts.last_minute.last_minute_changes(config)[source]

Bases: object

esm_runscripts.last_minute.restore_protected_last_minute_changes(config)[source]

esm_runscripts.methods module

esm_runscripts.methods.set_global_attr(fname, attribute, value)[source]

esm_runscripts.namelists module

esm-runscripts Core Plugins for dealing with Fortran Namelists.

Provides plugins for loading, modifying, deleting, and writing Fortran Namelists as part of the esm-runscripts recipe. All plugins are found under the class Namelist as static methods. A deprecated class namelist (small “n”) is provided, which warns you when it is used.

class esm_runscripts.namelists.Namelist[source]

Bases: object

Methods for dealing with FORTRAN namelists

static apply_echam_disturbance(config)[source]

Applies a disturbance to the DYNCTL chapter of the echam namelist via the enstdif

Relevant configuration entries: * disturbance_years (list of int): Which year to apply the disturbance * distrubance (float): Value to apply. Default can be found in echam.yaml

static nmls_finalize(mconfig, verbose)[source]

Writes namelists to disk after all modifications have finished.

User Information

Part of the main log output will be a section specifing the actual namelists that have been used for your simulation, including all relevant additions, removals, or changes.

Programmer Information

A copy of the f90nml object representations of the namelists is stored under the dictionary key “namelist_objs”, as a dictionary of (“namelist_name”, f90nml_objfect) key/value pairs.

Warning

Removing this step from your recipe might result in a broken run, as the namelists will not be present in their desired form! Even if your model runs, it might not contain all user-required changes.

Parameters

mconfig (dict) – The model (e.g. ECHAM, FESOM, NEMO or OIFS) configuration

Returns

mconfig – The modified configuration.

Return type

dict

static nmls_load(mconfig)[source]

Loads Fortran namelists into the configuration dictionary.

User Information

To associate namelists with a specific model, you should have a section in your configuration that lists the namelists:

fesom:
    namelists:
        - "namelist.config"
        - "namelist.oce"
        - "namelist.ice"
        - "namelist.diag"

Programmer Information

The namelists are represented by f90nml Namelist objects, and are stored under:

mconfig["namelists"]["namelist.echam"]``

This would point to the ECHAM namelist as a f90nml object, which closely resembles a dictionary.

The actual namelists to load are listed in the raw configuration as a list of strings:

mconfig['namelists'] = ['nml1', 'nml2', 'nml3', ...]

Namelists are assumed to have been copied to mconfig["thisrun_config_dir"], and are loaded from there.

If the mconfig has a key "namelist_case" equal to “uppercase”, the uppercase attribute of the f90nml representation of the namelist is set to True.

Parameters

mconfig (dict) – The model (e.g. ECHAM, FESOM, NEMO or OIFS) configuration

Returns

mconfig – The modified configuration.

Return type

dict

static nmls_modify(mconfig)[source]

Performs namelist changes.

User Information

In the configuration file, you should have a section as:

echam:
    namelist_changes:
        namelist.echam:
            radctl:
                co2vmr: 1200e-6

This would change the value of the echam namelist (namelist.echam), subsection radctl, entry co2vmr to the value 1200e-6.

Programmer Information

IDEA(PG): Maybe we can provide examples of how these functions are used in the code?

Note

Actual changes are performed by the f90nml package patch fuction. See here: https://tinyurl.com/y4ydz363

Parameters

mconfig (dict) – The model (e.g. ECHAM, FESOM, NEMO or OIFS) configuration

Returns

mconfig – The modified configuration.

Return type

dict

static nmls_output(mconfig)[source]
static nmls_output_all(config)[source]
static nmls_remove(mconfig)[source]

Removes an element from a namelist chapter.

User Information

In the configuration file, assume you have:

echam:
    namelist_changes:
        namelist.echam:
            radctl:
                co2vmr: "remove_from_namelist"

In this case, the entry co2vmr would be deleted from the radctl section of namelist.echam.

Programmer Information

IDEA(PG): Maybe we can provide examples of how these functions are used in the code?

Parameters

mconfig (dict) – The model (e.g. ECHAM, FESOM, NEMO or OIFS) configuration

Returns

mconfig – The modified configuration.

Return type

dict

class esm_runscripts.namelists.namelist(*args, **kwargs)[source]

Bases: esm_runscripts.namelists.Namelist

Legacy class name. Please use Namelist instead!

esm_runscripts.oasis module

class esm_runscripts.oasis.oasis(nb_of_couplings=1, coupled_execs=['echam', 'fesom'], runtime=1, debug_level=1, nnorest='F', mct_version='4.0', lucia=False)[source]

Bases: object

add_coupling(lefts, lgrid, rights, rgrid, direction, transformation, restart_file, time_step, lresume, export_mode='DEFAULT')[source]
add_input_coupling(field_name, freq, field_filepath)[source]
add_output_file(lefts, rights, leftmodel, rightmodel, config)[source]
add_restart_files(restart_file, fconfig)[source]
finalize(destination_dir)[source]
prepare_restarts(restart_file, all_fields, models, config)[source]
print_config_files()[source]

esm_runscripts.postprocess module

esm_runscripts.postprocess.run_job(config)[source]

esm_runscripts.prepare module

esm_runscripts.prepare.add_submission_info(config)[source]
esm_runscripts.prepare.check_model_lresume(config)[source]
esm_runscripts.prepare.finalize_config(config)[source]
esm_runscripts.prepare.find_last_prepared_run(config)[source]
esm_runscripts.prepare.initialize_batch_system(config)[source]
esm_runscripts.prepare.initialize_coupler(config)[source]
esm_runscripts.prepare.model_env_into_computer(config)[source]

This function allows to store in the computer dictionary, variables that were defined inside environment_changes or compile/runtime_environment_changes in the components.

It excludes module_actions and export_vars dictionaries as those are resolved later.

This function is necessary for controlling choose_ blocks in the computer file from the component configuration file (i.e. add useMPI case to the component to control which useMPI case is selected in the computer file).

This function works both for compilation time and run time, and the result is that all components work under the same environment. The only exception is for the compilation of components, where add_export_vars and add_module_actions are excluded from the merging into computer, and are included individually in respective compilation scripts.

Later on, it might be desirable to always split the environments both for compiling (done by Paul Gierz, but this function would need to be adapted) and running (not done yet).

If this script gives you problems contact Miguel Andres-Martinez (miguel.andres-martinez@awi.de).

Parameters

config (dict) – Dictionary containing the simulation/compilation information

Raises

User Note/Error – If the same variable is found in two or more different component environments. Asks the user how to proceed.

esm_runscripts.prepare.resolve_some_choose_blocks(config)[source]
esm_runscripts.prepare.run_job(config)[source]
esm_runscripts.prepare.set_leapyear(config)[source]
esm_runscripts.prepare.set_logfile(config)[source]
esm_runscripts.prepare.set_most_dates(config)[source]
esm_runscripts.prepare.set_overall_calendar(config)[source]
esm_runscripts.prepare.set_parent_info(config)[source]
esm_runscripts.prepare.set_prev_date(config)[source]
esm_runscripts.prepare.set_restart_chunk(config)[source]

esm_runscripts.sim_objects module

Documentation goes here

class esm_runscripts.sim_objects.SimulationSetup(command_line_config=None, user_config=None)[source]

Bases: object

add_esm_runscripts_defaults_to_config(config)[source]
compute(kill_after_submit=True)[source]

All steps needed for a model computation.

Parameters

kill_after_submit (bool) – Default True. If set, the entire Python instance is killed with a sys.exit() as the very last after job submission.

distribute_per_model_defaults(config)[source]
get_total_config_from_user_config(user_config)[source]
get_user_config_from_command_line(command_line_config)[source]
inspect()[source]
postprocess()[source]

Calls post processing routines for this run.

tidy()[source]

Performs steps for tidying up a simulation after a job has finished and submission of following jobs.

This method uses two lists, all_files_to_copy and all_listed_filetypes to sort finished data from the current run folder back to the main experiment folder and submit new compute and post-process jobs. Files for log, mon, outdata, and restart_out are gathered. The program waits until the job completes or an error is found (See ~self.wait_and_observe). Then, if necessary, the coupler cleans up it’s files (unless it’s a standalone run), and the files in the lists are copied from the work folder to the current run folder. A check for unknown files is performed (see ~self.check_for_unknown_files), files are moved from the the current run folder to the main experiment folder, and new compute and post process jobs are started.

Warning

The date is changed during this routine! Be careful where you put any calls that may depend on date information!

Note

This method is also responsible for calling the next compute job as well as the post processing job!

viz(kill_after_submit=True)[source]

Starts the Viz job.

Parameters

kill_after_submit (bool) – Default True. If set, the entire Python instance is killed with sys.exit().

esm_runscripts.slurm module

Contains functions for dealing with SLURM-based batch systems

class esm_runscripts.slurm.Slurm(config)[source]

Bases: object

Deals with SLURM, allowing you to check if a job is submitted, get the current job ID, generate a srun hostfile, get the current job state, and check if a job is still running.

filename

The filename for srun commands, defaults to hostfile_srun

Type

str

path

Full path to this file, defaults to thisrun_scripts_dir / filename

Type

str

Parameters

config (dict) – The run configuration, needed to determine where the script directory for this particular run is.

calc_requirements(config)[source]

Calculates requirements and writes them to self.path.

calc_requirements_multi_srun(config)[source]
static check_if_submitted()[source]

Determines if a job is submitted in the currently running shell by checking for SLURM_JOB_ID in the environment

Returns

Return type

bool

static get_job_state(jobid)[source]

Returns the jobstate full name. See man squeue, section JOB STATE CODES for more details.

Parameters

jobidstr or int. The SLURM job id as displayed in, e.g. squeue

Returns

The short job state.

Return type

str

static get_jobid()[source]

Gets the current SLURM JOB ID

Returns

Return type

str or None

static job_is_still_running(jobid)[source]

Returns a boolean if the job is still running

static mini_calc_reqs(config, model, hostfile, start_proc, end_proc)[source]

esm_runscripts.tidy module

class esm_runscripts.tidy.RunFolders(config)[source]

Bases: list

Logs the run_ directories in <experiment_id>/log/run_folders.log, updating it with new folders. The resulting object is a list of run_ paths that exist or existed during the run time (even if they got deleted). This is useful for indexing operations such as <object_name>[::<interval>] used when removing run_ folders. .. rubric:: Notes

It keeps the folder names sorted so there is no need of sorting out of the object, and it also prevents the existence of duplicates.

load()[source]

Loads the existing paths of the run_ folders.

save()[source]

Saves all folder names.

update()[source]

Updates the folders read from the log file with the currently existing folders, removes duplicates, sorts them and save them into the log file.

esm_runscripts.tidy.all_done(config)[source]
esm_runscripts.tidy.assemble_error_list(config)[source]
esm_runscripts.tidy.check_for_errors(config)[source]
esm_runscripts.tidy.clean_run_dir(config)[source]

This plugin allows you to clean up the run_${DATE} folders. To do that you can use the following variables under the general section of your runscript (documentation follows order of code as it is executed):

  • clean_runs: This is the most important variable for most users. It can take the following values:

    • True: removes the run_ directory after each run (overrides every other clean_ option).

    • False: does not remove any run_ directory (default) if no clean_ variable is defined.

    • <int>: giving an integer as a value results in deleting the run_ folders except for the last <int> runs (recommended option as it allows for debugging of crashed simulations).

    Note

    clean_runs: (bool) is incompatible with clean_this_rundir and clean_runs: (int) is incompatible with clean_old_rundirs_except (an error will be raised after the end of the first simulation). The functionality of clean_runs variable alone will suffice most of the standard user requirements. If finer tunning for the removal of run_ directories is required you can used the following variables instead of clean_runs.

  • clean_this_rundir: (bool) Removes the entire run directory (equivalent to clean_runs: (bool)). clean_this_rundir: True overrides every other clean_ option.

  • clean_old_rundirs_except: (int) Removes the entire run directory except for the last <x> runs (equivalent to clean_runs: (int)).

  • clean_old_rundirs_keep_every: (int) Removes the entire run directory except every <x>th run. Compatible with clean_old_rundirs_except or clean_runs: (int).

  • clean_<filetype>_dir: (bool) Erases the run directory for a specific filetype. Compatible with all the other options.

  • clean_size: (int or float) Erases all files with size greater than clean_size, must be specified in bytes! Compatible with all the other options.

Example

To delete all the run_ directories in your experiment include this into your runscript:

general:
        clean_runs: True

To keep the last 2 run_ directories:

general:
        clean_runs: 2

To keep the last 2 runs and every 5 runs:

general:
        clean_old_rundirs_except: 2
        clean_old_rundirs_keep_every: 5
esm_runscripts.tidy.copy_all_results_to_exp(config)[source]
esm_runscripts.tidy.copy_stuff_back_from_work(config)[source]
esm_runscripts.tidy.get_last_jobid(config)[source]
esm_runscripts.tidy.init_monitor_file(config)[source]
esm_runscripts.tidy.job_is_still_running(config)[source]
esm_runscripts.tidy.maybe_resubmit(config)[source]
esm_runscripts.tidy.rm_r(path)[source]

Python equivalent of rm -r

Parameters

path (str) – Path or directory to remove

esm_runscripts.tidy.run_job(config)[source]
esm_runscripts.tidy.signal_tidy_completion(config)[source]
esm_runscripts.tidy.size_bytes_to_human(num, suffix='B')[source]
esm_runscripts.tidy.size_human_to_bytes(s, suffix='B')[source]
esm_runscripts.tidy.start_post_job(config)[source]
esm_runscripts.tidy.start_various_jobtypes_after_compute(config)[source]
esm_runscripts.tidy.throw_away_some_infiles(config)[source]
esm_runscripts.tidy.tidy_coupler(config)[source]
esm_runscripts.tidy.wait_and_observe(config)[source]
esm_runscripts.tidy.wake_up_call(config)[source]

esm_runscripts.virtual_env_builder module

esm_runscripts.virtual_env_builder.find_package(pkg)[source]
esm_runscripts.virtual_env_builder.get_base_prefix_compat()[source]

Get base/real prefix, or sys.prefix if there is none.

esm_runscripts.virtual_env_builder.in_virtualenv()[source]
esm_runscripts.virtual_env_builder.venv_bootstrap(config)[source]

Bootstraps your run into a virtual environment

esm_runscripts.yac module

class esm_runscripts.yac.yac(full_config, nb_of_couplings=1, coupled_models=['echam', 'fesom'], grids=['atmo', 'feom'], runtime=1)[source]

Bases: object

Generates the configuration file for YAC coupler.

add_coupling(field, transient_id, direction, config)[source]
add_output_file(lefts, rights, leftmodel, rightmodel, config)[source]
add_restart_files(restart_file, fconfig)[source]
finalize(destination_dir)[source]
prepare_restarts(restart_file, all_fields, model, config)[source]
print_config_files()[source]