esm_parser package

Submodules

esm_parser.dict_to_yaml module

esm_parser.dict_to_yaml.add_eol_comments_with_provenance(commented_config, config)[source]

Add end-of-line comments to the commented_config with provenance information from the config.

Parameters

commented_config (dict) – Dictionary with the config values and ruamel.yaml comments.
config (dict) – Dictionary with the config values and provenance information.

esm_parser.dict_to_yaml.delete_prev_objects(config)[source]

Delete key-values in the config which values correspond to prev_ objects, for example, prev_run (contains values of the config of the previous run) and prev_chunk_<model> (that contains values of the config of a previous chunk in a offline coupled simulation). This deletion is necessary because otherwise the finished_config.yaml gets a lot of nested information from the previous runs that keeps growing each new run/chunk.

Parameters: config (dict) – Dictionary containing the simulation/compilation information

esm_parser.dict_to_yaml.yaml_dump(config, config_file_path=None)[source]

Dump the config dictionary to a YAML file. The YAML file will contain comments with provenance information.

Parameters

config (dict) – Dictionary containing the simulation/compilation information
config_file_path (str, optional) – Path to the YAML file where the config will be saved. If not provided, the config will be printed to the standard output.

esm_parser.esm_parser module

esm_parser.provenance module

Provenance’s dark magic. The basic idea is that one use the following to understand from which yaml file (line and column) a variable in config is coming from:

config["fesom"]["version"].provenance

And that will return a list of the provenance history of that variable, for example:

[{'category': 'components',
  'col': 10,
  'line': 6,
  'yaml_file': '/Users/mandresm/Codes/esm_tools/configs/components/fesom/fesom-2.0.yaml'},
 {'category': 'setups',
  'col': 18,
  'extended_by': 'dict.__setitem__',
  'line': 321,
  'yaml_file': '/Users/mandresm/Codes/esm_tools/configs/setups/awicm3/awicm3.yaml'}]

The last element in the provenance list represents the provenance of the current value (last provenance).

This module contains: * The provenance class, to store the provenance of values with extended functionality * A wrapper factory to create classes and objects dynamically that subclass the value

types and append provenances to them (WithProvenance classes)

Class attributes common to all WithProvenance classes
Classes for mappings with provenance (dictionaries and lists) to recursively put and
get provenance from nested values, and extend the standard mapping methods (__setitem__, update…)
A decorator to keep provenance in esm_parser’s recursive functions
A method to clean provenance recursively and get back the data without provenance

class esm_parser.provenance.BoolWithProvenance(value, provenance=None)[source]

Bases: ProvenanceClassForTheUnsubclassable

Class for emulating Bool behaviour, but with Provenance.

Objects of this class reproduce the following Bool behaviours: * isinstance(<obj>, bool) returns True * <True_obj> == True returns True * <True_obj> is True returns False. This is not reproducing the behavior!

class esm_parser.provenance.DictWithProvenance(dictionary, provenance)[source]

Bases: dict

A dictionary subclass that contains methods for: * recursively transforming leaf values into provenance (put_provenance and

set_provenance)

recursively retrieving provenance from nested values
extending the dict.__init__ method to recursively assign provenance to all
nested values
extending the dict.__setitem__ method to keep a record of previous history
when adding new values to a given key
extending the dict.update method to keep a record of the previous history
when updating the dictionary

Use

Instance a new DictWithProvenance object:

dict_with_provenance = DictWithProvenance(<a_dictionary>, <my_provenance>)

Redefine the provenance of an existing DictWithProvenance with the same provenance for all its nested values:

dict_with_provenance.set_provenance(<new_provenance>)

Set the provenace of a specific leaf within a nested dictionary:

dict_with_provenance["key1"]["key1"].provenance = <new_provenance>

Get the provenance representation of the whole dictionary:

provenance_dict = dict_with_provenance.get_provenance()

extract_first_nested_values_provenance()[source]

Recursively loops through the dictionary keys and returns the first provenance found in the nested values.

Returns: first_provenance – The first provenance found in the nested values
Return type: esm_parser.provenance.Provenance

get_provenance(index=-1)[source]

Returns a dictionary containing the all the nested provenance information of the current DictWithProvenance with a structure and keys equivalent to the self dictionary, but with values of the key leaves those of the provenance.

Parameters: index (int) – Defines the element of the provenance history to be returned. The default is -1, meaning the last provenance (the one of the current value).
Returns: provenance_dict – A dictionary with a structure and keys equivalent to the self dictionary, but with values of the key leaves those of the provenance
Return type: dict

put_provenance(provenance)[source]

Recursively transforms every value in DictWithProvenance into its corresponding WithProvenance object and appends its corresponding provenance. Each value has its corresponding provenance defined in the provenance dictionary, and this method just groups them together 1-to-1.

Parameters: provenance (dict) – The provenance that will be recursively assigned to all leaves of the dictionary tree. The provenance needs to be a dict with the same keys as self (same structure) so that it can successfully transfer each provenance value to its corresponding value on self (1-to-1 conrrespondance).

set_provenance(provenance)[source]

Recursively transforms every value in DictWithProvenance into its corresponding WithProvenance object and appends the same provenance to it. Note that this method differs from put_provenance in that the same provenance value is applied to the different values of self.

Parameters: provenance (any) – New provenance value to be set

update(dictionary, *args, **kwargs)[source]

Preserves the provenance history when using the update method

Parameters: dictionary (dict, esm_parser.provenance.DictWithProvenance) – Dictionary that will update self

yaml_dump(config_file_path=None)

Dump the config dictionary to a YAML file. The YAML file will contain comments with provenance information.

Parameters

config (dict) – Dictionary containing the simulation/compilation information
config_file_path (str, optional) – Path to the YAML file where the config will be saved. If not provided, the config will be printed to the standard output.

class esm_parser.provenance.ListWithProvenance(mylist, provenance)[source]

Bases: list

A list subclass that contains methods for: * recursively transforming leaf values into provenance (put_provenance and

set_provenance)

recursively retrieving provenance from nested values
extending the list.__init__ method to recursively assign provenance to all
nested values
extending the list.__setitem__ method to keep a record of previous history
when adding new values to a given key

Use

Instance a new ListWithProvenance object:

list_with_provenance = ListWithProvenance(<a_list>, <my_provenance>)

Redefine the provenance of an existing ListWithProvenance with the same provenance for all its nested values:

list_with_provenance.set_provenance(<new_provenance>)

Set the provenace of the element 0 of a list:

list_with_provenance[0].provenance = <new_provenance>

Get the provenance representation of the whole list:

provenance_list = list_with_provenance.get_provenance()

extract_first_nested_values_provenance()[source]

Recursively loops through the list elements and returns the first provenance found in the nested values.

Returns: first_provenance – The first provenance found in the nested values
Return type: esm_parser.provenance.Provenance

get_provenance(index=-1)[source]

Returns a list containing the all the nested provenance information of the current ListWithProvenance with a structure equivalent to the self list, but with list elements been provenance values.

Parameters: index (int) – Defines the element of the provenance history to be returned. The default is -1, meaning the last provenance (the one of the current value).
Returns: provenance_list – A list with a structure equivalent to that of the self list, but with the values of the provenance of each element
Return type: list

put_provenance(provenance)[source]

Recursively transforms every value in ListWithProvenance into its corresponding WithProvenance object and appends its corresponding provenance. Each value has its corresponding provenance defined in the provenance list, and this method just groups them together 1-to-1.

Parameters: provenance (list) – The provenance that will be recursively assigned to all elements of the list. The provenance needs to be a list with the same number of elements as self (same structure) so that it can successfully transfer each provenance value to its corresponding value on self (1-to-1 conrrespondance).

set_provenance(provenance)[source]

Recursively transforms every value in ListWithProvenance into its corresponding WithProvenance object and appends the same provenance to it. Note that this method differs from put_provenance in that the same provenance value is applied to the different values of self.

Parameters: provenance (any) – New provenance value to be set

yaml_dump(config_file_path=None)

Dump the config dictionary to a YAML file. The YAML file will contain comments with provenance information.

Parameters

config (dict) – Dictionary containing the simulation/compilation information
config_file_path (str, optional) – Path to the YAML file where the config will be saved. If not provided, the config will be printed to the standard output.

class esm_parser.provenance.NoneWithProvenance(value, provenance=None)[source]

Bases: ProvenanceClassForTheUnsubclassable

Class for emulating None behaviour, but with Provenance.

Objects of this class reproduce the following None behaviours: * isinstance(<obj>, None) returns True * <obj> == None returns True * <obj> is None returns False. This is not reproducing the behavior!

class esm_parser.provenance.Provenance(provenance_data)[source]

Bases: list

A subclass of list in which each element represents the provenance of the value at a point in the key-value history. The whole point of this class is to have a list subclass that allows us to include information about which function is changing the list within each provenance element.

To assign the provenance to a value, instanciate it as an attribute of that value (i.e. self.provenance = Provenance(my_provenance)). To be used from the WithProvenance classes created by wrapper_with_provenance_factory.

The following class methods provide the extended functionality to lists: * self.append_last_step_modified_by: to duplicate the last element of the list

and add to it information about the function that is modifying the value

self.extend_and_modified_by: to extend a list while including in the
provenance the function which is responsible for extending it

add_modified_by(provenance_step, func, modified_by='modified_by')[source]

Adds a variable of name defined by modified_by to the given provenance step with value func. This variable is used to label provenance steps of the provenance history with functions that modified it.

Parameters

provenance_step (dict) – Provenance entry of the current step
func (str) – Function triggering this method
modified_by (str) – Name of the key for the labelling the type of modification

Returns

provenance_step – Provenance entry of the current step with the modified_by item

Return type

dict

append_last_step_modified_by(func)[source]

Copies the last element in the provenance history and adds the entry modify_by with value func to the copy.

Parameters: func (str) – Function that is modifying the variable

extend_and_modified_by(additional_provenance, func)[source]

Extends the current provenance history with an additional_provenance. This happens when for example a variable comes originally from a file, but then the value is overwritten by another value that comes from a file higher in the hierarchy. This method keeps both histories, with the history of the second been on top of the first.

Parameters

additional_provenance (esm_parser.Provenance) – Additional provenance history to be used for extending self
func (str) – Function triggering this method

class esm_parser.provenance.ProvenanceClassForTheUnsubclassable(value, provenance=None)[source]

Bases: object

A class to reproduce the methods of the unclassable bool and NoneType classes, needed to add the provenance attribute to those versions of the types.

This class stores 2 attributes, one is the value of the target object (a bool or a NoneType) and the other is the provenance.

property provenance

To be used as the provenance property in WithProvenance classes.

Returns: self._provenance – The provenance history stored in self._provenance
Return type: esm_parser.provenance.Provenance

esm_parser.provenance.clean_provenance(data)[source]

Returns the values of provenance mappings in their original classes (without the provenance). Recurs through mappings. Make sure you copy.deepcopy the data mapping before running this function if you don’t want that your provenance information gets lost on the original data mapping.

Parameters: data (any) – Mapping or values with provenance.
Returns: value – Values in their original format, or lists and dictionaries containing provenance values.
Return type: any

esm_parser.provenance.keep_provenance_in_recursive_function(func)[source]

Decorator for recursive functions in esm_parser to preserve provenance.

Parameters: func (Callable) – The function to decorate

esm_parser.provenance.wrapper_with_provenance_factory(value, provenance=None)[source]

A function to subclass and instanciate all types of subclassable objects in the ESM-Tools config and add the provenance attribute to them. It also creates the {type(value)}WithProvenance classes globally on the fly depending on the value’s type, if it doesn’t exist yet. For classes that are not subclassable (Date, Bool and NoneType) intanciates an object that mimics their behaviour but also contains the provenance attribute.

Objects of type esm_calendar.esm_calendar.Date are not subclassed (and the provenance attribute is simply added to them, because they fail to be subclassed with in the DateWithProvenance with the following error:

__new__ method giving error object.__new__() takes exactly one argument
(the type to instantiate)

Parameters

value (any) – Value of the object to be subclassed and reinstanciated
provenance (any) – The provenance information

Returns

{type(value)}WithProvenance, esm_calendar.esm_calendar.Date, BoolWithProvenance,
NoneWithProvenance – The new instance with the provenance attribute

esm_parser.provenance.wrapper_with_provenance_init(self, value, provenance=None)[source]

To be used as the __init__ method for WithProvenance classes. Adds the provenance value as an instance of Provenance to the self._provenance attribute, and stores the original value to the self.value attribute.

Parameters

value (any) – Value of the object
provenance (any) – The provenance information

esm_parser.provenance.wrapper_with_provenance_new()

classmethod(function) -> method

Convert a function to be a class method.

A class method receives the class as implicit first argument, just like an instance method receives the instance. To declare a class method, use this idiom:

class C:
@classmethod def f(cls, arg1, arg2, …):

…

It can be called either on the class (e.g. C.f()) or on an instance (e.g. C().f()). The instance is ignored except for its class. If a class method is called for a derived class, the derived class object is passed as the implied first argument.

Class methods are different than C++ or Java static methods. If you want those, see the staticmethod builtin.

esm_parser package

Submodules

esm_parser.dict_to_yaml module

esm_parser.esm_parser module

esm_parser.provenance module

Use

Use

esm_parser.yaml_to_dict module