esm_parser package

Submodules

esm_parser.dict_to_yaml module

esm_parser.dict_to_yaml.add_eol_comments_with_provenance(commented_config, config)[source]

Add end-of-line comments to the commented_config with provenance information from the config.

Parameters

  • commented_config (dict) – Dictionary with the config values and ruamel.yaml comments.

  • config (dict) – Dictionary with the config values and provenance information.

esm_parser.dict_to_yaml.delete_prev_objects(config)[source]

Delete key-values in the config which values correspond to prev_ objects, for example, prev_run (contains values of the config of the previous run) and prev_chunk_<model> (that contains values of the config of a previous chunk in a offline coupled simulation). This deletion is necessary because otherwise the finished_config.yaml gets a lot of nested information from the previous runs that keeps growing each new run/chunk.

Parameters

config (dict) – Dictionary containing the simulation/compilation information

esm_parser.dict_to_yaml.yaml_dump(config, config_file_path=None)[source]

Dump the config dictionary to a YAML file. The YAML file will contain comments with provenance information.

Parameters

  • config (dict) – Dictionary containing the simulation/compilation information

  • config_file_path (str, optional) – Path to the YAML file where the config will be saved. If not provided, the config will be printed to the standard output.

esm_parser.esm_parser module

esm_parser.provenance module

Provenance’s dark magic. The basic idea is that one use the following to understand from which yaml file (line and column) a variable in config is coming from:

config["fesom"]["version"].provenance

And that will return a list of the provenance history of that variable, for example:

[{'category': 'components',
  'col': 10,
  'line': 6,
  'yaml_file': '/Users/mandresm/Codes/esm_tools/configs/components/fesom/fesom-2.0.yaml'},
 {'category': 'setups',
  'col': 18,
  'extended_by': 'dict.__setitem__',
  'line': 321,
  'yaml_file': '/Users/mandresm/Codes/esm_tools/configs/setups/awicm3/awicm3.yaml'}]

The last element in the provenance list represents the provenance of the current value (last provenance).

This module contains: * The provenance class, to store the provenance of values with extended functionality * A wrapper factory to create classes and objects dynamically that subclass the value

types and append provenances to them (WithProvenance classes)

  • Class attributes common to all WithProvenance classes

  • Classes for mappings with provenance (dictionaries and lists) to recursively put and

    get provenance from nested values, and extend the standard mapping methods (__setitem__, update…)

  • A decorator to keep provenance in esm_parser’s recursive functions

  • A method to clean provenance recursively and get back the data without provenance

class esm_parser.provenance.BoolWithProvenance(value, provenance=None)[source]

Bases: ProvenanceClassForTheUnsubclassable

Class for emulating Bool behaviour, but with Provenance.

Objects of this class reproduce the following Bool behaviours: * isinstance(<obj>, bool) returns True * <True_obj> == True returns True * <True_obj> is True returns False. This is not reproducing the behavior!

class esm_parser.provenance.DictWithProvenance(dictionary, provenance)[source]

Bases: dict

A dictionary subclass that contains methods for: * recursively transforming leaf values into provenance (put_provenance and

set_provenance)

  • recursively retrieving provenance from nested values

  • extending the dict.__init__ method to recursively assign provenance to all

    nested values

  • extending the dict.__setitem__ method to keep a record of previous history

    when adding new values to a given key

  • extending the dict.update method to keep a record of the previous history

    when updating the dictionary

Use

Instance a new DictWithProvenance object:

dict_with_provenance = DictWithProvenance(<a_dictionary>, <my_provenance>)

Redefine the provenance of an existing DictWithProvenance with the same provenance for all its nested values:

dict_with_provenance.set_provenance(<new_provenance>)

Set the provenace of a specific leaf within a nested dictionary:

dict_with_provenance["key1"]["key1"].provenance = <new_provenance>

Get the provenance representation of the whole dictionary:

provenance_dict = dict_with_provenance.get_provenance()
get_provenance(index=-1)[source]

Returns a dictionary containing the all the nested provenance information of the current DictWithProvenance with a structure and keys equivalent to the self dictionary, but with values of the key leaves those of the provenance.

Parameters

index (int) – Defines the element of the provenance history to be returned. The default is -1, meaning the last provenance (the one of the current value).

Returns

provenance_dict – A dictionary with a structure and keys equivalent to the self dictionary, but with values of the key leaves those of the provenance

Return type

dict

put_provenance(provenance)[source]

Recursively transforms every value in DictWithProvenance into its corresponding WithProvenance object and appends its corresponding provenance. Each value has its corresponding provenance defined in the provenance dictionary, and this method just groups them together 1-to-1.

Parameters

provenance (dict) – The provenance that will be recursively assigned to all leaves of the dictionary tree. The provenance needs to be a dict with the same keys as self (same structure) so that it can successfully transfer each provenance value to its corresponding value on self (1-to-1 conrrespondance).

set_provenance(provenance)[source]

Recursively transforms every value in DictWithProvenance into its corresponding WithProvenance object and appends the same provenance to it. Note that this method differs from put_provenance in that the same provenance value is applied to the different values of self.

Parameters

provenance (any) – New provenance value to be set

update(dictionary, *args, **kwargs)[source]

Preserves the provenance history when using the update method

Parameters

dictionary (dict, esm_parser.provenance.DictWithProvenance) – Dictionary that will update self

yaml_dump(config_file_path=None)

Dump the config dictionary to a YAML file. The YAML file will contain comments with provenance information.

Parameters

  • config (dict) – Dictionary containing the simulation/compilation information

  • config_file_path (str, optional) – Path to the YAML file where the config will be saved. If not provided, the config will be printed to the standard output.

class esm_parser.provenance.ListWithProvenance(mylist, provenance)[source]

Bases: list

A list subclass that contains methods for: * recursively transforming leaf values into provenance (put_provenance and

set_provenance)

  • recursively retrieving provenance from nested values

  • extending the list.__init__ method to recursively assign provenance to all

    nested values

  • extending the list.__setitem__ method to keep a record of previous history

    when adding new values to a given key

Use

Instance a new ListWithProvenance object:

list_with_provenance = ListWithProvenance(<a_list>, <my_provenance>)

Redefine the provenance of an existing ListWithProvenance with the same provenance for all its nested values:

list_with_provenance.set_provenance(<new_provenance>)

Set the provenace of the element 0 of a list:

list_with_provenance[0].provenance = <new_provenance>

Get the provenance representation of the whole list:

provenance_list = list_with_provenance.get_provenance()
get_provenance(index=-1)[source]

Returns a list containing the all the nested provenance information of the current ListWithProvenance with a structure equivalent to the self list, but with list elements been provenance values.

Parameters

index (int) – Defines the element of the provenance history to be returned. The default is -1, meaning the last provenance (the one of the current value).

Returns

provenance_list – A list with a structure equivalent to that of the self list, but with the values of the provenance of each element

Return type

list

put_provenance(provenance)[source]

Recursively transforms every value in ListWithProvenance into its corresponding WithProvenance object and appends its corresponding provenance. Each value has its corresponding provenance defined in the provenance list, and this method just groups them together 1-to-1.

Parameters

provenance (list) – The provenance that will be recursively assigned to all elements of the list. The provenance needs to be a list with the same number of elements as self (same structure) so that it can successfully transfer each provenance value to its corresponding value on self (1-to-1 conrrespondance).

set_provenance(provenance)[source]

Recursively transforms every value in ListWithProvenance into its corresponding WithProvenance object and appends the same provenance to it. Note that this method differs from put_provenance in that the same provenance value is applied to the different values of self.

Parameters

provenance (any) – New provenance value to be set

yaml_dump(config_file_path=None)

Dump the config dictionary to a YAML file. The YAML file will contain comments with provenance information.

Parameters

  • config (dict) – Dictionary containing the simulation/compilation information

  • config_file_path (str, optional) – Path to the YAML file where the config will be saved. If not provided, the config will be printed to the standard output.

class esm_parser.provenance.NoneWithProvenance(value, provenance=None)[source]

Bases: ProvenanceClassForTheUnsubclassable

Class for emulating None behaviour, but with Provenance.

Objects of this class reproduce the following None behaviours: * isinstance(<obj>, None) returns True * <obj> == None returns True * <obj> is None returns False. This is not reproducing the behavior!

class esm_parser.provenance.Provenance(provenance_data)[source]

Bases: list

A subclass of list in which each element represents the provenance of the value at a point in the key-value history. The whole point of this class is to have a list subclass that allows us to include information about which function is changing the list within each provenance element.

To assign the provenance to a value, instanciate it as an attribute of that value (i.e. self.provenance = Provenance(my_provenance)). To be used from the WithProvenance classes created by wrapper_with_provenance_factory.

The following class methods provide the extended functionality to lists: * self.append_last_step_modified_by: to duplicate the last element of the list

and add to it information about the function that is modifying the value

  • self.extend_and_modified_by: to extend a list while including in the

    provenance the function which is responsible for extending it

add_modified_by(provenance_step, func, modified_by='modified_by')[source]

Adds a variable of name defined by modified_by to the given provenance step with value func. This variable is used to label provenance steps of the provenance history with functions that modified it.

Parameters

  • provenance_step (dict) – Provenance entry of the current step

  • func (str) – Function triggering this method

  • modified_by (str) – Name of the key for the labelling the type of modification

Returns

provenance_step – Provenance entry of the current step with the modified_by item

Return type

dict

append_last_step_modified_by(func)[source]

Copies the last element in the provenance history and adds the entry modify_by with value func to the copy.

Parameters

func (str) – Function that is modifying the variable

extend_and_modified_by(additional_provenance, func)[source]

Extends the current provenance history with an additional_provenance. This happens when for example a variable comes originally from a file, but then the value is overwritten by another value that comes from a file higher in the hierarchy. This method keeps both histories, with the history of the second been on top of the first.

Parameters

  • additional_provenance (esm_parser.Provenance) – Additional provenance history to be used for extending self

  • func (str) – Function triggering this method

class esm_parser.provenance.ProvenanceClassForTheUnsubclassable(value, provenance=None)[source]

Bases: object

A class to reproduce the methods of the unclassable bool and NoneType classes, needed to add the provenance attribute to those versions of the types.

This class stores 2 attributes, one is the value of the target object (a bool or a NoneType) and the other is the provenance.

property provenance

To be used as the provenance property in WithProvenance classes.

Returns

self._provenance – The provenance history stored in self._provenance

Return type

esm_parser.provenance.Provenance

esm_parser.provenance.clean_provenance(data)[source]

Returns the values of provenance mappings in their original classes (without the provenance). Recurs through mappings. Make sure you copy.deepcopy the data mapping before running this function if you don’t want that your provenance information gets lost on the original data mapping.

Parameters

data (any) – Mapping or values with provenance.

Returns

value – Values in their original format, or lists and dictionaries containing provenance values.

Return type

any

esm_parser.provenance.keep_provenance_in_recursive_function(func)[source]

Decorator for recursive functions in esm_parser to preserve provenance.

Parameters

func (Callable) – The function to decorate

esm_parser.provenance.wrapper_with_provenance_factory(value, provenance=None)[source]

A function to subclass and instanciate all types of subclassable objects in the ESM-Tools config and add the provenance attribute to them. It also creates the {type(value)}WithProvenance classes globally on the fly depending on the value’s type, if it doesn’t exist yet. For classes that are not subclassable (Date, Bool and NoneType) intanciates an object that mimics their behaviour but also contains the provenance attribute.

Objects of type esm_calendar.esm_calendar.Date are not subclassed (and the provenance attribute is simply added to them, because they fail to be subclassed with in the DateWithProvenance with the following error:

__new__ method giving error object.__new__() takes exactly one argument
(the type to instantiate)
Parameters

  • value (any) – Value of the object to be subclassed and reinstanciated

  • provenance (any) – The provenance information

Returns

  • {type(value)}WithProvenance, esm_calendar.esm_calendar.Date, BoolWithProvenance,

  • NoneWithProvenance – The new instance with the provenance attribute

esm_parser.provenance.wrapper_with_provenance_init(self, value, provenance=None)[source]

To be used as the __init__ method for WithProvenance classes. Adds the provenance value as an instance of Provenance to the self._provenance attribute, and stores the original value to the self.value attribute.

Parameters

  • value (any) – Value of the object

  • provenance (any) – The provenance information

esm_parser.provenance.wrapper_with_provenance_new()

classmethod(function) -> method

Convert a function to be a class method.

A class method receives the class as implicit first argument, just like an instance method receives the instance. To declare a class method, use this idiom:

class C:

@classmethod def f(cls, arg1, arg2, …):

…

It can be called either on the class (e.g. C.f()) or on an instance (e.g. C().f()). The instance is ignored except for its class. If a class method is called for a derived class, the derived class object is passed as the implied first argument.

Class methods are different than C++ or Java static methods. If you want those, see the staticmethod builtin.

esm_parser.yaml_to_dict module