midgard.config

midgard.config.config

Midgard library module for handling of configuration settings

Description:

A Configuration consists of one or several sections. Each ConfigurationSection consists of one or more entries. Each ConfigurationEntry consists of a key and a value.

Examples:

For basic use, an entry is looked up by simple attribute access. For instance if cfg is a Configuration with the section midgard which has an entry foo = bar:

>>> cfg = Configuration("config_name")
>>> cfg.update("midgard", "foo", "bar")
>>> cfg.midgard.foo
ConfigurationEntry(key='foo', value='bar')

ConfigurationEntry has several access methods that convert the entry to a given data type:

>>> cfg.update("midgard", "foo_pi", 3.14, source="command line")
>>> cfg.midgard.foo_pi
ConfigurationEntry(key='foo_pi', value='3.14')
>>> cfg.midgard.foo_pi.float
3.14
>>> cfg.midgard.foo_pi.str
'3.14'
>>> cfg.midgard.foo_pi.tuple
('3.14',)

Sources:

Each configuration entry records its source. That is, where that entry was defined. Examples include read from file, set as a command line option, or programmatically from a dictionary. The source can be looked up on an individual entry, or for all entries in a configuration.

>>> cfg.midgard.foo_pi.source
'command line'
>>> cfg.sources  # doctest: +SKIP
{'/home/midgard/midgard.conf', 'command line'}

Profiles:

Fallback Configuration:

Master Section:

Replacement Variables:

Help text and Type hints:

CasedConfigParser

Full name: midgard.config.config.CasedConfigParser

Signature: (defaults=None, dict_type=<class 'collections.OrderedDict'>, allow_no_value=False, *, delimiters=('=', ':'), comment_prefixes=('#', ';'), inline_comment_prefixes=None, strict=True, empty_lines_in_values=True, default_section='DEFAULT', interpolation=<object object at 0x7f1507871200>, converters=<object object at 0x7f1507871200>)

ConfigParser with case-sensitive keys

CasedConfigParser.BOOLEAN_STATES (dict)

BOOLEAN_STATES = {'1': True, 'yes': True, 'true': True, 'on': True, '0': False, 'no': False, 'false': False, 'off': False}

CasedConfigParser.NONSPACECRE (SRE_Pattern)

NONSPACECRE = re.compile('\\S')

CasedConfigParser.OPTCRE (SRE_Pattern)

OPTCRE = re.compile('\n (?P<option>.*?) # very permissive!\n \\s*(?P<vi>=|:)\\s* # any number of space/tab,\n # followed by any of t, re.VERBOSE)

CasedConfigParser.OPTCRE_NV (SRE_Pattern)

OPTCRE_NV = re.compile('\n (?P<option>.*?) # very permissive!\n \\s*(?: # any number of space/tab,\n (?P<vi>=|:)\\s* # optionally followed , re.VERBOSE)

CasedConfigParser.SECTCRE (SRE_Pattern)

SECTCRE = re.compile('\n \\[ # [\n (?P<header>[^]]+) # very permissive!\n \\] # ]\n ', re.VERBOSE)

CasedConfigParser.optionxform()

Full name: midgard.config.config.optionxform

Signature: (self, optionstr:str) -> str

Do not turn optionstr (key) into lowercase

Configuration

Full name: midgard.config.config.Configuration

Signature: (name:str) -> None

Represents a Configuration

Configuration.as_dict()

Full name: midgard.config.config.as_dict

Signature: (self, getters:Union[Dict[str, Dict[str, str]], NoneType]=None, default_getter:str='str') -> Dict[str, Dict[str, Any]]

The configuration represented as a dictionary

Args:

Returns:

Representation of the configuration as a nested dictionary.

Configuration.as_str()

Full name: midgard.config.config.as_str

Signature: (self, width:Union[int, NoneType]=None, key_width:int=30, only_used:bool=False, metadata:bool=True) -> str

The configuration represented as a string

This is simililar to what is shown by str(configuration) (and implemented by __str__), but has more flexibility.

Args:

Returns:

String representation of the configuration.

Configuration.clear()

Full name: midgard.config.config.clear

Signature: (self) -> None

Clear the configuration

Configuration.clear_vars()

Full name: midgard.config.config.clear_vars

Signature: (self) -> None

Clear the configuration variables

Configuration.exists()

Full name: midgard.config.config.exists

Signature: (self, key:str, section:Union[str, NoneType]=None) -> bool

Check if a configuration entry exists

Args:

Returns:

True if the configuration entry exists, otherwise False.

Configuration.get()

Full name: midgard.config.config.get

Signature: (self, key:str, value:Union[str, NoneType]=None, section:Union[str, NoneType]=None, default:Union[str, NoneType]=None) -> 'ConfigurationEntry'

Get an entry from a configuration with possibility for override and default value

A value for an entry is found using the following priorities:

1. An explicit value given in `value`. None is used as a marker for no value.
2. Looked up in the current configuration.
3. Looked up in any fallback confiurations that are defined.
4. The default value is used.

If value is not None, that value is simply returned as a ConfigurationEntry. If default is not given (is None), and a value is not found in any other way, a MissingEntryError is raised.

Args:

Returns:

Entry representing the value.

Configuration.read_from_file()

Full name: midgard.config.config.read_from_file

Signature: (cfg_name:str, *file_paths:Union[str, pathlib.Path]) -> 'Configuration'

Read a configuration from one or more files

Args:

Returns:

A Configuration representing the file(s).

Configuration.update()

Full name: midgard.config.config.update

Signature: (self, section:str, key:str, value:str, *, profile:Union[str, NoneType]=None, source:str='unknown', meta:Union[Dict[str, str], NoneType]=None, allow_new:bool=True, _update_sections:bool=True) -> None

Update a configuration section with a configuration entry

If allow_new is False, the configuration entry must already exist. If it is True the update is allowed to create a new section and a new entry is necessary.

The _update_sections flag can be used to not update the sections of the configuration, only the profiles. This should typically not be done, but is used by some of the other update methods which update the sections themselves.

Args:

Configuration.update_from_config_section()

Full name: midgard.config.config.update_from_config_section

Signature: (self, other_section:'ConfigurationSection', section:Union[str, NoneType]=None, allow_new:bool=True) -> None

Configuration.update_from_dict()

Full name: midgard.config.config.update_from_dict

Signature: (self, cfg_dict:Dict[str, Any], section:Union[str, NoneType]=None, source:str='dictionary', allow_new:bool=True) -> None

Configuration.update_from_file()

Full name: midgard.config.config.update_from_file

Signature: (self, file_path:Union[str, pathlib.Path], allow_new:bool=True, interpolate:bool=False, case_sensitive:bool=False) -> None

Update the configuration from a configuration file

The Python ConfigParser is used to read the file. The file format that is supported is described at https://docs.python.org/library/configparser.html

Different profiles in a configuration file are denoted by double underscores in the sections names. For instance will the following configuration have a foo profile in the spam section (in addition to the default profile):

[spam]
...

[spam__foo]
...

The file may contain a special section called __replace__ which may contain key-value pairs which will replace format-style strings in keys and values in the rest of the file.

Additionally, the file may contain a special section called __vars__. The key-value pairs from this section will be added to the dictionary of the configuration.

If interpolate is set to True, ExtendedInterpolation of variables in the configuration file is used. This means that variables of the form ${key:section} can be used for references within the file. See https://docs.python.org/library/configparser.html#configparser.ExtendedInterpolation for details.

Args:

Configuration.update_from_options()

Full name: midgard.config.config.update_from_options

Signature: (self, options:Union[List[str], NoneType]=None, profile:Union[str, NoneType]=None, source:str='command line', allow_new:bool=False) -> None

Configuration.update_on_file()

Full name: midgard.config.config.update_on_file

Signature: (file_path:Union[str, pathlib.Path], **as_str_args:Any) -> Generator

Context manager for updating a configuration on file

Configuration.update_vars()

Full name: midgard.config.config.update_vars

Signature: (self, new_vars:Dict[str, str]) -> None

Update the configuration variables

Configuration.write_to_file()

Full name: midgard.config.config.write_to_file

Signature: (self, file_path:Union[str, pathlib.Path], **as_str_args:Any) -> None

Write the configuration to a file

In addition to the file path, arguments can be specified and will be passed on to the as_str() function. See as_str() for more information.

Todo: Use files.open_path

ConfigurationEntry

Full name: midgard.config.config.ConfigurationEntry

Signature: (key:str, value:Any, *, source:str='', meta:Union[Dict[str, str], NoneType]=None, vars_dict:Union[Dict[str, str], NoneType]=None, _used_as:Union[Set[str], NoneType]=None) -> None

ConfigurationEntry.as_bool()

Full name: midgard.config.config.as_bool

Signature: (self) -> bool

Value of ConfigurationEntry converted to a boolean

The conversion is done by looking up the string value of the entry in _BOOLEAN_STATES.

ConfigurationEntry.as_date()

Full name: midgard.config.config.as_date

Signature: (self, format:str='%Y-%m-%d') -> datetime.date

Value of ConfigurationEntry converted to a date object

Args:

format (String): Format string, see strftime for information about the string.

Returns:

ConfigurationEntry.as_datetime()

Full name: midgard.config.config.as_datetime

Signature: (self, format:str='%Y-%m-%d %H:%M:%S') -> datetime.datetime

Value of ConfigurationEntry converted to a datetime object

Args:

format (String): Format string, see strftime for information about the string.

Returns:

ConfigurationEntry.as_dict()

Full name: midgard.config.config.as_dict

Signature: (self, item_split_re:str='[\\s,]', key_value_split_re:str='[:]', convert:Callable=<class 'str'>, maxsplit:int=0) -> Dict[str, Any]

Value of ConfigurationEntry converted to a dictionary

By default the dictionary is created by splitting items at commas and whitespace, and key from value at colons.

Args:

Returns:

Value of entry as dict.

ConfigurationEntry.as_enum()

Full name: midgard.config.config.as_enum

Signature: (self, enum:str) -> enum.Enum

Value of ConfigurationEntry converted to an enumeration

Args:

enum (String): Name of Enum.

Returns:

ConfigurationEntry.as_float()

Full name: midgard.config.config.as_float

Signature: (self) -> float

Value of ConfigurationEntry converted to a float

ConfigurationEntry.as_int()

Full name: midgard.config.config.as_int

Signature: (self) -> int

Value of ConfigurationEntry converted to an integer

ConfigurationEntry.as_list()

Full name: midgard.config.config.as_list

Signature: (self, split_re:str='[\\s,]', convert:Callable=<class 'str'>, maxsplit:int=0) -> List[Any]

Value of ConfigurationEntry converted to a list

The entry is converted to a list by using the split_re-regular expression. By default the entry will be split at commas and whitespace.

Args:

Returns:

Value of entry as list.

ConfigurationEntry.as_list_of_lists()

Full name: midgard.config.config.as_list_of_lists

Signature: (self, split_res:Tuple[str, ...]=('[\\s,]', '[^_\\w]'), num_elements:Union[int, NoneType]=None, convert:Callable=<class 'str'>) -> List[List[Any]]

ConfigurationEntry.as_path()

Full name: midgard.config.config.as_path

Signature: (self) -> pathlib.Path

Value of ConfigurationEntry interpreted as a path string

ConfigurationEntry.as_str()

Full name: midgard.config.config.as_str

Signature: (self) -> str

Value of ConfigurationEntry as string

ConfigurationEntry.as_tuple()

Full name: midgard.config.config.as_tuple

Signature: (self, split_re:str='[\\s,]', convert:Callable=<class 'str'>, maxsplit:int=0) -> Tuple[Any, ...]

Value of ConfigurationEntry converted to a tuple

The entry is converted to a tuple by using the split_re-regular expression. By default the entry will be split at commas and whitespace.

Args:

Returns:

Value of entry as tuple.

ConfigurationEntry.entry_as_str()

Full name: midgard.config.config.entry_as_str

Signature: (self, width:Union[int, NoneType]=None, key_width:int=30, metadata:bool=True) -> str

The configuration entry represented as a string

This is simililar to what is shown by str(entry) (and implemented by __str__), but has more flexibility.

Args:

Returns:

String representation of the configuration entry.

ConfigurationEntry.replace()

Full name: midgard.config.config.replace

Signature: (self, default:Union[str, NoneType]=None, **replace_vars:str) -> 'ConfigurationEntry'

ConfigurationSection

Full name: midgard.config.config.ConfigurationSection

Signature: (name:str) -> None

ConfigurationSection.as_dict()

Full name: midgard.config.config.as_dict

Signature: (self, getters:Dict[str, str]=None, default_getter:str='str') -> Dict[str, Any]

The configuration section represented as a dictionary

Args:

Returns:

Representation of the configuration section as a dictionary.

ConfigurationSection.as_list()

Full name: midgard.config.config.as_list

Signature: (self) -> List[str]

List of keys of entries in configuration section

Returns:

List of keys of entries in configuration section.

ConfigurationSection.as_str()

Full name: midgard.config.config.as_str

Signature: (self, width:Union[int, NoneType]=None, key_width:int=30, only_used:bool=False, metadata:bool=True) -> str

The configuration section represented as a string

This is simililar to what is shown by str(section) (and implemented by __str__), but has more flexibility.

Args:

Returns:

String representation of the configuration section.

ConfigurationSection.exists()

Full name: midgard.config.config.exists

Signature: (self, key:str) -> bool

Check if key exists in section

Args:

Returns:

True if key is in section, False otherwise.

FMT_date (str)

FMT_date = '%Y-%m-%d'

FMT_datetime (str)

FMT_datetime = '%Y-%m-%d %H:%M:%S'

FMT_dt_file (str)

FMT_dt_file = '%Y%m%d-%H%M%S'

midgard.config.files

Midgard library module for opening files based on a special configuration

Example:

from midgard.config import files
with files.open('eopc04_iau', mode='rt') as fid:
    for line in fid:
        print(line.strip())

Description:

This module handles opening of files registered in a special configuration, typically a configuration file.

The cfg.files.open and cfg.files.open_path methods are both wrappers around the built-in open function, and behave mainly similar. In particular, they accept all the same keyword arguments (like for instance mode). Furthermore, to make sure files are properly closed they should normally be used with a context manager as in the example above.

FileConfiguration

Full name: midgard.config.files.FileConfiguration

Signature: (name:str) -> None

Configuration for handling files

FileConfiguration.download_file()

Full name: midgard.config.files.download_file

Signature: (self, file_key:str, file_vars:Union[Dict[str, str], NoneType]=None, file_path:Union[pathlib.Path, NoneType]=None, create_dirs:bool=True, **path_args:Any) -> Union[pathlib.Path, NoneType]

Download a file from the web and save it to disk

Use pycurl (libcurl) to do the actual downloading. Requests might be nicer for this, but turned out to be much slower (and in practice unusable for bigger files) and also not really supporting ftp-downloads.

Args:

Returns:

Path to downloaded file, None if no file was downloaded.

FileConfiguration.download_missing (bool)

download_missing = True

FileConfiguration.encoding()

Full name: midgard.config.files.encoding

Signature: (self, file_key)

Look up the encoding for a given file key

Args:

file_key (String): Key that is looked up in the configuration.

Returns:

FileConfiguration.glob_paths()

Full name: midgard.config.files.glob_paths

Signature: (self, file_key:str, file_vars:Union[Dict[str, str], NoneType]=None, is_zipped:Union[bool, NoneType]=None) -> List[pathlib.Path]

Find all filepaths matching a filename pattern

Using pathlib.Path.glob() here is not trivial because we need to split into a base directory to start searching from and a pattern which may include directories. With glob.glob() this is trivial. The downside is that it only returns strings and not pathlib.Paths.

FileConfiguration.glob_variable()

Full name: midgard.config.files.glob_variable

Signature: (self, file_key, variable, pattern, file_vars=None)

Find all possible values of variable

FileConfiguration.is_path_zipped()

Full name: midgard.config.files.is_path_zipped

Signature: (file_path)

Indicate whether a path is to a gzipped file or not

For now, this simply checks whether the path ends in .gz or not.

Args:

file_path (Path): Path to a file.

Returns:

FileConfiguration.open()

Full name: midgard.config.files.open

Signature: (self, file_key:str, file_vars:Dict[str, str]=None, create_dirs:bool=False, is_zipped:Union[bool, NoneType]=None, download_missing:bool=True, use_aliases:bool=True, logger:Union[Callable, NoneType]=None, **open_args:Any) -> Iterator

Open a file based on information in a configuration

Open a file based on file key which is looked up in the configuration.

The method automatically handles reading from gzipped files if the filename is specified with the special {gz}-ending (including the curly braces) in the file list. In that case, the mode should be specified to be 'rt' if the contents of the file should be treated as text. If both a zipped and an unzipped version is available, the zipped version is used. This can be overridden by specifying True or False for the is_zipped-parameter.

This function behaves similar to the built-in open-function, and should typically be used with a context manager as follows:

Example:

with cfg.open('eopc04_iau', mode='rt') as fid:
    for line in fid:
        print(line.strip())

Args:

Returns:

File object representing the file.

FileConfiguration.path()

Full name: midgard.config.files.path

Signature: (self, file_key:str, file_vars:Union[Dict[str, str], NoneType]=None, default:Union[str, NoneType]=None, is_zipped:Union[bool, NoneType]=None, download_missing:bool=False, use_aliases:bool=True) -> pathlib.Path

Construct a filepath for a given file with variables

If is_zipped is None, and the file_path contains <filename>{gz}, the file will be assumed to be a gzip-file if there exists a file named <filename>.gz.

When setting use_aliases to True, the aliases as specified in the files configuration file represent alternative filenames. In particular:

- if directory / file_name exists it is returned
- otherwise the first directory / alias that exists is returned
- if none of these exist, directory / file_name is returned

Args:

file_key (String): Key that is looked up in the configuration. file_vars (Dict): Values used to replace variables in file name and path. default (String): Value to use for variables that are not in file_vars. is_zipped (Bool/None): True, False or None. If True, open with gzip. If None automatically decide. download_missing (Bool): Whether to try to download missing files. use_aliases (Bool): Fall back on aliases if file does not exist.

Return: Path: Full path with replaced variables in file name and path.

FileConfiguration.url()

Full name: midgard.config.files.url

Signature: (self, file_key, file_vars=None, default=None, is_zipped=None, use_aliases=False)

Construct a URL for a given file with variables

If is_zipped is None, and the url contains <filename>{gz}, the url will be assumed to point to a gzip-file if there exists a file named <filename>.gz on the server.

Args:

Return: String: Full URL with replaced variables in file name and url.