resconfig¶

A minimalistic application configuration library for Python.

User’s Guide¶

Introduction¶

resconfig is a minimalistic application configuration library for Python. It is a thin wrapper around nested dict objects with added features that make it easy to deal with the data structure as a centralized storage of application configuration.

ResConfig supports

multiple configuration file formats: INI, JSON, TOML, and YAML;
environment variables: Configuration can be easily overridden with environment variables;
command-line arguments: Configuration can be easily overridden with ArgumentParser command-line arguments.
“.”-delimited nested keys: config["foo.bar"] is equivalent to config["foo"]["bar"].

The advanced usage of ResConfig allows:

Dynamic reloading of configuration at run time: Watch functions can be attached to any keys within the configuration to trigger actions to manage resources.

Installation¶

resconfig is available at PyPI. To install using pip:

$ pip install resconfig

If you wish to use TOML and/or YAML files for configuration, install extra dependencies:

$ pip install resconfig[toml]  # TOML support
$ pip install resconfig[yaml]  # YAML support

To install from source, download the code from GitHub:

$ git clone git@github.com:okomestudio/resconfig.git
$ cd resconfig
$ pip install .[toml,yaml]  # install both TOML and YAML support
$ pip install -e .[dev]  # for development

Quickstart¶

This quickstart provides a short introduction on how to get started with ResConfig. Read Installation first, if you have not installed the resconfig package.

Let us first create an ResConfig object with a simple default configuration for your application, myapp.py:

from resconfig ResConfig

config = ResConfig({"db": {"host": "localhost", "port": 5432}})

By default, ResConfig loads configuration immediately after its initialization. To control the timing of load, use the load_on_init flag:

config = ResConfig({"db": {"host": "localhost", "port": 5432}},
                   load_on_init=False)
config.load()

The following sections introduce you to the basic usage of ResConfig object.

The “.”-Style Key Notation¶

ResConfig exposes dict-like interface for value access but additionally allows the “.”-style notation for nested keys. The following methods all return the same value, localhost:

host = config["db"]["host"]
host = config["db.host"]
host = config.get("db.host")  # similar to dict.get

The “.”-style can be used elsewhere, e.g.,

config = ResConfig({"db.host": "localhost", "db.port": 5432})

This will be the same default configuration shown earlier. ResConfig takes care of nesting the dict for you.

Use with Configuration Files¶

To read configuration from (multiple) files, supply a list of paths on object initialization:

config = ResConfig({"db.host": "localhost", "db.port": 5432},
                   config_files=["myconf.yml",
                                 "~/.myconf.yml,
                                 "/etc/myconf.yml"])

If any of the files exists, they are read in the reverse order, i.e., /etc/myconf.yml, ~/.myconf.yml, and then myconf.yml, and the configuration read from them get merged in that order, overriding the default. This allows layered configuration based on specificity by filesystem location. Read Configuration from Files for more detail.

Use with Environment Variables¶

Properly named environment variables can override default configuration. When you run your myapp.py app with the DB_HOST and/or DB_PORT environment variables set, their values override the default:

$ DB_HOST=remotehost DB_PORT=3306 python myapp.py

That is, config["db.host"] and config["db.port"] will return remotehost and 3306, respectively. As a rule of thumb, a configuration key maps to an uppercased, “_”-delimited (when nested) environment variable name. Read Environment Variables for more detail.

Use with ArgumentParser¶

A ResConfig object can dynamically generate ArgumentParser arguments from default configuration:

parser = argparse.ArgumentParser()
parser.add_argument(...)  # Define other arguments

config.add_arguments_to_argparse(parser)
# --pg-host and --pg-port arguments are now available

After actually parsing the (command-line) arguments, pass the parse result to ResConfig and then load the configuration:

args = parser.parse_args()
config.prepare_from_argparse(args)
config.load()

For more detail, read ArgumentParser Integration.

Adding Actions on Changes¶

A ResConfig object is aware of changes to its configuration. Watch functions watch changes happening at any nested key to act on them:

from resconfig import Action

@config.watch("db.host")
def act_on_nested_key(action, old, new):
    if action == Action.ADDED:
        # db.host added
    elif action == Action.MODIFIED:
        # db.host modified
    elif action == Action.RELOADED:
        # db.host reloaded
    elif action == Action.REMOVED:
        # db.host removed

Here, the act_on_nested_key() function is called whenever configuration changes occur at db.host and can decide what to do with the old and/or new values. For more detail, read Watch Functions.

The “.”-Style Key Notation¶

To access nested item, e.g.,

config = ResConfig({"hosts": {"localhost": {"user": "John"}}})

you may use the “.”-style notation, so that

print(config["hosts.localhost.user"])

will print John.

Similarly, the same style can be used when supplying a dict-like object to ResConfig:

config = ResConfig({"hosts.localhost.user": "John"})

This will set exactly the same default configuration as the one above.

Note

Question: What if the key includes one of more “.” characters? It is quite common for us to see configuration with IP addresses like this:

config = ResConfig(
    {"hosts": {"127.0.0.1": {"user": "John"}}}
)

Will this configuration be butchers into a mess like this?

{"hosts": {"127": {"0": {"0": {"1": {"user": "John"}}}}}}

Is this the end of the world?

Answer: Indeed, if you write the nested key like this,

config = ResConfig({"hosts.127.0.0.1.user": "John"})

the mess you mentioned will ensue. In order to avoid period to be interpreted as delimiter, you will need to escape them, e.g.,

config = ResConfig({"hosts.127\.0\.0\.1.user": "John"})

This way, they will not be interpret as nested keys:

{"hosts": {"127\.0\.0\.1": {"user": "John"}}}

Unfortunately, they need to be escaped all the time. For this reason, it is strongly recommended to use “.” only as the delimiter for nested keys, not as part of a key. While the end may not be near, this makes the world a bit messier place to be.

Configuration from Files¶

ResConfig understands INI (.ini), JSON (.json), TOML (.toml), and YAML (.yaml or .yml). If not given, the file format is inferred from these filename extensions. The filename with no extension is assumed to be of INI.

Merge Behavior¶

When multiple files are supplied, ResConfig handles them in two different ways, depending on the merge_config_files switch. When True, all existing files are read, but merging will be in the reverse of the input file list. For example, in the following case,

config = ResConfig(config_files=["myconf.yml",
                                 "~/.myconf.yml,
                                 "/etc/myconf.yml"],
                   merge_config_files=True)

the configurations read from these files are merged in the following order: /etc/myconf.yml, ~/.myconf.yml, and myconf.yml. This effectively allows overriding configuration based on environment, i.e., personal configuration has a higher precedence to the system configuration in this example, typical of UNIX-like systems.

When False, only the first existing file will be read. For example, suppose that ~/.myconf.yml and /etc/myconf.yml exist but not myconf.yml. Then

config = ResConfig(config_files=["myconf.yml",
                                 "~/.myconf.yml,
                                 "/etc/myconf.yml"],
                   merge_config_files=False)

will read only from ~/.myconf.yml. ResCongif skips myconf.yml and ignores /etc/myconf.yml.

The default behavior is merge_config_files=True.

In general, non-existing files are simply skipped without throwing errors.

The “~” character in file paths will be expanded to the path defined in the HOME environment variable.

File Types¶

ResConfig understands the following file formats:

INI (.ini)
JSON (.json)
TOML (.toml)
YAML (.yaml or .yml)

with the given filename extensions.

By default, a filename without extension is assumed to be of INI type. If you wish to explicitly specify file type, you may do so using a ConfigPath subclass as follows:

from resconfig.io.paths import YAMLPath

config = ResConfig(config_files=[YAMLPath("myconf"),
                                 YAMLPath("~/.myconf),
                                 YAMLPath("/etc/myconf")])

This way, all the files are considered to be of YAML type, regardless of extension. ResConfig supplies INIPath, JSONPath, TOMLPath, and YAMLPath for this purpose.

Environment Variables¶

Often, being able to override configuration with environment variables is desirable. ResConfig by default looks for environment variables that map to configuration keys in a simple way, converting “.”-delimited configuration keys to “_”-delimited, uppercase environment variable names.

For example, the configuration key db.host will be mapped to DB_HOST.

To avoid conflicts with variable names used for other purposes, pprefix can be used. If you want MYAPP_ to be your prefix, supply it as the envvar_prefix option to ResConfig:

config = ResConfig({"db.host": "localhost", "db.port": 5432},
                   envvar_prefix="MYAPP_")

Then, the MYAPP_DB_HOST and MYAPP_DB_PORT will map to db.host and db.port configuration keys.

Note

The escaped “.” in keys will map to a “_” character for environment variable names.

ArgumentParser Integration¶

argparse is a standard library tool to add command-line argument parsing to your application. ResConfig makes it easy to add command-line arguments to set configuration values.

In order to respect command-line arguments, the configuration needs to be loaded after the ArgumentParser object completes its parsing. By default, ResConfig loads the configuration immediately after the initialization of itself. You can delay this by setting the load_on_init flag to False and load it yourself at an appropriate timing.

Dynamic Argument Generation¶

ResConfig object can generate and add command-line arguments to ArgumentParser object from the default configuration by the add_arguments_to_argparse() method.

Once ArgumentParser object does its parsing, the result should be passed to the prepare_from_argparse() method to prepare the configuration from the parse result. At that point, the configuration is good to be loaded.

In summary, the following is the standard procedure:

config = ResConfig({"db.host": "localhost", "db.port": 5432},
                   load_on_init=False)

parser = argparse.ArgumentParser()
parser.add_argument(...)  # Define other arguments
config.add_arguments_to_argparse(parser)

args = parser.parse_args()
config.prepare_from_argparse(args)
config.load()

In this case, add_arguments_to_argparse() adds --db-host and --db-port as command-line arguments.

As a rule of thumb, a nested key maps to a “-”-delimited long argument. To avoid conflict with other options, the prefix option can supply a custom prefix:

config.add_arguments_to_argparse(parser, prefix="myapp")

With this, add_arguments_to_argparse() adds --myapp-db-host and --myapp-db-port as command-line arguments.

If you want certain items from the default to be skipped, provide their keys as a set in ignore:

config.add_arguments_to_argparse(parser, ignore={"db.port"})

This way, --db-host will be added to the parser but not --db-port.

Reading from Argument¶

You may also manually define arguments and let prepare_from_argparse() automatically pick them up by naming pattern, e.g.,

config = ResConfig({"db.host": "localhost", "db.port": 5432},
                   load_on_init=False)

parser = argparse.ArgumentParser()
parser.add_argument(...)  # Define other arguments
parser.add_argument("--db-host", default="localhost")
parser.add_argument("--db-port", default=5432)

args = parser.parse_args()
config.prepare_from_argparse(args)
config.load()

Here, --db-host and --db-port are mapped to the db.host and db.port keys in configuration.

If you used a common prefix, use the prefix option to supply it:

parser.add_argument("--myapp-db-host", default="localhost")
parser.add_argument("--myapp-db-port", default=5432)
...
config.prepare_from_argparse(args, prefix="myapp")

Or if you want a more generic mapping that does not match expected pattern, use the keymap option to supply the mapping:

parser.add_argument("--hostname", default="localhost")
parser.add_argument("--portnumber", default=5432)
...
config.prepare_from_argparse(args,
                             keymap={"hostname": "db.host",
                                     "portnumber": "db.port"})

Often you want to supply configuration files as command line argument. Indicate the argument for configuration file as the config_file_arg option:

parser.add_argument("--db-host", default="localhost")
parser.add_argument("--db-port", default=5432)
parser.add_argument("--conf", action="append")
...
config.prepare_from_argparse(args, config_file_arg="conf")

The multiple files are handled just as config_files in ResConfig, but with a higher precedence over those given at the initialization; see Configuration from Files for detail of multi-file configuration.

Note

The escaped “.” in keys will map to a “.” character for command-line argument by the add_arguments_to_argparse() method. For example, if the config key is 127\.0\.0\.1, then the corresponding long option will be --127.0.0.1 (and vice versa in prepare_from_argparse()).

Watch Functions¶

The ResConfig object is aware of changes to its configuration. Watch functions can be registered to watch changes happening at any nested key to act on them.

The following example shows how a watch function, named manage_db_host(), can be triggered on changes happening for the db.host item in configuration:

import signal
from resconfig import Action, ResConfig

config = ResConfig(load_on_init=False)

@config.watch("db.host")
def manage_db_host(action, old, new):
    if action == Action.ADDED:
        # Initialize database connection?
    elif action == Action.MODIFIED:
        # Change database connection?
    elif action == Action.RELOADED:
        # Refresh database connection?
    elif action == Action.REMOVED:
        # Clean up on database connection?

def reload(signum=None, stack_frame=None):
    config.reload()

signal.signal(signal.SIGHUP, reload)  # run reload on SIGHUP

config.load()

Here, the manage_db_host() function is called whenever a change occurs at the db.host item in the configuration and can decide what to do with the old and/or new values based on the actual Action. In this example, the configuration reload function is a handler for SIGHUP and is triggered when the process receives the signal, for example, via kill -SIGHUP <pid> where <pid> is the application process ID.

The watch function can be registered in a few different ways: watch(), register(), and the watchers argument to the ResConfig initializer. There are no differences among the styles in terms of functionality. The decorator style:

@config.watch("db.host")
def watch_function(action, old, new):
    ...

The method style:

def watch_function(action, old, new):
    ...

config.register("db.host", watch_function)

The initial argument style:

def watch_function(action, old, new):
    ...

config = ResConfig(watchers={"db.host": watch_function})

Config Precedence¶

With several ways to set configuration, the precedence matters when multiple methods are used simultaneously. ResConfig resolves conflicts based on the following order, with the earlier methods in the list take precedence when defined:

Command-line arguments.
Configuration files specified as command-line arguments.
Environment variables.
Configuration files supplied to ResConfig initializer.
Default configuration supplied to ResConfig initializer.

As an example, consider the following code, which we call myapp.py:

config = ResConfig({"db.host": "localhost", "db.port": 5432},
                   config_files=["~/.myapp.conf",
                                 "/etc/myapp.conf"],
                   load_on_init=False)

parser = argparse.ArgumentParser()
parser.add_argument("--conf")
config.add_arguments_to_argparse(parser)

args = parser.parse_args()
config.prepare_from_argparse(args, config_file_arg="conf")
config.load()

and the following files:

myapp.conf:

[db]
host = local.org

~/.myapp.conf:

[db]
host = home.org

(Let us assume that /etc/myapp.conf does not exist.)

What would conf["db.host"] return? The answer depends on how myapp.py is started.

$ DB_HOST=env.org python myapp.py --conf=myapp.conf \
                               --db-host=cmdline.org

$ DB_HOST=env.org python myapp.py --conf=myapp.conf \

```
$ DB_HOST=env.org python myapp.py
```
```
$ python myapp.py
```

$ python myapp.py  # with ~/.myapp.conf not found

Answers: (a) cmdline.org, (b) local.org, (c) env.org, (d) home.org, (e) localhost.

Notes¶

Changelog¶

Unreleased¶

Added:

Documentation for using ConfigPath objects to specify file types (#20)

Fixed:

Reading from an empty JSON/YAML file results in load error (2fc94fa, 1346f3f).

20.4.3a (April 19, 2020)¶

Added:

Allow file formats to be hinted by wrapping as ConfigPath objects (#20).
The config precedence page in documentation (#13).
Logging on #17:
- config value setting by environment variable and command-line argument.
- watch function de/registration.

Changed:

Move changelog to the repo root and include it into docs.

Fixed:

None values from CL arg default passed through (447adc1).
Environment variable overridden by command-line argument if the latter does not default to None (#21).

20.4.2a (April 18, 2020)¶

Initial alpha version.

API Reference¶

API¶

ResConfig Object¶

class resconfig.ResConfig(default=None, config_files=None, envvar_prefix='', load_on_init=True, merge_config_files=True, watchers=None)¶

Bases: resconfig.watchers.Watchable, resconfig.io.io.IO, resconfig.clargs.CLArgs

An application resource configuration.

This object holds the state and the internal data structure for the configuration.

Parameters

default (Optional[dict]) – Default config.
config_files (Optional[List[Union[str, Path]]]) – List of config filename paths.
envvar_prefix (str) – Prefix used for environment variables used as configuration.
load_on_init (bool) – True to load config on instantiation, False to skip.
merge_config_files (bool) – True to merge all configs from existing files, False to read only the config from the first existing file.
watchers (Optional[Dict[Union[str, Tuple[str]], List[Callable[[Action, Any, Any], None]]]]) – Config watchers.

add_arguments_to_argparse(parser, prefix='', ignore=None)¶

Add config arguments to ArgumentParser.

The method add to the parser arguments that exist in the default config supplied on the instantiation of ResConfig object. The long option will be “-”-delimited for nested keys, e.g., config["foo.bar"] will be mapped to --foo-bar. If prefix is supplied, the prefix is prepended to the long option, e.g., --prefix-foo-bar.

To prevent the method from adding arguments to the parser for some keys, supply ignore with a set of keys to ignore, e.g., {"foo.bar"}, so that --foo-bar will not be added.

Parameters

parser (ArgumentParser) – Parser object to which config arguments are added.
prefix (str) – Argument prefix.
ignore (Optional[Set[str]]) – Config keys to ignore.

deregister(key, func=None)¶: Deregister the watch function for the key.

get(key, default=None)¶

Return the config value for key if it exists, else default.

Parameters

key (Union[str, Tuple[str]]) – Config key.
default (Optional[Any]) – Default value if key is not in config.

Return type

Any

Returns

The value found for the key.

load()¶: Load the prepared config.

prepare_from_argparse(args, config_file_arg=None, prefix='', keymap=None)¶

Prepare config from ArgumentParser result.

See the docstring of add_arguments_to_argparse() for the rule on how parser long options map to config keys. If long options do not directly map to config keys by that rule, you can supply prefix or keymap to define your own mapping. For example, if you want to map --long-opt (which would be parsed as the long_opt attribute) to the foo.bar config key, use {"long_opt": "foo.bar"} for keymap.

If you want to allow config file(s) to be specified through ArgumentParser, specify the attribute name for the argument to config_file_arg.

Parameters

args (Namespace) – Parse result returned from parse_args().
config_file_arg (Optional[str]) – The attribute name for config files.
prefix (str) – Argument prefix.
keymap (Optional[Dict[str, str]]) – Key mapping from the parsed argument name to the config key.

register(key, func)¶: Register the watch function for the key.

reload()¶

Trigger all watch functions using the current configuration.

Note that the watch functions for the keys that do not exist in the current configuration will not be triggered.

replace(conf)¶

Perform replacement of config.

Parameters: conf (dict) – Config for replacement.

reset()¶: Reset config to default.

save_to_file(filename)¶

Experimental: Save config to the file.

The file type is inferred from the filename extension.

save_to_ini(filename)¶: Experimental: Save config to the INI file.

save_to_json(filename)¶: Experimental: Save config to the JSON file.

save_to_toml(filename)¶: Experimental: Save config to the TOML file.

save_to_yaml(filename)¶: Experimental: Save config to the YAML file.

unload()¶: Empty the configuration.

update(conf)¶

Perform update of config.

Parameters: conf (dict) – Config to update with.

update_from_file(filename)¶

Update config from the file.

The file type is inferred from the filename extension.

update_from_files(paths, merge=False)¶

Update config from files.

How the config is constructed depends on the merge flag. If True, the files are searched in the reverse order, and the configs are read from all existing files and merged in that order. If False, then the files are searched for in the order given in paths, and the first existing file provides the config to be read (and the rest are ignored).

Parameters

paths (List[Union[str, Path]]) – A list of config file paths.
merge – The flag for the merge mode; see the function description.

update_from_ini(filename)¶: Update config from the INI file.

update_from_json(filename)¶: Update config from the JSON file.

update_from_toml(filename)¶: Update config from the TOML file.

update_from_yaml(filename)¶: Update config from the YAML file.

watch(key)¶

Decorate a function to make it a watch function for the key.

Return type: Callable[[Action, Any, Any], None]

ConfigPath Object¶

class resconfig.io.paths.ConfigPath(*args, **kwargs)¶

Bases: pathlib.PosixPath

Path for configuration file.

class resconfig.io.paths.INIPath(*args, **kwargs)¶

Bases: resconfig.io.paths.ConfigPath

Wrapper for INI config file path.

class resconfig.io.paths.JSONPath(*args, **kwargs)¶

Bases: resconfig.io.paths.ConfigPath

Wrapper for JSON file path.

class resconfig.io.paths.TOMLPath(*args, **kwargs)¶

Bases: resconfig.io.paths.ConfigPath

Wrapper for TOML file path.

class resconfig.io.paths.YAMLPath(*args, **kwargs)¶

Bases: resconfig.io.paths.ConfigPath

Wrapper for YAML file path.

Actions¶

class resconfig.actions.Action(value)¶

Action performed by the update method.

The watch function receives this value as its first argument to change its behavior based on the action.

ADDED = 1¶: The item at the key has been added.

MODIFIED = 2¶: The item at the key has been modified.

RELOADED = 4¶: The item at the key has been reloaded.

REMOVED = 3¶: The item at the key has been removed.

Developer API¶

ONDict Object¶

class resconfig.ondict.ONDict(*args, **kwargs)¶

Bases: collections.OrderedDict

allkeys(as_str=False)¶: Generate all keys to the leaves

asdict()¶

Get a built-in dict representation of itself.

All the nested mappings are converted into dict.

Return type: dict
Returns: Built-in dict object.

clear() → None. Remove all items from od.¶

copy() → a shallow copy of od¶

classmethod fromkeys(iterable, value=None)¶: Create a new ordered dictionary with keys from iterable and values set to value.

get(key, default=None)¶: Return the value for key if key is in the dictionary, else default.

items() → a set-like object providing a view on D’s items¶

keys() → a set-like object providing a view on D’s keys¶

merge(*args, **kwargs)¶

Merge from dict and/or iterable.

This method takes in the same argument(s) as dict.update(), but merge the input instead of dict-like update. Merging extends the update behavior to allow nested updates and to support nested key notation.

Raises

TypeError – When more than one positional argument is supplied.
ValueError – When the iterable does not hold two-element items.

move_to_end(key, last=True)¶

Move an existing element to the end (or beginning if last is false).

Raise KeyError if the element does not exist.

pop(k[, d]) → v, remove specified key and return the corresponding¶: value. If key is not found, d is returned if given, otherwise KeyError is raised.

popitem(last=True)¶

Remove and return a (key, value) pair from the dictionary.

Pairs are returned in LIFO order if last is true or FIFO order if false.

setdefault(key, default=None)¶

Insert key with a value of default if key is not in the dictionary.

Return the value for key if key is in the dictionary, else default.

update(*args, **kwargs)¶

Update from dict and/or iterable.

This method takes in the same argument(s) as dict.update(). Compared to the built-in dict object, the update behavior is expanded to allow nested key notation.

Note that update happens only on the top-level keys, just like built-in dict, to supply consistent behavior. If you desire a full merging behavior, use ONDict.merge().

Raises

TypeError – When more than one positional argument is supplied.
ValueError – When the iterable does not hold two-element items.

values() → an object providing a view on D’s values¶

resconfig¶

User’s Guide¶

Introduction¶

Installation¶

Quickstart¶

The “.”-Style Key Notation¶

Use with Configuration Files¶

Use with Environment Variables¶

Use with ArgumentParser¶

Adding Actions on Changes¶

The “.”-Style Key Notation¶

Configuration from Files¶

Merge Behavior¶

File Types¶

Environment Variables¶

ArgumentParser Integration¶

Dynamic Argument Generation¶

Reading from Argument¶

Watch Functions¶

Config Precedence¶

Notes¶

Changelog¶

Unreleased¶

20.4.3a (April 19, 2020)¶

20.4.2a (April 18, 2020)¶

API Reference¶

API¶

ResConfig Object¶

ConfigPath Object¶

Actions¶

Developer API¶

ONDict Object¶

Indices and tables¶

resconfig

Navigation

Related Topics