Configuration Loaders¶
The history of config files:
— David Capello (@davidcapello) May 19, 2020
.ini: maybe we need a little more
.xml: ok, this is too much
.json: ok, now we need comments back
.yaml: I'm not sure about this python approach
.toml: back to .ini
Loaders are in charge of loading configuration from various sources, like
.ini
files or environment variables, and expose configuration as a
dict-like object. Loaders are ment to be chained, so that classyconf checks
one by one for a given configuration variable.
If a loader doesn’t find the configuration variable it raises a KeyError
so that the next loader get’s checked. If no loader returns any value, and no
default value was set, an
UnknownConfiguration
exception is thrown.
Classyconf comes with some loaders already included in
classyconf.loaders
.
By default the library will check the environment with the
Environment
loader. You can
change that behaviour, by customizing the loaders and the order in wich
configuration discovery happens.
Loaders can be set in the Meta
class when extending
Configuration
or passed as a
param when instantiating it. The later takes precedence and overrides loaders
defined in Meta
. The order within the list of loaders matters and defines
the lookup order.
from classyconf import Configuration, IniFile, Environment, Value, EnvFile
class AppConfig(Configuration):
class Meta:
loaders = [
Environment(),
IniFile("/path/to/config.ini")
]
DEBUG = Value(default=False)
OTHER_CONFIG = Value(default=0)
# Checks for enviroment variables first
# Then lookup settings in the `config.ini` file
config = AppConfig()
# Override loaders and only lookup in the `.env` file
test_config = AppConfig(loaders=[EnvFile("/path/to/.env")])
Naming conventions and namespaces for settings¶
There happen to be some formatting conventions for configuration parameters based on where they are set. For example, it is common to name environment variables in uppercase:
$ DEBUG=yes OTHER_CONFIG=10 ./app.py
Since the environment is a global and shared dictionary, it is a good
practice to also apply some prefix to each setting to avoid collisions with
other known settings, like LOCALE
, TZ
, etc. This prefix works as a
namespace for your app.
$ MY_APP_DEBUG=yes MY_APP_OTHER_CONFIG=10 ./app.py
but if you were to set this config in an .ini
file, each setting should
probably be in lower case, the namespace is implicit in the file path, i.e:
/etc/myapp/config.ini
.
[settings]
debug=yes
other_config=10
Command line arguments have yet another conventions:
$ ./app.py --debug=yes --another-config=10
Classyconf let’s you follow these aesthetics patterns by setting a
keyfmt
function when instantiating the loaders.
By default, the Environment
is
instantiated with keyfmt=EnvPrefix('')
so that it looks for
UPPER_CASED settings. But it can be easyly tweaked to address the prefix
issue by using keyfmt=EnvPrefix("MY_APP_")
, and look for
MY_APP_UPPER_CASED to play nice with other env variables.
from classyconf import Configuration, IniFile, Environment, Value, EnvPrefix
class AppConfig(Configuration):
class Meta:
loaders = [
Environment(keyfmt=EnvPrefix(prefix="MY_APP_")),
IniFile("/etc/myapp/config.ini")
]
DEBUG = Value(default=False)
OTHER_CONFIG = Value(default=0)
config = AppConfig()
# looks for `MY_APP_DEBUG` in environment, then `debug` in `settings` section of config.ini
config.DEBUG
Keep reading to find out more about different loaders and their configurations.
Environment¶
-
class
classyconf.loaders.
Environment
(keyfmt=EnvPrefix(""))¶ Get’s configuration from the environment, by inspecting
os.environ
.Parameters: keyfmt (function) – A function to pre-format variable names.
The Environment
loader gets configuration from os.environ
. Since it
is a common pattern to write env variables in caps, the loader accepts a
keyfmt
function to pre-format the variable name before the lookup
occurs. By default it is EnvPrefix("")
which combines str.upper()
and an empty prefix.
Note
In the case of CLI apps, it would be recommended to set some sort of
namespace so that you don’t accidentally override other programs
behaviour, like LOCALE or EDITOR, but instead MY_APP_LOCALE, etc. So
consider using the EnvPrefix("MY_APP_")
approach.
from classyconf import Configuration, Environment, Value
class AppConf(Configuration):
debug = Value(default=False)
config = AppConf(loaders=[Environment(keyfmt=str.upper)])
config.debug # will look for a `DEBUG` variable
EnvFile¶
-
class
classyconf.loaders.
EnvFile
(filename='.env', keyfmt=EnvPrefix(""))¶ Parameters: - filename (str) – Path to the
.env
file. - keyfmt (function) – A function to pre-format variable names.
- filename (str) – Path to the
The EnvFile
loader gets configuration from .env
file. If the file
doesn’t exist, this loader will be skipped without raising any errors.
# .env file
DEBUG=1
from classyconf import Configuration, EnvFile, Value
class AppConf(Configuration):
debug = Value(default=False)
config = AppConf(loaders=[EnvFile(file='.env', keyfmt=str.upper)])
config.debug # will look for a `DEBUG` variable instead of `debug`
Note
You might want to use dump-env, a utility to create .env
files.
IniFile¶
-
class
classyconf.loaders.
IniFile
(filename, section='settings', keyfmt=<function IniFile.<lambda>>)¶ Parameters: - filename (str) – Path to the
.ini/.cfg
file. - section (str) – Section name inside the config file.
- keyfmt (function) – A function to pre-format variable names.
- filename (str) – Path to the
The IniFile
loader gets configuration from .ini
or .cfg
files. If
the file doesn’t exist, this loader will be skipped without raising any errors.
CommandLine¶
-
class
classyconf.loaders.
CommandLine
(parser, get_args=<function get_args>)¶ Extract configuration from an
argparse
parser.Parameters: - parser (argparse.ArgumentParser) – An argparse parser instance to extract variables from.
- get_args (function) – A function to extract args from the parser.
This loader lets you extract configuration variables from parsed command line arguments. By default it works with argparse parsers.
import argparse
from classyconf import Configuration, Value, NOT_SET, CommandLine
parser = argparse.ArgumentParser(description='Does something useful.')
parser.add_argument('--debug', '-d', dest='debug', default=NOT_SET, help='set debug mode')
class AppConf(Configuration):
DEBUG = Value(default=False)
config = AppConf(loaders=[CommandLine(parser=parser)])
print(config.DEBUG)
Something to notice here is the
NOT_SET
value. CLI parsers often
force you to put a default value so that they don’t fail. In that case, to
play nice with classyconf, you must set one. But that would break the
discoverability chain that classyconf encourages. So by setting this special
default value, you will allow classyconf to keep the lookup going.
The get_args
function converts the
argparse parser’s values to a dict that ignores
NOT_SET
values.
Dict¶
-
class
classyconf.loaders.
Dict
(values_mapping)¶ Parameters: values_mapping (dict) – A dictionary of hardcoded settings.
This loader is great when you want to pin certain settings without having to
change/override other loaders, files or defaults. It really comes handy when
you are extending a
Configuration
class.
from classyconf import Configuration, Value, IniFile, Dict
class AppConfig(Configuration):
class Meta:
loaders = [IniFile("/opt/myapp/config.ini"), IniFile("/etc/myapp/config.ini")]
NUMBER = Value(default=1)
DEBUG = Value(default=False)
LABEL = Value(default="foo")
OTHER = Value(default="bar")
class TestConfig(AppConfig):
class Meta:
loders = [Dict({"DEBUG": True, "NUMBER": 0})]
RecursiveSearch¶
-
class
classyconf.loaders.
RecursiveSearch
(starting_path=None, filetypes=(('.env', <class 'classyconf.loaders.EnvFile'>), (('*.ini', '*.cfg'), <class 'classyconf.loaders.IniFile'>)), root_path='/')¶ Parameters: - starting_path (str) – The path to begin looking for configuration files.
- filetypes (tuple) – tuple of tuples with configuration loaders, order matters.
Defaults to
(('*.env', EnvFile), (('*.ini', *.cfg',), IniFile)
- root_path (str) – Configuration lookup will stop at the given path. Defaults to the current user directory
This loader tries to find .env
or *.ini|*.cfg
files and load them
with the EnvFile
and
IniFile
loaders respectively.
It will start looking at the starting_path
directory for configuration
files and walking up the filesystem tree until it finds any or reaches the
root_path
.
Warning
It is important to note that this loader uses the glob module internally to
discover .env
and *.ini|*.cfg
files. This could be problematic if
the project includes many files that are unrelated, like a pytest.ini
file along side with a settings.ini
. An unexpected file could be found
and be considered as the configuration to use.
Consider the following file structure:
project/
settings.ini
app/
settings.py
When instantiating your
RecursiveSearch
, if you pass
/absolute/path/to/project/app/
as starting_path
the loader will start
looking for configuration files at project/app
.
# Code example in project/app/settings.py
import os
from classyconf import config
from classyconf.loaders import RecursiveSearch
app_path = os.path.dirname(__file__)
config.loaders = [RecursiveSearch(starting_path=app_path)]
By default, the loader will try to look for configuration files until it finds
valid configuration files or it reaches root_path
. The root_path
is
set to the root directory /
initialy.
Suppose the following file structure:
projects/
any_settings.ini
project/
app/
settings.py
You can change this behaviour by setting any parent directory of the
starting_path
as the root_path
when instantiating
RecursiveSearch
:
# Code example in project/app/settings.py
import os
from classyconf import Configuration
from classyconf.loaders import RecursiveSearch
app_path = os.path.dirname(__file__)
project_path = os.path.realpath(os.path.join(app_path, '..'))
rs = RecursiveSearch(starting_path=app_path, root_path=project_path)
config = Configuration(loaders=[rs])
The example above will start looking for files at project/app/
and will stop looking
for configuration files at project/
, actually never looking at any_settings.ini
and no configuration being loaded at all.
The root_path
must be a parent directory of starting_path
, otherwise
it raises an InvalidPath
exception:
from classyconf.loaders import RecursiveSearch
# /baz is not parent of /foo/bar, so this raises an InvalidPath exception here
rs = RecursiveSearch(starting_path="/foo/bar", root_path="/baz")
Writing your own loader¶
If you need a custom loader, you should just extend the
AbstractConfigurationLoader
.
-
class
classyconf.loaders.
AbstractConfigurationLoader
¶
For example, say you want to write a Yaml loader. It is important to note
that by raising a KeyError
exception from the loader, classyconf knows
that it has to keep looking down the loaders chain for a specific config.
import yaml
from classyconf.loaders import AbstractConfigurationLoader
class YamlFile(AbstractConfigurationLoader):
def __init__(self, filename, keyfmt=str.lower):
self.filename = filename
self.config = None
self.keyfmt = keyfmt
def _parse(self):
if self.config is not None:
return
with open(self.filename, 'r') as f:
self.config = yaml.load(f)
def __contains__(self, item):
try:
self._parse()
except:
return False
return self.keyfmt(item) in self.config
def __getitem__(self, item):
try:
self._parse()
except:
# KeyError tells classyconf to keep looking elsewhere!
raise KeyError("{!r}".format(item))
return self.config[self.keyfmt(item)]
def reset(self):
self.config = None
Then configure classyconf to use it.
from classyconf import Configuration
class AppConf(Configuration):
class Meta:
loaders = [YamlFile('/path/to/config.yml')]