Configuration.load_yaml() - Code Metrics - Inspection of "Merge pull request #345 from dmitriy-serdyuk/fix-w..." - mila-udem/fuel - Measure and Improve Code Quality continuously with Scrutinizer

Completed

Push — master ( 3e1d4c...f31f72 )

by Bart

created 2016-04-13 21:38 UTC

Configuration.load_yaml() B

↳ Parent: Project

Complexity

Conditions

Size

Total Lines

Duplication

Lines	0
Ratio	0 %

Metric	Value
cc	6
dl	0
loc	12
rs	8

"""Module level configuration.

Fuel allows module-wide configuration values to be set using a YAML_
configuration file and `environment variables`_. Environment variables
override the configuration file which in its turn overrides the defaults.

The configuration is read from ``~/.fuelrc`` if it exists. A custom
configuration file can be used by setting the ``FUEL_CONFIG`` environment
variable. A configuration file is of the form:

.. code-block:: yaml

   data_path: /home/user/datasets

Which could be overwritten by using environment variables:

.. code-block:: bash

   $ FUEL_DATA_PATH=/home/users/other_datasets python

This data path is a sequence of paths separated by an os-specific
delimiter (':' for Linux and OSX, ';' for Windows).

If a setting is not configured and does not provide a default, a
:class:`~.ConfigurationError` is raised when it is
accessed.

Configuration values can be accessed as attributes of
:const:`fuel.config`.

    >>> from fuel import config
    >>> print(config.data_path) # doctest: +SKIP
    '~/datasets'

The following configurations are supported:

.. option:: data_path

   The path where dataset files are stored. Can also be set using the
   environment variable ``FUEL_DATA_PATH``. Expected to be a sequence
   of paths separated by an os-specific delimiter (':' for Linux and
   OSX, ';' for Windows).


.. todo::

   Implement this.

.. option:: floatX

   The default :class:`~numpy.dtype` to use for floating point numbers. The
   default value is ``float64``. A lower value can save memory.

.. option:: extra_downloaders

   A list of package names which, like fuel.downloaders, define an
   `all_downloaders` attribute listing available downloaders. By default,
   an empty list.

.. option:: extra_converters

   A list of package names which, like fuel.converters, define an
   `all_converters` attribute listing available converters. By default,
   an empty list.

.. _YAML: http://yaml.org/
.. _environment variables:
   https://en.wikipedia.org/wiki/Environment_variable

"""
import logging
import os

import six
import yaml

from .exceptions import ConfigurationError

logger = logging.getLogger(__name__)

NOT_SET = object()


def extra_downloader_converter(value):
    """Parses extra_{downloader,converter} arguments.

    Parameters
    ----------
    value : iterable or str
        If the value is a string, it is split into a list using spaces
        as delimiters. Otherwise, it is returned as is.

    """
    if isinstance(value, six.string_types):
        value = value.split(" ")
    return value


def multiple_paths_parser(value):
    """Parses data_path argument.

    Parameters
    ----------
    value : str
        a string of data paths separated by  ":".

    Returns
    -------
    value : list
        a list of strings indicating each data paths.

    """
    if isinstance(value, six.string_types):
        value = value.split(os.path.pathsep)
    return value


class Configuration(object):
    def __init__(self):
        self.config = {}

    def load_yaml(self):
        if 'FUEL_CONFIG' in os.environ:
            yaml_file = os.environ['FUEL_CONFIG']
        else:
            yaml_file = os.path.expanduser('~/.fuelrc')
        if os.path.isfile(yaml_file):
            with open(yaml_file) as f:
                for key, value in yaml.safe_load(f).items():
                    if key not in self.config:
                        raise ValueError("Unrecognized config in YAML: {}"
                                         .format(key))
                    self.config[key]['yaml'] = value

    def __getattr__(self, key):
        if key == 'config' or key not in self.config:
            raise AttributeError
        config_setting = self.config[key]
        if 'value' in config_setting:
            value = config_setting['value']
        elif ('env_var' in config_setting and
              config_setting['env_var'] in os.environ):
            value = os.environ[config_setting['env_var']]
        elif 'yaml' in config_setting:
            value = config_setting['yaml']
        elif 'default' in config_setting:
            value = config_setting['default']
        else:
            raise ConfigurationError("Configuration not set and no default "
                                     "provided: {}.".format(key))
        return config_setting['type'](value)

    def __setattr__(self, key, value):
        if key != 'config' and key in self.config:
            self.config[key]['value'] = value
        else:
            super(Configuration, self).__setattr__(key, value)

    def add_config(self, key, type_, default=NOT_SET, env_var=None):
        """Add a configuration setting.

        Parameters
        ----------
        key : str
            The name of the configuration setting. This must be a valid
            Python attribute name i.e. alphanumeric with underscores.
        type : function
            A function such as ``float``, ``int`` or ``str`` which takes
            the configuration value and returns an object of the correct
            type.  Note that the values retrieved from environment
            variables are always strings, while those retrieved from the
            YAML file might already be parsed. Hence, the function provided
            here must accept both types of input.
        default : object, optional
            The default configuration to return if not set. By default none
            is set and an error is raised instead.
        env_var : str, optional
            The environment variable name that holds this configuration
            value. If not given, this configuration can only be set in the
            YAML configuration file.

        """
        self.config[key] = {'type': type_}
        if env_var is not None:
            self.config[key]['env_var'] = env_var
        if default is not NOT_SET:
            self.config[key]['default'] = default

config = Configuration()

# Define configuration options
config.add_config('data_path', type_=multiple_paths_parser,
                  env_var='FUEL_DATA_PATH')
config.add_config('default_seed', type_=int, default=1)
config.add_config('extra_downloaders', type_=extra_downloader_converter,
                  default=[], env_var='FUEL_EXTRA_DOWNLOADERS')
config.add_config('extra_converters', type_=extra_downloader_converter,
                  default=[], env_var='FUEL_EXTRA_CONVERTERS')

# Default to Theano's floatX if possible
try:
    from theano import config as theano_config
    default_floatX = theano_config.floatX

except Exception:

    default_floatX = 'float64'

config.add_config('floatX', type_=str, env_var='FUEL_FLOATX',
                  default=default_floatX)

config.load_yaml()


1			"""Module level configuration.
2
3			Fuel allows module-wide configuration values to be set using a YAML_
4			configuration file and `environment variables`_. Environment variables
5			override the configuration file which in its turn overrides the defaults.
6
7			The configuration is read from ``~/.fuelrc`` if it exists. A custom
8			configuration file can be used by setting the ``FUEL_CONFIG`` environment
9			variable. A configuration file is of the form:
10
11			.. code-block:: yaml
12
13			data_path: /home/user/datasets
14
15			Which could be overwritten by using environment variables:
16
17			.. code-block:: bash
18
19			$ FUEL_DATA_PATH=/home/users/other_datasets python
20
21			This data path is a sequence of paths separated by an os-specific
22			delimiter (':' for Linux and OSX, ';' for Windows).
23
24			If a setting is not configured and does not provide a default, a
25			:class:`~.ConfigurationError` is raised when it is
26			accessed.
27
28			Configuration values can be accessed as attributes of
29			:const:`fuel.config`.
30
31			>>> from fuel import config
32			>>> print(config.data_path) # doctest: +SKIP
33			'~/datasets'
34
35			The following configurations are supported:
36
37			.. option:: data_path
38
39			The path where dataset files are stored. Can also be set using the
40			environment variable ``FUEL_DATA_PATH``. Expected to be a sequence
41			of paths separated by an os-specific delimiter (':' for Linux and
42			OSX, ';' for Windows).
43
44
45			.. todo::
46
47			Implement this.
48
49			.. option:: floatX
50
51			The default :class:`~numpy.dtype` to use for floating point numbers. The
52			default value is ``float64``. A lower value can save memory.
53
54			.. option:: extra_downloaders
55
56			A list of package names which, like fuel.downloaders, define an
57			`all_downloaders` attribute listing available downloaders. By default,
58			an empty list.
59
60			.. option:: extra_converters
61
62			A list of package names which, like fuel.converters, define an
63			`all_converters` attribute listing available converters. By default,
64			an empty list.
65
66			.. _YAML: http://yaml.org/
67			.. _environment variables:
68			https://en.wikipedia.org/wiki/Environment_variable
69
70			"""
71			import logging
72			import os
73
74			import six
75			import yaml
76
77			from .exceptions import ConfigurationError
78
79			logger = logging.getLogger(__name__)
80
81			NOT_SET = object()
82
83
84			def extra_downloader_converter(value):
85			"""Parses extra_{downloader,converter} arguments.
86
87			Parameters
88			----------
89			value : iterable or str
90			If the value is a string, it is split into a list using spaces
91			as delimiters. Otherwise, it is returned as is.
92
93			"""
94			if isinstance(value, six.string_types):
95			value = value.split(" ")
96			return value
97
98
99			def multiple_paths_parser(value):
100			"""Parses data_path argument.
101
102			Parameters
103			----------
104			value : str
105			a string of data paths separated by ":".
106
107			Returns
108			-------
109			value : list
110			a list of strings indicating each data paths.
111
112			"""
113			if isinstance(value, six.string_types):
114			value = value.split(os.path.pathsep)
115			return value
116
117
118			class Configuration(object):
119			def __init__(self):
120			self.config = {}
121
122			def load_yaml(self):
123			if 'FUEL_CONFIG' in os.environ:
124			yaml_file = os.environ['FUEL_CONFIG']
125			else:
126			yaml_file = os.path.expanduser('~/.fuelrc')
127			if os.path.isfile(yaml_file):
128			with open(yaml_file) as f:
129			for key, value in yaml.safe_load(f).items():
130			if key not in self.config:
131			raise ValueError("Unrecognized config in YAML: {}"
132			.format(key))
133			self.config[key]['yaml'] = value
134
135			def __getattr__(self, key):
136			if key == 'config' or key not in self.config:
137			raise AttributeError
138			config_setting = self.config[key]
139			if 'value' in config_setting:
140			value = config_setting['value']
141			elif ('env_var' in config_setting and
142			config_setting['env_var'] in os.environ):
143			value = os.environ[config_setting['env_var']]
144			elif 'yaml' in config_setting:
145			value = config_setting['yaml']
146			elif 'default' in config_setting:
147			value = config_setting['default']
148			else:
149			raise ConfigurationError("Configuration not set and no default "
150			"provided: {}.".format(key))
151			return config_setting['type'](value)
152
153			def __setattr__(self, key, value):
154			if key != 'config' and key in self.config:
155			self.config[key]['value'] = value
156			else:
157			super(Configuration, self).__setattr__(key, value)
158
159			def add_config(self, key, type_, default=NOT_SET, env_var=None):
160			"""Add a configuration setting.
161
162			Parameters
163			----------
164			key : str
165			The name of the configuration setting. This must be a valid
166			Python attribute name i.e. alphanumeric with underscores.
167			type : function
168			A function such as ``float``, ``int`` or ``str`` which takes
169			the configuration value and returns an object of the correct
170			type. Note that the values retrieved from environment
171			variables are always strings, while those retrieved from the
172			YAML file might already be parsed. Hence, the function provided
173			here must accept both types of input.
174			default : object, optional
175			The default configuration to return if not set. By default none
176			is set and an error is raised instead.
177			env_var : str, optional
178			The environment variable name that holds this configuration
179			value. If not given, this configuration can only be set in the
180			YAML configuration file.
181
182			"""
183			self.config[key] = {'type': type_}
184			if env_var is not None:
185			self.config[key]['env_var'] = env_var
186			if default is not NOT_SET:
187			self.config[key]['default'] = default
188
189			config = Configuration()
190
191			# Define configuration options
192			config.add_config('data_path', type_=multiple_paths_parser,
193			env_var='FUEL_DATA_PATH')
194			config.add_config('default_seed', type_=int, default=1)
195			config.add_config('extra_downloaders', type_=extra_downloader_converter,
196			default=[], env_var='FUEL_EXTRA_DOWNLOADERS')
197			config.add_config('extra_converters', type_=extra_downloader_converter,
198			default=[], env_var='FUEL_EXTRA_CONVERTERS')
199
200			# Default to Theano's floatX if possible
201			try:
202			from theano import config as theano_config
203			default_floatX = theano_config.floatX
			0 ignored issues – show Coding Style Naming introduced 2015-12-02 16:30 UTC by Report Bug Copy Issue Report The name `default_floatX` does not conform to the constant naming conventions (`(([A-Z_][A-Z0-9_])\|(__.__))$`). This check looks for invalid names for a range of different identifiers. You can set regular expressions to which the identifiers must conform if the defaults do not match your requirements. If your project includes a Pylint configuration file, the settings contained in that file take precedence. To find out more about Pylint, please refer to their site. Loading history...
204			except Exception:
			0 ignored issues – show Best Practice introduced 2015-12-02 16:30 UTC by Report Bug Copy Issue Report Catching very general exceptions such as `Exception` is usually not recommended. Generally, you would want to handle very specific errors in the exception handler. This ensure that you do not hide other types of errors which should be fixed. So, unless you specifically plan to handle any error, consider adding a more specific exception. Loading history...
205			default_floatX = 'float64'
			0 ignored issues – show Coding Style Naming introduced 2015-12-02 16:30 UTC by Report Bug Copy Issue Report The name `default_floatX` does not conform to the constant naming conventions (`(([A-Z_][A-Z0-9_])\|(__.__))$`). This check looks for invalid names for a range of different identifiers. You can set regular expressions to which the identifiers must conform if the defaults do not match your requirements. If your project includes a Pylint configuration file, the settings contained in that file take precedence. To find out more about Pylint, please refer to their site. Loading history...
206			config.add_config('floatX', type_=str, env_var='FUEL_FLOATX',
207			default=default_floatX)
208
209			config.load_yaml()
210

mila-udem / fuel

Push — master ( 3e1d4c...f31f72 )

Configuration.load_yaml() B

Complexity

Size

Duplication

Duplication Side-by-Side

Filter issues like