Add PEP 391: logging configuration using dictionaries.

pep-0391.txt

@@ -0,0 +1,461 @@
+PEP: 391
+Title: Dictionary-Based Configuration For Logging
+Version: $Revision$
+Last-Modified: $Date$
+Author: Vinay Sajip <vinay_sajip at red-dove.com>
+Status: Draft
+Type: Standards Track
+Content-Type: text/x-rst
+Created: 15-Oct-2009
+Python-Version: 2.7, 3.2
+Post-History:
+
+
+Abstract
+========
+
+This PEP describes a new way of configuring logging using a dictionary to hold
+configuration information.
+
+Rationale
+=========
+
+The present means for configuring Python's logging package is either by using
+the logging API to configure logging programmatically, or else by means of
+ConfigParser-based configuration files.
+
+Programmatic configuration, while offering maximal control, fixes the
+configuration in Python code.  This does not facilitate changing it easily at
+runtime, and, as a result, the ability to flexibly turn the verbosity of
+logging up and down for different parts of a using application is lost.  This
+limits the usability of logging as an aid to diagnosing problems - and
+sometimes, logging is the only diagnostic aid available in production
+environments.
+
+The ConfigParser-based configuration system is usable, but does not allow its
+users to configure all aspects of the logging package.  For example, Filters
+cannot be configured using this system.  Furthermore, the ConfigParser format
+appears to engender dislike (sometimes strong dislike) in some quarters.
+Though it was chosen because it was the only configuration format supported in
+the Python standard at that time, many people regard it (or perhaps just the
+particular schema chosen for logging's configuration) as 'crufty' or 'ugly',
+in some cases apparently on purely aesthetic grounds.
+
+Recent versions of Python inlude JSON support in the standard library, and
+this is also usable as a configuration format.  In other environments, such as
+Google App Engine, YAML is used to configure applications, and usually the
+configuration of logging would be considered an integral part of the
+application configuration.  Although the standard library does not contain
+YAML support at present, support for both JSON and YAML can be provided in a
+common way because both of these serialization formats allow deserialization
+of Python dictionaries.
+
+By providing a way to configure logging by passing the configuration in a
+dictionary, logging will be easier to configure not only for users of JSON
+and/or YAML, but also for users of bespoke configuration methods, by providing
+a common format in which to describe the desired configuration.
+
+Another drawback of the current ConfigParser-based configuration system is
+that it does not support incremental configuration: a new configuration
+completely replaces the existing configuration.  Although full flexibility for
+incremental configuration is difficult to provide in a multi-threaded
+environment, the new configuration mechanism will allow the provision of
+limited support for incremental configuration.
+
+Specification
+=============
+
+The specification consists of two parts: the API and the format of the
+dictionary used to convey configuration information (i.e. the schema to which
+it must conform).
+
+Naming
+------
+
+Historically, the logging package has not been PEP-8 conformant.  At some
+future time, this will be corrected by changing method and function names in
+the package in order to conform with PEP-8.  However, in the interests of
+uniformity, the proposed additions to the API use a naming scheme which is
+consistent with the present scheme used by logging.
+
+API
+---
+
+The logging.config module will have the following additions:
+
+* A class, called ``DictConfigurator``, whose constructor is passed the
+  dictionary used for configuration, and which has a ``configure()`` method.
+
+* A callable, called ``dictConfigClass``, which will (by default) be set to
+  ``DictConfigurator``.  This is provided so that if desired,
+  ``DictConfigurator`` can be replaced with a suitable user-defined
+  implementation.
+
+* A function, called ``dictConfig()``, which takes a single argument - the
+  dictionary holding the configuration.  This function will call
+  ``dictConfigClass`` passing the specified dictionary, and then call the
+  ``configure()`` method on the returned object to actually put the 
+  configuration into effect::
+  
+    def dictConfig(config):
+        dictConfigClass(config).configure()
+
+Dictionary Schema - Overview
+----------------------------
+
+Before describing the schema in detail, it is worth saying a few words about
+object connections, support for user-defined objects and access to external
+objects.
+
+Object connections
+''''''''''''''''''
+
+The schema is intended to describe a set of logging objects - loggers,
+handlers, formatters, filters - which are connected to each other in an
+object graph.  Thus, the schema needs to represent connections between the
+objects.  For example, say that, once configured, a particular logger has an
+attached to it a particular handler.  For the purposes of this discussion,
+we can say that the logger represents the source, and the handler the
+destination, of a connection between the two.  Of course in the configured
+objects this is represented by the logger holding a reference to the
+handler.  In the configuration dict, this is done by giving each destination
+object an id which identifies it unambiguously, and then using the id in the
+source object's configuration to indicate that a connection exists between
+the source and the destination object with that id.
+  
+So, for example, consider the following YAML snippet::
+  
+  handlers:
+    h1: #This is an id
+     # configuration of handler with id h1 goes here
+    h2: #This is another id
+     # configuration of handler with id h2 goes here
+  loggers:
+    foo.bar.baz:
+      # other configuration for logger "foo.bar.baz"
+      handlers: [h1, h2]
+
+(Note: YAML will be used in this document as it is more readable than the
+equivalent Python source form for the dictionary.)
+  
+The ids for loggers are the logger names which would be used
+programmatically to obtain a reference to those loggers, e.g.
+``foo.bar.baz``.  The ids for other objects can be any string value (such as
+``h1``, ``h2`` above) and they are transient, in that they are only
+meaningful for processing the configuration dictionary and used to
+determine connections between objects, and are not persisted anywhere when
+the configuration call is complete.
+  
+The above snippet indicates that logger named ``foo.bar.baz`` should have
+two handlers attached to it, which are described by the handler ids ``h1``
+and ``h2``.
+
+User-defined objects
+''''''''''''''''''''
+
+The schema should support user-defined objects for handlers, filters and
+formatters.  (Loggers do not need to have different types for different
+instances, so there is no support - in the configuration - for user-defined
+logger classes.)
+
+Objects to be configured will typically be described by dictionaries which
+detail their configuration.  In some places, the logging system will be able
+to infer from the context how an object is to be instantiated, but when a
+user-defined object is to be instantiated, the system will not know how to do
+this.  In order to provide complete flexibility for user-defined object
+instantiation, the user will need to provide a 'factory' - a callable which
+is called with a configuration dictionary and which returns the instantiated
+object.  This will be signalled by the factory being made available under
+the special key ``'()'``.  Here's a concrete example::
+  
+  formatters:
+    brief:
+      format: '%(message)s'
+    default:
+      format: '%(asctime)s %(levelname)-8s %(name)-15s %(message)s'
+      datefmt: '%Y-%m-%d %H:%M:%S'
+    custom:
+        (): my.package.customFormatterFactory
+        bar: baz
+        spam: 99.9
+        answer: 42
+
+The above YAML snippet defines three formatters.  The first, with id
+``brief``, is a standard ``logging.Formatter`` instance with the
+specified format string.  The second, with id ``default``, has a longer
+format and also defines the time format explicitly, and will result in a
+``logging.Formatter`` initialized with those two format strings.  Shown in
+Python source form, the ``brief`` and ``default`` formatters have
+have configuration sub-dictionaries::
+  
+  {
+    'format' : '%(message)s'
+  }
+    
+and::
+  
+  {
+    'format' : '%(asctime)s %(levelname)-8s %(name)-15s %(message)s',
+    'datefmt' : '%Y-%m-%d %H:%M:%S'
+  }
+  
+respectively, and as these dictionaries do not contain the special key
+``'()'``, the instantiation is inferred from the context: as a result,
+standard ``logging.Formatter`` instances are created.  The configuration
+sub-dictionary for the third formatter, with id ``custom``, is::
+
+  {
+    '()' : 'my.package.customFormatterFactory',
+    'bar' : 'baz',
+    'spam' : 99.9,
+    'answer' : 42
+  }
+  
+and this contains the special key ``'()'``, which means that user-defined
+instantiation is wanted.  In this case, the specified factory callable will be
+located using normal import mechanisms and called with the *remaining* items
+in the configuration sub-dictionary as keyword arguments.  In the above
+example, the formatter with id ``custom`` will be assumed to be returned by
+the call::
+
+  my.package.customFormatterFactory(bar="baz", spam=99.9, answer=42)  
+
+The key ``'()'`` has been used as the special key because it is not a valid
+keyword parameter name, and so will not clash with the names of the keyword
+arguments used in the call.  The ``'()'`` also serves as a mnemonic that the
+corresponding value is a callable.
+
+Access to external objects
+''''''''''''''''''''''''''
+
+There are times where a configuration will need to refer to objects external
+to the configuration, for example ``sys.stderr``.  If the configuration dict
+is constructed using Python code then this is straightforward, but a problem
+arises when the configuration is provided via a text file (e.g. JSON, YAML).
+In a text file, there is no standard way to distinguish ``sys.stderr`` from
+the literal string ``'sys.stderr'``.  To facilitate this distinction, the
+configuration system will look for certain special prefixes in string values
+and treat them specially. For example, if the literal string
+``'ext://sys.stderr'`` is provided as a value in the configuration, then the
+``ext://`` will be stripped off and the remainder of the value processed using
+normal import mechanisms.
+
+The handling of such prefixes will be done in a way analogous to protocol
+handling: there will be a generic mechanism to look for prefixes which match
+the regular expression ``^(?P<prefix>[a-z]+)://(?P<suffix>.*)$`` whereby, if
+the ``prefix`` is recognised, the ``suffix`` is processed in a
+prefix-dependent manner and the result of the processing replaces the string
+value.  If the prefix is not recognised, then the string value will be left
+as-is.
+
+The implementation will provide for a set of standard prefixes such as
+``ext://`` but it will be possible to disable the mechanism completely or
+provide additional or different prefixes for special handling.
+
+Dictionary Schema - Detail
+--------------------------
+
+The dictionary passed to ``dictConfig()`` must contain the following keys:
+
+* `version` - to be set to an integer value representing the schema
+  version.  The only valid value at present is 1, but having this key allows
+  the schema to evolve while still preserving backwards compatibility.
+
+All other keys are optional, but if present they will be interpreted as described
+below.  In all cases below where a 'configuring dict' is mentioned, it will be
+checked for the special ``'()'`` key to see if a custom instantiation is
+required.  If so, the mechanism described above is used to instantiate;
+otherwise, the context is used to determine how to instantiate.
+
+* `formatters` - the corresponding value will be a dict in which each key is
+  a formatter id and each value is a dict describing how to configure the
+  corresponding Formatter instance.
+  
+  The configuring dict is searched for keys ``format`` and ``datefmt`` (with
+  defaults of ``None``) and these are used to construct a
+  ``logging.Formatter`` instance.
+
+* `filters` - the corresponding value will be a dict in which each key is
+  a filter id and each value is a dict describing how to configure the
+  corresponding Filter instance.
+
+  The configuring dict is searched for key ``name`` (defaulting to the empty
+  string) and this is used to construct a ``logging.Filter`` instance.
+
+* `handlers` - the corresponding value will be a dict in which each key is
+  a handler id and each value is a dict describing how to configure the
+  corresponding Handler instance.
+
+  The configuring dict is searched for the following keys:
+  
+  * ``class`` (mandatory).  This is the fully qualified name of the handler
+    class.
+    
+  * ``level`` (optional).  The level of the handler.
+  
+  * ``formatter`` (optional).  The id of the formatter for this handler.
+  
+  * ``filters`` (optional).  A list of ids of the filters for this handler.
+
+  All *other* keys are passed through as keyword arguments to the handler's
+  constructor.  For example, given the snippet::
+
+    handlers:
+      console:
+        class : logging.StreamHandler
+        formatter: brief
+        level   : INFO
+        filters: [allow_foo]
+        stream  : ext://sys.stdout
+      file:
+        class : logging.handlers.RotatingFileHandler
+        formatter: precise
+        filename: logconfig.log
+        maxBytes: 1024
+        backupCount: 3
+
+  the handler with id ``console`` is instantiated as a
+  ``logging.StreamHandler``, using ``sys.stdout`` as the underlying stream.
+  The handler with id ``file`` is instantiated as a
+  ``logging.handlers.RotatingFileHandler`` with the keyword arguments
+  ``filename="logconfig.log", maxBytes=1024, backupCount=3``.
+
+* `loggers` - the corresponding value will be a dict in which each key is
+  a logger name and each value is a dict describing how to configure the
+  corresponding Logger instance.
+
+  The configuring dict is searched for the following keys:
+  
+  * ``level`` (optional).  The level of the logger.
+  
+  * ``propagate`` (optional).  The propagation setting of the logger.
+  
+  * ``filters`` (optional).  A list of ids of the filters for this logger.
+
+  * ``handlers`` (optional).  A list of ids of the handlers for this logger.
+
+  The specified loggers will be configured according to the level,
+  propagation, filters and handlers specified.
+
+* `root` - this will be the configuration for the root logger. Processing of
+  the configuration will be as for any logger, except that the ``propagate``
+  setting will not be applicable.
+  
+* `incremental` - whether the configuration is to be interpreted as
+  incremental to the existing configuration. This value defaults to False,
+  which means that the specified configuration replaces the existing
+  configuration with the same semantics as used by the existing
+  ``fileConfig()`` API.
+  
+  If the specified value is True, the configuration is processed as described
+  in the section on "Incremental Configuration", below.
+
+A Working Example
+-----------------
+
+The following is an actual working configuration in YAML format (except that
+the email addresses are bogus)::
+
+    formatters:
+      brief:
+        format: '%(levelname)-8s: %(name)-15s: %(message)s'
+      precise:
+        format: '%(asctime)s %(name)-15s %(levelname)-8s %(message)s'
+    filters:
+      allow_foo:
+        name: foo
+    handlers:
+      console:
+        class : logging.StreamHandler
+        formatter: brief
+        level   : INFO
+        stream  : ext://sys.stdout
+        filters: [allow_foo]
+      file:
+        class : logging.handlers.RotatingFileHandler
+        formatter: precise
+        filename: logconfig.log
+        maxBytes: 1024
+        backupCount: 3
+      debugfile:
+        class : logging.FileHandler
+        formatter: precise
+        filename: logconfig-detail.log
+        mode: a
+      email:
+        class: logging.handlers.SMTPHandler
+        mailhost: localhost
+        fromaddr: my_app@domain.tld
+        toaddrs:
+          - support_team@domain.tld
+          - dev_team@domain.tld
+        subject: Houston, we have a problem.
+    loggers:
+      foo:
+        level : ERROR
+        handlers: [debugfile]
+      spam:
+        level : CRITICAL
+        handlers: [debugfile]
+        propagate: no
+      bar.baz:
+        level: WARNING
+    root:
+      level     : DEBUG
+      handlers  : [console, file]
+
+Incremental Configuration
+=========================
+
+It is difficult to provide complete flexibility for incremental configuration.
+For example, because objects such as handlers, filters and formatters are
+anonymous, once a configuration is set up, it is not possible to refer to such
+anonymous objects when augmenting a configuration. For example, if an initial
+call is made to configure the system where logger ``foo`` has a handler with
+id ``console`` attached, then a subsequent call to configure a logger ``bar``
+with id ``console`` would create a new handler instance, as the id ``console``
+from the first call isn't kept.
+
+Furthermore, there is not a compelling case for arbitrarily altering the
+object graph of loggers, handlers, filters, formatters at run-time, once a 
+configuration is set up; the verbosity of loggers can be controlled just by
+setting levels (and perhaps propagation flags).
+
+Thus, when the ``incremental`` key of a configuration dict is present and
+is ``True``, the system will ignore the ``formatters``, ``filters``,
+``handlers`` entries completely, and process only the ``level`` and
+``propagate`` settings in the ``loggers`` and ``root`` entries.
+
+Configuration Errors
+====================
+
+If an error is encountered during configuration, the system will raise a
+``ValueError`` or a ``TypeError`` with a suitably descriptive message. The
+following is a (possibly incomplete) list of conditions which will raise an
+error:
+
+* A ``level`` which is not a string or which is a string not corresponding to
+  an actual logging level
+
+* A ``propagate`` value which is not a Boolean
+
+* An id which does not have a corresponding destination
+
+* An invalid logger name
+
+Copyright
+=========
+
+This document has been placed in the public domain.
+
+
+
+..
+   Local Variables:
+   mode: indented-text
+   indent-tabs-mode: nil
+   sentence-end-double-space: t
+   fill-column: 70
+   coding: utf-8
+   End:
+