ocrd_utils.logging.initLogging() - Code Metrics - Inspection of "ocrd_utils.initLogging: also add handler to root l..." - OCR-D/core - Measure and Improve Code Quality continuously with Scrutinizer

Passed

Pull Request — master (#1288)

unknown

created 2024-10-09 16:53 UTC

ocrd_utils.logging.initLogging() F

↳ Parent: ocrd_utils.logging

Complexity

Conditions

Size

Total Lines	59
Code Lines	31

Duplication

Lines	0
Ratio	0 %

Importance

Changes

Metric	Value
eloc	31
dl	0
loc	59
rs	3.6
c	0
b	0
f	0
cc	14
nop	3

How to fix Long Method Complexity

"""
Logging setup

By default: Log with lastResort logger, usually STDERR.

Logging can be overridden either programmatically in code using the library or by creating one or more of

- /etc/ocrd_logging.py
- $HOME/ocrd_logging.py
- $PWD/ocrd_logging.py

These files will be executed in the context of ocrd/ocrd_logging.py, with `logging` global set.

Changes as of 2023-08-20:

    - Try to be less intrusive with OCR-D specific logging conventions to
      make it easier and less surprising to define logging behavior when
      using OCR-D/core as a library
    - Change setOverrideLogLevel to only override the log level of the ``ocrd``
      logger and its descendants
    - initLogging will set exactly one handler, for the root logger or for the
      ``ocrd`` logger.
    - Child loggers should propagate to the ancestor logging (default
      behavior of the logging library - no more PropagationShyLogger)
    - disableLogging only removes any handlers from the ``ocrd`` logger
"""
# pylint: disable=no-member

from __future__ import absolute_import

from traceback import format_stack

import logging
import logging.config
from pathlib import Path
import sys

from .constants import LOG_FORMAT, LOG_TIMEFMT
from .config import config


__all__ = [
    'disableLogging',
    'getLevelName',
    'getLogger',
    'initLogging',
    'logging',
    'setOverrideLogLevel',
]

# These are the loggers we add handlers to
ROOT_OCRD_LOGGERS = [
    '',
    'ocrd',
    'ocrd_network'
]

LOGGING_DEFAULTS = {
    'ocrd': logging.INFO,
    'ocrd_network': logging.INFO,
    # 'ocrd.resolver': logging.INFO,
    # 'ocrd.resolver.download_to_directory': logging.INFO,
    # 'ocrd.resolver.add_files_to_mets': logging.INFO,
    # To cut back on the `Self-intersection at or near point` INFO messages
    'shapely.geos': logging.ERROR,
    'tensorflow': logging.ERROR,
    'PIL': logging.INFO,
    'paramiko.transport': logging.INFO,
    'uvicorn.access': logging.DEBUG,
    'uvicorn.error': logging.DEBUG,
    'uvicorn': logging.INFO,
    'multipart': logging.INFO,
}

_initialized_flag = False

_ocrdLevel2pythonLevel = {
    'TRACE': 'DEBUG',
    'OFF': 'CRITICAL',
    'FATAL': 'ERROR',
}

def tf_disable_interactive_logs():
    try:
        from os import environ
        # This env variable must be set before importing from Keras
        environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
        from tensorflow.keras.utils import disable_interactive_logging
        # Enabled interactive logging throws an exception
        # due to a call of sys.stdout.flush()
        disable_interactive_logging()
    except ImportError:
        # Nothing should be handled here if TF is not available
        pass

def getLevelName(lvl):
    """
    Get (string) python logging level for (string) spec-defined log level name.
    """
    lvl = _ocrdLevel2pythonLevel.get(lvl, lvl)
    return logging.getLevelName(lvl)

def getLogger(*args, **kwargs):
    """
    Wrapper around ``logging.getLogger`` that calls :py:func:`initLogging` if
    that wasn't explicitly called before.
    """
    logger = logging.getLogger(*args, **kwargs)
    return logger

def setOverrideLogLevel(lvl, silent=not config.OCRD_LOGGING_DEBUG):
    """
    Override the output log level of the handlers attached to the ``ocrd`` logger.

    Args:
        lvl (string): Log level name.
        silent (boolean): Whether to log the override call
    """
    if not _initialized_flag:
        initLogging(silent=silent)
    ocrd_logger = logging.getLogger('ocrd')

    if lvl is None:
        if not silent:
            print('[LOGGING] Reset log level override', file=sys.stderr)
        ocrd_logger.setLevel(logging.NOTSET)
    else:
        if not silent:
            print(f'[LOGGING] Overriding ocrd log level to {lvl}', file=sys.stderr)
        ocrd_logger.setLevel(lvl)

def get_logging_config_files():
    """
    Return a list of all ``ocrd_logging.conf`` files found in CWD, HOME or /etc.
    """
    CONFIG_PATHS = [
        Path.cwd(),
        Path.home(),
        Path('/etc'),
    ]
    return [f for f \
            in [p / 'ocrd_logging.conf' for p in CONFIG_PATHS] \
            if f.exists()]

def initLogging(builtin_only=False, force_reinit=False, silent=not config.OCRD_LOGGING_DEBUG):
    """
    Reset ``ocrd`` logger, read logging configuration if exists, otherwise use basicConfig

    initLogging is to be called by OCR-D/core once, i.e.
        -  for the ``ocrd`` CLI
        -  for the processor wrapper methods

    Other processes that use OCR-D/core as a library can, but do not have to, use this functionality.

    Keyword Args:
        - builtin_only (bool, False): Whether to search for logging configuration
                                      on-disk (``False``) or only use the
                                      hard-coded config (``True``). For testing
        - force_reinit (bool, False): Whether to ignore the module-level
                                      ``_initialized_flag``. For testing only.
        - silent (bool, True): Whether to log logging behavior by printing to stderr
    """
    global _initialized_flag
    if _initialized_flag and not force_reinit:
        return
    # disableLogging()

    # https://docs.python.org/3/library/logging.html#logging.disable
    # If logging.disable(logging.NOTSET) is called, it effectively removes this
    # overriding level, so that logging output again depends on the effective
    # levels of individual loggers.
    logging.disable(logging.NOTSET)

    # remove all handlers for the ocrd root loggers
    for logger_name in ROOT_OCRD_LOGGERS:
        for handler in logging.getLogger(logger_name).handlers[:]:
            logging.getLogger(logger_name).removeHandler(handler)

    config_file = None
    if not builtin_only:
        config_file = get_logging_config_files()
    if config_file:
        if len(config_file) > 1 and not silent:
            print(f"[LOGGING] Multiple logging configuration files found at {config_file}, using first one", file=sys.stderr)
        config_file = config_file[0]
        if not silent:
            print(f"[LOGGING] Picked up logging config at {config_file}", file=sys.stderr)
        logging.config.fileConfig(config_file)
    else:
        if not silent:
            print("[LOGGING] Initializing logging with built-in defaults", file=sys.stderr)
        # Default logging config
        ocrd_handler = logging.StreamHandler(stream=sys.stderr)
        ocrd_handler.setFormatter(logging.Formatter(fmt=LOG_FORMAT, datefmt=LOG_TIMEFMT))
        ocrd_handler.setLevel(logging.DEBUG)
        for logger_name in ROOT_OCRD_LOGGERS:
            logger = logging.getLogger(logger_name)
            logger.addHandler(ocrd_handler)
            if logger_name:
                logger.propagate = False # avoid duplication (from root handler)
        for logger_name, logger_level in LOGGING_DEFAULTS.items():
            logging.getLogger(logger_name).setLevel(logger_level)
    _initialized_flag = True

def disableLogging(silent=not config.OCRD_LOGGING_DEBUG):
    """
    Disables all logging of the ``ocrd`` logger and descendants

    Keyword Args:
        - silent (bool, True): Whether to log logging behavior by printing to stderr
    """
    global _initialized_flag # pylint: disable=global-statement
    if _initialized_flag and not silent:
        print("[LOGGING] Disabling logging", file=sys.stderr)
    _initialized_flag = False
    # logging.basicConfig(level=logging.CRITICAL)
    # logging.disable(logging.ERROR)
    # remove all handlers for the ocrd logger
    for logger_name in ROOT_OCRD_LOGGERS:
        for handler in logging.getLogger(logger_name).handlers[:]:
            logging.getLogger(logger_name).removeHandler(handler)
    for logger_name in LOGGING_DEFAULTS:
        logging.getLogger(logger_name).setLevel(logging.NOTSET)
    # Python default log level is WARNING
    logging.root.setLevel(logging.WARNING)

# Initializing stream handlers at module level
# would cause message output in all runtime contexts,
# including those which are already run for std output
# (--dump-json, --version, ocrd-tool, bashlib etc).
# So this needs to be an opt-in from the CLIs/decorators:
#initLogging()
# Also, we even have to block log output for libraries
# (like matplotlib/tensorflow) which set up logging
# themselves already:
disableLogging()


1			"""
2			Logging setup
3
4			By default: Log with lastResort logger, usually STDERR.
5
6			Logging can be overridden either programmatically in code using the library or by creating one or more of
7
8			- /etc/ocrd_logging.py
9			- $HOME/ocrd_logging.py
10			- $PWD/ocrd_logging.py
11
12			These files will be executed in the context of ocrd/ocrd_logging.py, with `logging` global set.
13
14			Changes as of 2023-08-20:
15
16			- Try to be less intrusive with OCR-D specific logging conventions to
17			make it easier and less surprising to define logging behavior when
18			using OCR-D/core as a library
19			- Change setOverrideLogLevel to only override the log level of the ``ocrd``
20			logger and its descendants
21			- initLogging will set exactly one handler, for the root logger or for the
22			``ocrd`` logger.
23			- Child loggers should propagate to the ancestor logging (default
24			behavior of the logging library - no more PropagationShyLogger)
25			- disableLogging only removes any handlers from the ``ocrd`` logger
26			"""
27			# pylint: disable=no-member
28
29			from __future__ import absolute_import
30
31			from traceback import format_stack
32
33			import logging
34			import logging.config
35			from pathlib import Path
36			import sys
37
38			from .constants import LOG_FORMAT, LOG_TIMEFMT
39			from .config import config
40
41
42			__all__ = [
43			'disableLogging',
44			'getLevelName',
45			'getLogger',
46			'initLogging',
47			'logging',
48			'setOverrideLogLevel',
49			]
50
51			# These are the loggers we add handlers to
52			ROOT_OCRD_LOGGERS = [
53			'',
54			'ocrd',
55			'ocrd_network'
56			]
57
58			LOGGING_DEFAULTS = {
59			'ocrd': logging.INFO,
60			'ocrd_network': logging.INFO,
61			# 'ocrd.resolver': logging.INFO,
62			# 'ocrd.resolver.download_to_directory': logging.INFO,
63			# 'ocrd.resolver.add_files_to_mets': logging.INFO,
64			# To cut back on the `Self-intersection at or near point` INFO messages
65			'shapely.geos': logging.ERROR,
66			'tensorflow': logging.ERROR,
67			'PIL': logging.INFO,
68			'paramiko.transport': logging.INFO,
69			'uvicorn.access': logging.DEBUG,
70			'uvicorn.error': logging.DEBUG,
71			'uvicorn': logging.INFO,
72			'multipart': logging.INFO,
73			}
74
75			_initialized_flag = False
76
77			_ocrdLevel2pythonLevel = {
78			'TRACE': 'DEBUG',
79			'OFF': 'CRITICAL',
80			'FATAL': 'ERROR',
81			}
82
83			def tf_disable_interactive_logs():
84			try:
85			from os import environ
86			# This env variable must be set before importing from Keras
87			environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
88			from tensorflow.keras.utils import disable_interactive_logging
89			# Enabled interactive logging throws an exception
90			# due to a call of sys.stdout.flush()
91			disable_interactive_logging()
92			except ImportError:
93			# Nothing should be handled here if TF is not available
94			pass
95
96			def getLevelName(lvl):
97			"""
98			Get (string) python logging level for (string) spec-defined log level name.
99			"""
100			lvl = _ocrdLevel2pythonLevel.get(lvl, lvl)
101			return logging.getLevelName(lvl)
102
103			def getLogger(args, *kwargs):
104			"""
105			Wrapper around ``logging.getLogger`` that calls :py:func:`initLogging` if
106			that wasn't explicitly called before.
107			"""
108			logger = logging.getLogger(args, *kwargs)
109			return logger
110
111			def setOverrideLogLevel(lvl, silent=not config.OCRD_LOGGING_DEBUG):
112			"""
113			Override the output log level of the handlers attached to the ``ocrd`` logger.
114
115			Args:
116			lvl (string): Log level name.
117			silent (boolean): Whether to log the override call
118			"""
119			if not _initialized_flag:
120			initLogging(silent=silent)
121			ocrd_logger = logging.getLogger('ocrd')
122
123			if lvl is None:
124			if not silent:
125			print('[LOGGING] Reset log level override', file=sys.stderr)
126			ocrd_logger.setLevel(logging.NOTSET)
127			else:
128			if not silent:
129			print(f'[LOGGING] Overriding ocrd log level to {lvl}', file=sys.stderr)
130			ocrd_logger.setLevel(lvl)
131
132			def get_logging_config_files():
133			"""
134			Return a list of all ``ocrd_logging.conf`` files found in CWD, HOME or /etc.
135			"""
136			CONFIG_PATHS = [
137			Path.cwd(),
138			Path.home(),
139			Path('/etc'),
140			]
141			return [f for f \
142			in [p / 'ocrd_logging.conf' for p in CONFIG_PATHS] \
143			if f.exists()]
144
145			def initLogging(builtin_only=False, force_reinit=False, silent=not config.OCRD_LOGGING_DEBUG):
146			"""
147			Reset ``ocrd`` logger, read logging configuration if exists, otherwise use basicConfig
148
149			initLogging is to be called by OCR-D/core once, i.e.
150			- for the ``ocrd`` CLI
151			- for the processor wrapper methods
152
153			Other processes that use OCR-D/core as a library can, but do not have to, use this functionality.
154
155			Keyword Args:
156			- builtin_only (bool, False): Whether to search for logging configuration
157			on-disk (``False``) or only use the
158			hard-coded config (``True``). For testing
159			- force_reinit (bool, False): Whether to ignore the module-level
160			``_initialized_flag``. For testing only.
161			- silent (bool, True): Whether to log logging behavior by printing to stderr
162			"""
163			global _initialized_flag
164			if _initialized_flag and not force_reinit:
165			return
166			# disableLogging()
167
168			# https://docs.python.org/3/library/logging.html#logging.disable
169			# If logging.disable(logging.NOTSET) is called, it effectively removes this
170			# overriding level, so that logging output again depends on the effective
171			# levels of individual loggers.
172			logging.disable(logging.NOTSET)
173
174			# remove all handlers for the ocrd root loggers
175			for logger_name in ROOT_OCRD_LOGGERS:
176			for handler in logging.getLogger(logger_name).handlers[:]:
177			logging.getLogger(logger_name).removeHandler(handler)
178
179			config_file = None
180			if not builtin_only:
181			config_file = get_logging_config_files()
182			if config_file:
183			if len(config_file) > 1 and not silent:
184			print(f"[LOGGING] Multiple logging configuration files found at {config_file}, using first one", file=sys.stderr)
185			config_file = config_file[0]
186			if not silent:
187			print(f"[LOGGING] Picked up logging config at {config_file}", file=sys.stderr)
188			logging.config.fileConfig(config_file)
189			else:
190			if not silent:
191			print("[LOGGING] Initializing logging with built-in defaults", file=sys.stderr)
192			# Default logging config
193			ocrd_handler = logging.StreamHandler(stream=sys.stderr)
194			ocrd_handler.setFormatter(logging.Formatter(fmt=LOG_FORMAT, datefmt=LOG_TIMEFMT))
195			ocrd_handler.setLevel(logging.DEBUG)
196			for logger_name in ROOT_OCRD_LOGGERS:
197			logger = logging.getLogger(logger_name)
198			logger.addHandler(ocrd_handler)
199			if logger_name:
200			logger.propagate = False # avoid duplication (from root handler)
201			for logger_name, logger_level in LOGGING_DEFAULTS.items():
202			logging.getLogger(logger_name).setLevel(logger_level)
203			_initialized_flag = True
204
205			def disableLogging(silent=not config.OCRD_LOGGING_DEBUG):
206			"""
207			Disables all logging of the ``ocrd`` logger and descendants
208
209			Keyword Args:
210			- silent (bool, True): Whether to log logging behavior by printing to stderr
211			"""
212			global _initialized_flag # pylint: disable=global-statement
213			if _initialized_flag and not silent:
214			print("[LOGGING] Disabling logging", file=sys.stderr)
215			_initialized_flag = False
216			# logging.basicConfig(level=logging.CRITICAL)
217			# logging.disable(logging.ERROR)
218			# remove all handlers for the ocrd logger
219			for logger_name in ROOT_OCRD_LOGGERS:
220			for handler in logging.getLogger(logger_name).handlers[:]:
221			logging.getLogger(logger_name).removeHandler(handler)
222			for logger_name in LOGGING_DEFAULTS:
223			logging.getLogger(logger_name).setLevel(logging.NOTSET)
224			# Python default log level is WARNING
225			logging.root.setLevel(logging.WARNING)
226
227			# Initializing stream handlers at module level
228			# would cause message output in all runtime contexts,
229			# including those which are already run for std output
230			# (--dump-json, --version, ocrd-tool, bashlib etc).
231			# So this needs to be an opt-in from the CLIs/decorators:
232			#initLogging()
233			# Also, we even have to block log output for libraries
234			# (like matplotlib/tensorflow) which set up logging
235			# themselves already:
236			disableLogging()
237

OCR-D / core

Pull Request — master (#1288)

ocrd_utils.logging.initLogging() F

Complexity

Size

Duplication

Importance

How to fix Long Method Complexity

Long Method

Complexity

Duplication Side-by-Side

Filter issues like