reference resources: https://www.cnblogs.com/yyds/p/6901864.html

1. The log level increases from top to bottom, that is, debug < info < warning < error < critical
2. When a log level is specified for an application, the application will record all log information with a log level greater than or equal to the specified log level, rather than only log information at the specified level. Similarly, the logging module can also specify the log level of the logger. Only log records with a level greater than or equal to the specified log level will be output, and log records less than the level will be discarded
3. The logging level set by the logger used in the logging function provided by the logging module is WARNING. Therefore, only the logging records of WARNING level and the logging records of ERROR and CRITICAL levels greater than it are output, while the logging records of DEBUG and INFO levels less than it are discarded
4. The log format set by the logger used by the logging function provided by the logging module is basic by default_ Format, whose value is: "% (levelname)s:%(name)s:%(message)s"
Therefore, the meaning of each field of each line of log record in the output result is: log level: logger Name: log content
5. The logging implementation can view the code of the basicConfig() method, and change the value of the default setting through the basicConfig() method, such as the default log level set by the logger

Log using the module level functions provided by logging

function	explain
logging.debug(msg, *args, **kwargs)	Create a log record with severity level DEBUG
logging.info(msg, *args, **kwargs)	Create a log record with severity INFO
logging.warning(msg, *args, **kwargs)	Create a log record with severity WARNING
logging.error(msg, *args, **kwargs)	Create a log record with severity ERROR
logging.critical(msg, *args, **kwargs)	Create a log record with CRITICAL severity
logging.log(level, *args, **kwargs)	Create a log record with severity level
logging.basicConfig(**kwargs)	Configure the root logger at one time to specify the log level to be recorded, log format, log output location, log file opening mode and other information

• logging.basicConfig(**kwargs) can receive keyword parameters:

Keyword parameters	explain
filename	Specify the file name of the log output target file. After specifying this setting, the log will not be output to the console
filemode	Specify the opening mode of the log file. The default is' a '. Note that this option is only valid when filename is specified
format	Specify the log format string, that is, specify the field information contained in the log output and their order. The format fields defined by the logging module are listed below
datefmt	Specify the date / time format. It should be noted that this option is only valid when the format contains the time field% (actime) s
level	Specifies the log level of the logger
stream	Specify log output target streams, such as sys.stdout, sys.stderr, and network stream. It should be noted that stream and filename cannot be provided at the same time, otherwise ValueError exception will be thrown
style	New configuration item added in Python 3.2. Specify the style of format string. The values can be '%', '{' and '$', and the default is'% '
handlers	A newly added configuration item in Python 3.3. If this option is specified, it should be an iterative object that creates multiple handlers, and these handlers will be added to the root logger. It should be noted that only one of the three configuration items filename, stream and handlers can exist, and two or three can not appear at the same time, otherwise a ValueError exception will be thrown

• Format string field defined by the logging module

Format string field	Use format	explain
asctime	%(asctime)s	Time of log event – human readable time, such as: 2003-07-08 16:49:45896
created	%(created)f	The time of the log event – timestamp, which is the value returned by calling the time.time() function at that time
relativeCreated	%(relativeCreated)d	The relative number of milliseconds of the log event occurrence time relative to the loading time of the logging module (I don't know why to use it at present)
msecs	%(msecs)d	The millisecond portion of the log event occurrence event
levelname	%(levelname)s	The log level in text form of the log record ('DEBUG ',' INFO ',' WARNING ',' ERROR ',' CRITICAL ')
levelno	%(levelno)s	The log level (10, 20, 30, 40, 50) of the digital form of the log record
name	%(name)s	The name of the logger used is' root 'by default, because the rootLogger is used by default
message	%(message)s	The text content of the log record is calculated by MSG% args
pathname	%(pathname)s	The full path of the source file that calls the logging function
filename	%(filename)s	The file name part of pathname, including the file suffix
module	%(module)s	The name part of filename, excluding the suffix
lineno	%(lineno)d	The line number of the source code that calls the logging function
funcName	%(funcName)s	The name of the function that called the logging function
process	%(process)d	Process ID
processName	%(processName)s	Process name, Python 3.1 NEW
thread	%(thread)d	Thread ID
threadName	%(thread)s	Thread name

be careful:

1. The logging.basicConfig() function is a one-time simple configuration tool, that is, it works only when the function is called for the first time, and no operation will be generated when the function is called again. The setting of multiple calls is not an accumulation operation
2. If the log to be recorded contains variable data, you can use a format string as the description message of this event (the first parameter of logging.debug, logging.info and other functions), and then pass the variable data as the value of the second parameter * args, such as logging.warning('% s is% d years old,' 'Tom', 10) The output is WARNING:root:Tom is 10 years old
3. In the definitions of logging.debug(), logging.info(), there is a * * kwargs parameter in addition to msg and args parameters. They support three keyword parameters: exc_info, stack_info and extra

Description of exc_info, stack_info, extra keyword parameters:
exc_info: its value is Boolean. If the value of this parameter is set to True, exception information will be added to the log message. If there is no exception information, None will be added to the log message.
stack_info: its value is also Boolean, and the default value is False. If the value of this parameter is set to True, the stack information will be added to the log information.
extra: This is a dict parameter. It can be used to customize the fields contained in the message format, but its key cannot conflict with the fields defined in the logging module

example:

import logging
import time
# You can use this module to write the error message to the traceback.format_exc() time file
import traceback
# Level indicates the log recording level (all levels above this level are recorded in the log), filename indicates the log path, and filemode indicates the log writing mode
# Format log
LOG_FORMAT = "%(asctime)s - %(levelname)s - %(message)s"
# Format date
DATE_FORMAT = "%m/%d/%Y %H:%M:%S %p"
logging.basicConfig(level=logging.DEBUG,filename='d:/loginfo11.log',filemode='a',format=LOG_FORMAT,datefmt=DATE_FORMAT)
logging.info('This is a info')
logging.debug('This is a debug')
logging.warning('This is a warning'+' '+time.strftime('%y-%m-%d %H:%M:%S')+' '+traceback.format_exc())
logging.error('This is a error',exc_info=True, stack_info=True, extra={'user': 'Tom', 'ip':'47.98.53.222'})
logging.critical('This is a critical')
logging.log(logging.CRITICAL,"This is a critical")

# Wrong type. The parent or ancestor of all exception classes is exception. The parent of exception is BaseException
# NameError undefined variable
# IndexError subscript out of bounds
# The denominator of ZeroDivisionError is 0
# IOError input / output exception
print(Exception.__bases__)
print(BaseException.__bases__)

Use the four components of the Logging system to record logs

Component name	Corresponding class name	explain
Logger	Loggers	Provide an interface directly used by the application code. The main tasks are: 1) expose several methods to the application code so that the application can record log messages at run time; 2) Determine which logs to perform subsequent processing based on the log severity level (default filtering facility) or filter object; 3) Send log messages to all interested log handlers
processor	Handlers	It is used to send log records to the specified destination. The main task is to distribute messages to the location specified by the handler (file, network, mail, etc.) based on the level of log messages
filter	Filters	Provide finer grained log filtering function to determine which log records will be output (other log records will be ignored). 1) its instance object can pass a log name to control the logs that can be filtered; 2) Whether the method filter(record) record under filters can pass filtering. If the return value of the method is 0, it means it cannot pass filtering, and if the return value is non-0, it means it can pass filtering
Formatter	Formatters	It is used to control the final output format of log information. You can directly instantiate the Formatter class

• Implementation steps:

1. Create log object
2. Set log message level
3. Create the corresponding Handler
4. Set the message level handled by the handler
5. Set the log output format for the handler
6. Create handler log filter
7. handler adds the log filter from step 6
8. Add the handler created in step 3 to the log

• Get Logger object

1. Create an instance of the Logger class through the instantiation method of the Logger class
2. Get through the Logger = logging.getLogger() method (this method has an optional parameter name, which indicates the name and identification of the logger to be returned. If this parameter is not provided, its value is' root '. If the getLogger() method is called multiple times with the same name parameter value, a reference to the same logger object will be returned)

1. The name of a logger is a hierarchical structure divided by '.'. Each subsequent logger is the children of the previous logger. For example, there is a logger named foo, and the other names are foo.bar, foo.bar.baz and foo.bam, which are descendants of foo.
2. Logger has a concept of "effective level". If a level is not explicitly set on a logger, the logger uses the level of its parent; If the level of its parent is not explicitly set, continue to look up the valid level of its parent, and so on until an ancestor with level explicitly set is found. It should be noted that the root logger always has an explicit level setting (WARNING by default). When deciding whether to process an event that has occurred, the validity level of the logger will be used to decide whether to pass the event to the handlers of the logger for processing.
3. After processing log messages, child loggers will pass the log messages to the handlers related to their ancestor loggers by default. Therefore, we do not need to define and configure handlers for all loggers used in an application. We only need to configure handlers for a top-level logger and then create child loggers as needed. We can also turn off this delivery mechanism by setting the propagate property of a logger to False

• Common methods for Logger objects

method	describe
Logger.setLevel()	Sets the minimum severity level of log messages that the logger will process
Logger.addHandler() and Logger.removeHandler()	Add and remove a handler object for the logger object
Logger.addFilter() and Logger.removeFilter()	Add and remove a filter object for the logger object

• Create logging method

method	describe
Logger.debug(), Logger.info(), Logger.warning(), Logger.error(), Logger.critical()	Create a level of logging corresponding to their method names
Logger.exception()	Create a log message similar to Logger.error()
Logger.log()	You need to get an explicit log level parameter to create a log record

• Common handlers:

Handler	describe
logging.StreamHandler	Send log messages to the output Stream, such as std.out, std.err, or any file like object
logging.FileHandler	Send log messages to disk files. By default, the file size will grow indefinitely
logging.handlers.RotatingFileHandler	Send log messages to disk files and support cutting log files by size
logging.hanlders.TimedRotatingFileHandler	Send log messages to disk files and support log file cutting by time
logging.handlers.HTTPHandler	Send log messages to an HTTP server in the form of GET or POST
logging.handlers.SMTPHandler	Send the log message to a specified email address
logging.NullHandler	This Handler instance ignores error messages, which is usually used by library developers who want to use logging to avoid the message 'No handlers could be found for logger XXX'

Note: application code should not instantiate and use Handler instances directly. Because Handler is a base class, it only defines the interfaces that all existing handlers should have, and provides some default behaviors that subclasses can directly use or override

• Methods of common handler classes:

method	describe
Handler.setLevel()	Set the minimum severity level of log messages that the handler will process
Handler.setFormatter()	Set a formatter object for the handler
Handler.addFilter() and Handler.removeFilter()	Add and remove a filter object for the handler

• Constructor of Formatter class:

logging.Formatter.init(fmt=None, datefmt=None, style='%')
fmt: Specifies the message format string. If this parameter is not specified, the original value of message is used by default
datefmt: Specifies the date format string. If this parameter is not specified, "% Y -% m -% d% H:% m:% s" will be used by default
style: a new parameter added in Python 3.2. The values can be '%', '{' and '$'. If this parameter is not specified, '%' will be used by default

• Filter can be used by the Handler and logger to perform finer grained and more complex filtering functions than level. Filter is a filter base class, which only allows log events under a logger level to pass filtering. This class is defined as follows: whether record records can pass filtering. If the return value of this method is 0, it means it cannot pass filtering, and if the return value is non-0, it means it can pass filtering

class logging.Filter(name='')
    filter(record)

Example:
1) All logs at all levels are required to be written to disk files
2) All log information is recorded in the all.log file. The log format is: date and time - log level - log information
3) The log information of error and above levels is recorded separately in the error.log file. The log format is: date and time - log level - file name [: line number] - log information
4) all.log is required to perform log cutting every morning

import logging
import logging.handlers
import datetime

ERROR_LOG_FORMAT = "%(asctime)s - %(levelname)s - %(filename)s[:%(lineno)d] - %(message)s"
ALL_LOG_FORMAT = "%(asctime)s - %(levelname)s - %(message)s"
DATA_FORMAT = '%y-%m-%d %H:%M:%S %p '

loggings = logging.getLogger('mylogger')
loggings.setLevel(logging.DEBUG)
# Send log information to disk file, and support log file cutting by time
handlerAll = logging.handlers.TimedRotatingFileHandler('all.log')
handlerAll.setFormatter(logging.Formatter(ALL_LOG_FORMAT))
# Send log messages to disk files. By default, the file size will grow indefinitely
handlerError = logging.FileHandler('error.log')
handlerError.setLevel(logging.ERROR)
handlerError.setFormatter(logging.Formatter(ERROR_LOG_FORMAT))

# Add handler to logging
loggings.addHandler(handlerAll)
loggings.addHandler(handlerError)

loggings.debug('debug message')
loggings.info('info message')
loggings.warning('warning message')
loggings.error('error message')
loggings.critical('critical message')

Several methods of configuring log

reference resources: https://www.cnblogs.com/yyds/p/6885182.html

1) Use Python code to explicitly create loggers, handlers and formatters and call their configuration functions respectively;
2) Create a log configuration file, and then use the fileConfig() function to read the contents of the file;
3) Create a dict containing configuration information and pass it to the dictConfig() function

1. Explicit creation of python code:

# Create a logger and set its log level to DEBUG
logger = logging.getLogger('simple_logger')
logger.setLevel(logging.DEBUG)

# Create a stream handler and set its log level to DEBUG
handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.DEBUG)

# Create a formatter formatter and add it to the processor handler
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
handler.setFormatter(formatter)

# Add the processor handler created above to the logger
logger.addHandler(handler)

# Log output
logger.debug('debug message')
logger.info('info message')
logger.warn('warn message')
logger.error('error message')
logger.critical('critical message')

2. Log configuration file, read with fileConfig() function:

logging.config.fileConfig(fname, defaults=None, disable_existing_loggers=True)
fname: represents the file name or file object of the configuration file
defaults: Specifies the default value passed to ConfigParser
disable_existing_loggers: This is a Boolean value. The default value is True (for backward compatibility), which means that existing loggers are disabled unless they or their ancestors explicitly appear in the log configuration; If the value is False, the existing loggers remain started

Example:

# Read log configuration file contents
logging.config.fileConfig('logging.conf')

# Create a logger logger
logger = logging.getLogger('simpleExample')

# Log output
logger.debug('debug message')
logger.info('info message')
logger.warn('warn message')
logger.error('error message')
logger.critical('critical message')

Configuration file logging.conf:

[loggers]
keys=root,simpleExample

[handlers]
keys=fileHandler,consoleHandler

[formatters]
keys=simpleFormatter

[logger_root]
level=DEBUG
handlers=fileHandler

[logger_simpleExample]
level=DEBUG
handlers=consoleHandler
qualname=simpleExample
propagate=0

[handler_consoleHandler]
class=StreamHandler
args=(sys.stdout,)
level=DEBUG
formatter=simpleFormatter

[handler_fileHandler]
class=FileHandler
args=('logging.log', 'a')
level=ERROR
formatter=simpleFormatter

[formatter_simpleFormatter]
format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
class = logging.Formatter
datefmt=

Description of configuration file format:
1) The configuration file must contain sections such as loggers, handlers and formatters. They specify the defined loggers, handlers and formatters in the configuration file through the keys option. Multiple values are separated by commas; In addition, the keys in the loggers section must contain the value root;

2) The loggers, handlers and formatters specified in loggers, handlers and formatters need to be defined in separate section s below. The naming rules of seciton are [logger_loggerName], [formatter_formatterName], [handler_handlerName]

3) The section defining the logger must specify two option s: level and handlers. The desirable values of level are DEBUG, INFO, WARNING, ERROR, CRITICAL and NOTSET. NOTSET means that all levels of log messages should be recorded, including user-defined levels; The value of handlers is a comma separated list of handler names. The handler appearing here must appear in the section [handlers], and the corresponding handler must have a corresponding section definition in the configuration file;

4) For non root loggers, in addition to the two options of level and handlers, some additional options are required. Among them, qualname is the option that must be provided. It represents the name in the logger level, and the logger is obtained through this name in the application code; propagate is optional. Its default value is 1, which means that the message will be delivered to the handler of the high-level logger. Generally, we need to specify its value as 0, otherwise the log record will be delivered to the superior root logger for processing again; In addition, if the level of a non root logger is set to NOTSET, the system will find a high-level logger to determine the effective level of the logger.

5) Two options, class and args, must be specified in the section defining the handler. level and formatter are optional options; Class represents the class name used to create the handler, and args represents the initialization method parameters passed to the handler class specified by class
, it must be in the form of a tuple, even if there is only one parameter value, it must be in the form of a tuple; Level is the same as the level in logger, and formatter specifies the formatter used by the processor. The formatter name specified here must appear in the section of formatters, and the section definition of this formatter must be in the configuration file; If no formatter is specified, the handler will record the message itself as a log message without adding additional information such as time and logger name;

6) The options in the sectioin defining the formatter are optional, including format, which is used to specify the format string, and the default is the message string itself; datefmt is used to specify the time format of actime. The default is'% Y -% m -% d% H:% m:% s'; Class is used to specify the formatter class name. The default is logging.Formatter

Note: 1. When the class in the configuration file specifies the class name, the class name can be a relative value relative to the logging module, such as FileHandler, handlers.TimeRotatingFileHandler; It can also be an absolute path value, which can be resolved through the ordinary import mechanism, such as the custom handler class mypackage.mymodule.MyHandler, but mypackage needs to be in the import path available to Python - sys.path
2. When a logger is not set with any processor, the system will find the log processor set on the upper logger of the logger to process log records

3. Create dict containing configuration information and pass it to dictConfig() function:

logging.config.dictConfig(config)
The log configuration information can be obtained from a dictionary object, and the config parameter is the dictionary object

Example:

import logging
import logging.config
import yaml

with open('logging.yml', 'r') as f_conf:
    dict_conf = yaml.load(f_conf)
logging.config.dictConfig(dict_conf)

logger = logging.getLogger('simpleExample')
logger.debug('debug message')
logger.info('info message')
logger.warn('warn message')
logger.error('error message')
logger.critical('critical message')

Configuration file logging.yml:

version: 1
formatters:
  simple:
    format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
handlers:
  console:
    class: logging.StreamHandler
    level: DEBUG
    formatter: simple
    stream: ext://sys.stdout
  console_err:
    class: logging.StreamHandler
    level: ERROR
    formatter: simple
    stream: ext://sys.stderr
loggers:
  simpleExample:
    level: DEBUG
    handlers: [console]
    propagate: yes
root:
  level: DEBUG
  handlers: [console_err]

Description of configuration file format:

Configuration item	explain
version	Required. Its value is an integer value, indicating the version of the configuration format. The only available value at present is 1
formatters	Optional. Its value is a dictionary object. The key of each element of the dictionary object is the name of the formatter to be defined, and value is the dict composed of the configuration information of the formatter, such as format and datefmt
filters	Optional. Its value is a dictionary object. The key of each element of the dictionary object is the name of the filter to be defined, and value is the dict composed of filter configuration information, such as name
handlers	Optional. Its value is a dictionary object. The key of each element of the dictionary object is the processor name to be defined, and value is the dcit composed of processor configuration information, such as class, level, formatter and filters, where class is a required option and others are optional; Other configuration information will be passed to the constructor of the processor class specified by class, such as stream, filename, maxBytes and backupCount in the following handlers definition example
loggers	Optional. Its value is a dictionary object. The key of each element of the dictionary object is the name of the logger to be defined, and value is the dcit composed of the configuration information of the logger, such as level, handlers, filters and propagate (yes)
root	Optional. This is the configuration information of the root logger, and its value is also a dictionary object. Unless the propagate value is explicitly specified as no when defining other loggers, the handlers defined by root logger will be applied to other loggers
incremental	Optional. The default value is False. The meaning of this option is whether the definitions of the objects defined here are applied to the existing objects if they already exist. A value of False indicates that existing objects will be redefined
disable_existing_loggers	Optional. The default value is True. This option is used to specify whether to disable the existing logger loggers. If the value of incremental is True, this option will be ignored

Example of handler definition:

handlers:
  console:
    class : logging.StreamHandler
    formatter: brief
    level   : INFO
    filters: [allow_foo]
    stream  : ext://sys.stdout
  file:
    class : logging.handlers.RotatingFileHandler
    formatter: precise
    filename: logconfig.log
    maxBytes: 1024
    backupCount: 3

Add context information to log output

reference resources: https://www.cnblogs.com/yyds/p/6897964.html

1. Introduce context information by passing an extra parameter to the logging function (generally not recommended)

import logging
import sys

fmt = logging.Formatter("%(asctime)s - %(name)s - %(ip)s - %(username)s - %(message)s")
h_console = logging.StreamHandler(sys.stdout)
h_console.setFormatter(fmt)
logger = logging.getLogger("myPro")
logger.setLevel(logging.DEBUG)
logger.addHandler(h_console)

extra_dict = {"ip": "113.208.78.29", "username": "Petter"}
logger.debug("User Login!", extra=extra_dict)

extra_dict = {"ip": "223.190.65.139", "username": "Jerry"}
logger.info("User Access!", extra=extra_dict)

2. Introducing context information using LoggerAdapters

import logging
import sys

# Override the process() method in LoggerAdapter to prevent the parameter extra from being overwritten by the default value
class MyLoggerAdapter(logging.LoggerAdapter):

    def process(self, msg, kwargs):
        if 'extra' not in kwargs:
            kwargs["extra"] = self.extra
        return msg, kwargs

if __name__ == '__main__':
    # Initialize a logger instance to be passed to the LoggerAdapter constructor
    fmt = logging.Formatter("%(asctime)s - %(name)s - %(ip)s - %(username)s - %(message)s")
    h_console = logging.StreamHandler(sys.stdout)
    h_console.setFormatter(fmt)
    init_logger = logging.getLogger("myPro")
    init_logger.setLevel(logging.DEBUG)
    init_logger.addHandler(h_console)
    
    # Initialize a context dictionary object to be passed to the LoggerAdapter constructor
    extra_dict = {"ip": "IP", "username": "USERNAME"}
    
    # Get an instance of the custom LoggerAdapter class
    logger = MyLoggerAdapter(init_logger, extra_dict)
    
    # Logging method call in application
    logger.info("User Login!")
    logger.info("User Login!", extra={"ip": "113.208.78.29", "username": "Petter"})
    logger.info("User Login!")
    

3. Introducing context information using Filters
Using the custom Filter.Filter instance, modify the passed LogRecord instance in the filter(record) method, assign the context information to be added to the instance as a new attribute, and output these context information by specifying the string format of the formatter

import logging
from random import choice


class ContextFilter(logging.Filter):

        ip = 'IP'
        username = 'USER'

        def filter(self, record):
            record.ip = self.ip
            record.username = self.username
            return True

if __name__ == '__main__':
    levels = (logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR, logging.CRITICAL)
    users = ['Tom', 'Jerry', 'Peter']
    ips = ['113.108.98.34', '219.238.78.91', '43.123.99.68']

    logging.basicConfig(level=logging.DEBUG,
                        format='%(asctime)-15s %(name)-5s %(levelname)-8s %(ip)-15s %(username)-8s %(message)s')
    logger = logging.getLogger('myLogger')
    filter = ContextFilter()
    logger.addFilter(filter)
    logger.debug('A debug message')
    logger.info('An info message with %s', 'some parameters')

    for x in range(5):
        lvl = choice(levels)
        lvlname = logging.getLevelName(lvl)
        filter.ip = choice(ips)
        filter.username = choice(users)
        logger.log(lvl, 'A message at %s level with %d %s' , lvlname, 2, 'parameters')