reference resources: https://www.cnblogs.com/yyds/p/6901864.html
1. The log level increases from top to bottom, that is, debug < info < warning < error < critical
2. When a log level is specified for an application, the application will record all log information with a log level greater than or equal to the specified log level, rather than only log information at the specified level. Similarly, the logging module can also specify the log level of the logger. Only log records with a level greater than or equal to the specified log level will be output, and log records less than the level will be discarded
3. The logging level set by the logger used in the logging function provided by the logging module is WARNING. Therefore, only the logging records of WARNING level and the logging records of ERROR and CRITICAL levels greater than it are output, while the logging records of DEBUG and INFO levels less than it are discarded
4. The log format set by the logger used by the logging function provided by the logging module is basic by default_ Format, whose value is: "% (levelname)s:%(name)s:%(message)s"
Therefore, the meaning of each field of each line of log record in the output result is: log level: logger Name: log content
5. The logging implementation can view the code of the basicConfig() method, and change the value of the default setting through the basicConfig() method, such as the default log level set by the logger
Log using the module level functions provided by logging
function | explain |
---|---|
logging.debug(msg, *args, **kwargs) | Create a log record with severity level DEBUG |
logging.info(msg, *args, **kwargs) | Create a log record with severity INFO |
logging.warning(msg, *args, **kwargs) | Create a log record with severity WARNING |
logging.error(msg, *args, **kwargs) | Create a log record with severity ERROR |
logging.critical(msg, *args, **kwargs) | Create a log record with CRITICAL severity |
logging.log(level, *args, **kwargs) | Create a log record with severity level |
logging.basicConfig(**kwargs) | Configure the root logger at one time to specify the log level to be recorded, log format, log output location, log file opening mode and other information |
- logging.basicConfig(**kwargs) can receive keyword parameters:
Keyword parameters | explain |
---|---|
filename | Specify the file name of the log output target file. After specifying this setting, the log will not be output to the console |
filemode | Specify the opening mode of the log file. The default is' a '. Note that this option is only valid when filename is specified |
format | Specify the log format string, that is, specify the field information contained in the log output and their order. The format fields defined by the logging module are listed below |
datefmt | Specify the date / time format. It should be noted that this option is only valid when the format contains the time field% (actime) s |
level | Specifies the log level of the logger |
stream | Specify log output target streams, such as sys.stdout, sys.stderr, and network stream. It should be noted that stream and filename cannot be provided at the same time, otherwise ValueError exception will be thrown |
style | New configuration item added in Python 3.2. Specify the style of format string. The values can be '%', '{' and '$', and the default is'% ' |
handlers | A newly added configuration item in Python 3.3. If this option is specified, it should be an iterative object that creates multiple handlers, and these handlers will be added to the root logger. It should be noted that only one of the three configuration items filename, stream and handlers can exist, and two or three can not appear at the same time, otherwise a ValueError exception will be thrown |
- Format string field defined by the logging module
Format string field | Use format | explain |
---|---|---|
asctime | %(asctime)s | Time of log event – human readable time, such as: 2003-07-08 16:49:45896 |
created | %(created)f | The time of the log event – timestamp, which is the value returned by calling the time.time() function at that time |
relativeCreated | %(relativeCreated)d | The relative number of milliseconds of the log event occurrence time relative to the loading time of the logging module (I don't know why to use it at present) |
msecs | %(msecs)d | The millisecond portion of the log event occurrence event |
levelname | %(levelname)s | The log level in text form of the log record ('DEBUG ',' INFO ',' WARNING ',' ERROR ',' CRITICAL ') |
levelno | %(levelno)s | The log level (10, 20, 30, 40, 50) of the digital form of the log record |
name | %(name)s | The name of the logger used is' root 'by default, because the rootLogger is used by default |
message | %(message)s | The text content of the log record is calculated by MSG% args |
pathname | %(pathname)s | The full path of the source file that calls the logging function |
filename | %(filename)s | The file name part of pathname, including the file suffix |
module | %(module)s | The name part of filename, excluding the suffix |
lineno | %(lineno)d | The line number of the source code that calls the logging function |
funcName | %(funcName)s | The name of the function that called the logging function |
process | %(process)d | Process ID |
processName | %(processName)s | Process name, Python 3.1 NEW |
thread | %(thread)d | Thread ID |
threadName | %(thread)s | Thread name |
be careful:
1. The logging.basicConfig() function is a one-time simple configuration tool, that is, it works only when the function is called for the first time, and no operation will be generated when the function is called again. The setting of multiple calls is not an accumulation operation
2. If the log to be recorded contains variable data, you can use a format string as the description message of this event (the first parameter of logging.debug, logging.info and other functions), and then pass the variable data as the value of the second parameter * args, such as logging.warning('% s is% d years old,' 'Tom', 10) The output is WARNING:root:Tom is 10 years old
3. In the definitions of logging.debug(), logging.info(), there is a * * kwargs parameter in addition to msg and args parameters. They support three keyword parameters: exc_info, stack_info and extra
Description of exc_info, stack_info, extra keyword parameters:
exc_info: its value is Boolean. If the value of this parameter is set to True, exception information will be added to the log message. If there is no exception information, None will be added to the log message.
stack_info: its value is also Boolean, and the default value is False. If the value of this parameter is set to True, the stack information will be added to the log information.
extra: This is a dict parameter. It can be used to customize the fields contained in the message format, but its key cannot conflict with the fields defined in the logging module
example:
import logging import time # You can use this module to write the error message to the traceback.format_exc() time file import traceback # Level indicates the log recording level (all levels above this level are recorded in the log), filename indicates the log path, and filemode indicates the log writing mode # Format log LOG_FORMAT = "%(asctime)s - %(levelname)s - %(message)s" # Format date DATE_FORMAT = "%m/%d/%Y %H:%M:%S %p" logging.basicConfig(level=logging.DEBUG,filename='d:/loginfo11.log',filemode='a',format=LOG_FORMAT,datefmt=DATE_FORMAT) logging.info('This is a info') logging.debug('This is a debug') logging.warning('This is a warning'+' '+time.strftime('%y-%m-%d %H:%M:%S')+' '+traceback.format_exc()) logging.error('This is a error',exc_info=True, stack_info=True, extra={'user': 'Tom', 'ip':'47.98.53.222'}) logging.critical('This is a critical') logging.log(logging.CRITICAL,"This is a critical") # Wrong type. The parent or ancestor of all exception classes is exception. The parent of exception is BaseException # NameError undefined variable # IndexError subscript out of bounds # The denominator of ZeroDivisionError is 0 # IOError input / output exception print(Exception.__bases__) print(BaseException.__bases__)
Use the four components of the Logging system to record logs
Component name | Corresponding class name | explain |
---|---|---|
Logger | Loggers | Provide an interface directly used by the application code. The main tasks are: 1) expose several methods to the application code so that the application can record log messages at run time; 2) Determine which logs to perform subsequent processing based on the log severity level (default filtering facility) or filter object; 3) Send log messages to all interested log handlers |
processor | Handlers | It is used to send log records to the specified destination. The main task is to distribute messages to the location specified by the handler (file, network, mail, etc.) based on the level of log messages |
filter | Filters | Provide finer grained log filtering function to determine which log records will be output (other log records will be ignored). 1) its instance object can pass a log name to control the logs that can be filtered; 2) Whether the method filter(record) record under filters can pass filtering. If the return value of the method is 0, it means it cannot pass filtering, and if the return value is non-0, it means it can pass filtering |
Formatter | Formatters | It is used to control the final output format of log information. You can directly instantiate the Formatter class |
- Implementation steps:
1. Create log object
2. Set log message level
3. Create the corresponding Handler
4. Set the message level handled by the handler
5. Set the log output format for the handler
6. Create handler log filter
7. handler adds the log filter from step 6
8. Add the handler created in step 3 to the log
- Get Logger object
1. Create an instance of the Logger class through the instantiation method of the Logger class
2. Get through the Logger = logging.getLogger() method (this method has an optional parameter name, which indicates the name and identification of the logger to be returned. If this parameter is not provided, its value is' root '. If the getLogger() method is called multiple times with the same name parameter value, a reference to the same logger object will be returned)
1. The name of a logger is a hierarchical structure divided by '.'. Each subsequent logger is the children of the previous logger. For example, there is a logger named foo, and the other names are foo.bar, foo.bar.baz and foo.bam, which are descendants of foo.
2. Logger has a concept of "effective level". If a level is not explicitly set on a logger, the logger uses the level of its parent; If the level of its parent is not explicitly set, continue to look up the valid level of its parent, and so on until an ancestor with level explicitly set is found. It should be noted that the root logger always has an explicit level setting (WARNING by default). When deciding whether to process an event that has occurred, the validity level of the logger will be used to decide whether to pass the event to the handlers of the logger for processing.
3. After processing log messages, child loggers will pass the log messages to the handlers related to their ancestor loggers by default. Therefore, we do not need to define and configure handlers for all loggers used in an application. We only need to configure handlers for a top-level logger and then create child loggers as needed. We can also turn off this delivery mechanism by setting the propagate property of a logger to False
- Common methods for Logger objects
method | describe |
---|---|
Logger.setLevel() | Sets the minimum severity level of log messages that the logger will process |
Logger.addHandler() and Logger.removeHandler() | Add and remove a handler object for the logger object |
Logger.addFilter() and Logger.removeFilter() | Add and remove a filter object for the logger object |
- Create logging method
method | describe |
---|---|
Logger.debug(), Logger.info(), Logger.warning(), Logger.error(), Logger.critical() | Create a level of logging corresponding to their method names |
Logger.exception() | Create a log message similar to Logger.error() |
Logger.log() | You need to get an explicit log level parameter to create a log record |
- Common handlers:
Handler | describe |
---|---|
logging.StreamHandler | Send log messages to the output Stream, such as std.out, std.err, or any file like object |
logging.FileHandler | Send log messages to disk files. By default, the file size will grow indefinitely |
logging.handlers.RotatingFileHandler | Send log messages to disk files and support cutting log files by size |
logging.hanlders.TimedRotatingFileHandler | Send log messages to disk files and support log file cutting by time |
logging.handlers.HTTPHandler | Send log messages to an HTTP server in the form of GET or POST |
logging.handlers.SMTPHandler | Send the log message to a specified email address |
logging.NullHandler | This Handler instance ignores error messages, which is usually used by library developers who want to use logging to avoid the message 'No handlers could be found for logger XXX' |
Note: application code should not instantiate and use Handler instances directly. Because Handler is a base class, it only defines the interfaces that all existing handlers should have, and provides some default behaviors that subclasses can directly use or override
- Methods of common handler classes:
method | describe |
---|---|
Handler.setLevel() | Set the minimum severity level of log messages that the handler will process |
Handler.setFormatter() | Set a formatter object for the handler |
Handler.addFilter() and Handler.removeFilter() | Add and remove a filter object for the handler |
- Constructor of Formatter class:
logging.Formatter.init(fmt=None, datefmt=None, style='%')
fmt: Specifies the message format string. If this parameter is not specified, the original value of message is used by default
datefmt: Specifies the date format string. If this parameter is not specified, "% Y -% m -% d% H:% m:% s" will be used by default
style: a new parameter added in Python 3.2. The values can be '%', '{' and '$'. If this parameter is not specified, '%' will be used by default
- Filter can be used by the Handler and logger to perform finer grained and more complex filtering functions than level. Filter is a filter base class, which only allows log events under a logger level to pass filtering. This class is defined as follows: whether record records can pass filtering. If the return value of this method is 0, it means it cannot pass filtering, and if the return value is non-0, it means it can pass filtering
class logging.Filter(name='') filter(record)
Example:
1) All logs at all levels are required to be written to disk files
2) All log information is recorded in the all.log file. The log format is: date and time - log level - log information
3) The log information of error and above levels is recorded separately in the error.log file. The log format is: date and time - log level - file name [: line number] - log information
4) all.log is required to perform log cutting every morning
import logging import logging.handlers import datetime ERROR_LOG_FORMAT = "%(asctime)s - %(levelname)s - %(filename)s[:%(lineno)d] - %(message)s" ALL_LOG_FORMAT = "%(asctime)s - %(levelname)s - %(message)s" DATA_FORMAT = '%y-%m-%d %H:%M:%S %p ' loggings = logging.getLogger('mylogger') loggings.setLevel(logging.DEBUG) # Send log information to disk file, and support log file cutting by time handlerAll = logging.handlers.TimedRotatingFileHandler('all.log') handlerAll.setFormatter(logging.Formatter(ALL_LOG_FORMAT)) # Send log messages to disk files. By default, the file size will grow indefinitely handlerError = logging.FileHandler('error.log') handlerError.setLevel(logging.ERROR) handlerError.setFormatter(logging.Formatter(ERROR_LOG_FORMAT)) # Add handler to logging loggings.addHandler(handlerAll) loggings.addHandler(handlerError) loggings.debug('debug message') loggings.info('info message') loggings.warning('warning message') loggings.error('error message') loggings.critical('critical message')
Several methods of configuring log
reference resources: https://www.cnblogs.com/yyds/p/6885182.html
1) Use Python code to explicitly create loggers, handlers and formatters and call their configuration functions respectively;
2) Create a log configuration file, and then use the fileConfig() function to read the contents of the file;
3) Create a dict containing configuration information and pass it to the dictConfig() function
1. Explicit creation of python code:
# Create a logger and set its log level to DEBUG logger = logging.getLogger('simple_logger') logger.setLevel(logging.DEBUG) # Create a stream handler and set its log level to DEBUG handler = logging.StreamHandler(sys.stdout) handler.setLevel(logging.DEBUG) # Create a formatter formatter and add it to the processor handler formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s") handler.setFormatter(formatter) # Add the processor handler created above to the logger logger.addHandler(handler) # Log output logger.debug('debug message') logger.info('info message') logger.warn('warn message') logger.error('error message') logger.critical('critical message')
2. Log configuration file, read with fileConfig() function:
logging.config.fileConfig(fname, defaults=None, disable_existing_loggers=True)
fname: represents the file name or file object of the configuration file
defaults: Specifies the default value passed to ConfigParser
disable_existing_loggers: This is a Boolean value. The default value is True (for backward compatibility), which means that existing loggers are disabled unless they or their ancestors explicitly appear in the log configuration; If the value is False, the existing loggers remain started
Example:
# Read log configuration file contents logging.config.fileConfig('logging.conf') # Create a logger logger logger = logging.getLogger('simpleExample') # Log output logger.debug('debug message') logger.info('info message') logger.warn('warn message') logger.error('error message') logger.critical('critical message')
Configuration file logging.conf:
[loggers] keys=root,simpleExample [handlers] keys=fileHandler,consoleHandler [formatters] keys=simpleFormatter [logger_root] level=DEBUG handlers=fileHandler [logger_simpleExample] level=DEBUG handlers=consoleHandler qualname=simpleExample propagate=0 [handler_consoleHandler] class=StreamHandler args=(sys.stdout,) level=DEBUG formatter=simpleFormatter [handler_fileHandler] class=FileHandler args=('logging.log', 'a') level=ERROR formatter=simpleFormatter [formatter_simpleFormatter] format=%(asctime)s - %(name)s - %(levelname)s - %(message)s class = logging.Formatter datefmt=
Description of configuration file format:
1) The configuration file must contain sections such as loggers, handlers and formatters. They specify the defined loggers, handlers and formatters in the configuration file through the keys option. Multiple values are separated by commas; In addition, the keys in the loggers section must contain the value root;
2) The loggers, handlers and formatters specified in loggers, handlers and formatters need to be defined in separate section s below. The naming rules of seciton are [logger_loggerName], [formatter_formatterName], [handler_handlerName]
3) The section defining the logger must specify two option s: level and handlers. The desirable values of level are DEBUG, INFO, WARNING, ERROR, CRITICAL and NOTSET. NOTSET means that all levels of log messages should be recorded, including user-defined levels; The value of handlers is a comma separated list of handler names. The handler appearing here must appear in the section [handlers], and the corresponding handler must have a corresponding section definition in the configuration file;
4) For non root loggers, in addition to the two options of level and handlers, some additional options are required. Among them, qualname is the option that must be provided. It represents the name in the logger level, and the logger is obtained through this name in the application code; propagate is optional. Its default value is 1, which means that the message will be delivered to the handler of the high-level logger. Generally, we need to specify its value as 0, otherwise the log record will be delivered to the superior root logger for processing again; In addition, if the level of a non root logger is set to NOTSET, the system will find a high-level logger to determine the effective level of the logger.
5) Two options, class and args, must be specified in the section defining the handler. level and formatter are optional options; Class represents the class name used to create the handler, and args represents the initialization method parameters passed to the handler class specified by class
, it must be in the form of a tuple, even if there is only one parameter value, it must be in the form of a tuple; Level is the same as the level in logger, and formatter specifies the formatter used by the processor. The formatter name specified here must appear in the section of formatters, and the section definition of this formatter must be in the configuration file; If no formatter is specified, the handler will record the message itself as a log message without adding additional information such as time and logger name;
6) The options in the sectioin defining the formatter are optional, including format, which is used to specify the format string, and the default is the message string itself; datefmt is used to specify the time format of actime. The default is'% Y -% m -% d% H:% m:% s'; Class is used to specify the formatter class name. The default is logging.Formatter
Note: 1. When the class in the configuration file specifies the class name, the class name can be a relative value relative to the logging module, such as FileHandler, handlers.TimeRotatingFileHandler; It can also be an absolute path value, which can be resolved through the ordinary import mechanism, such as the custom handler class mypackage.mymodule.MyHandler, but mypackage needs to be in the import path available to Python - sys.path
2. When a logger is not set with any processor, the system will find the log processor set on the upper logger of the logger to process log records
3. Create dict containing configuration information and pass it to dictConfig() function:
logging.config.dictConfig(config)
The log configuration information can be obtained from a dictionary object, and the config parameter is the dictionary object
Example:
import logging import logging.config import yaml with open('logging.yml', 'r') as f_conf: dict_conf = yaml.load(f_conf) logging.config.dictConfig(dict_conf) logger = logging.getLogger('simpleExample') logger.debug('debug message') logger.info('info message') logger.warn('warn message') logger.error('error message') logger.critical('critical message')
Configuration file logging.yml:
version: 1 formatters: simple: format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s' handlers: console: class: logging.StreamHandler level: DEBUG formatter: simple stream: ext://sys.stdout console_err: class: logging.StreamHandler level: ERROR formatter: simple stream: ext://sys.stderr loggers: simpleExample: level: DEBUG handlers: [console] propagate: yes root: level: DEBUG handlers: [console_err]
Description of configuration file format:
Configuration item | explain |
---|---|
version | Required. Its value is an integer value, indicating the version of the configuration format. The only available value at present is 1 |
formatters | Optional. Its value is a dictionary object. The key of each element of the dictionary object is the name of the formatter to be defined, and value is the dict composed of the configuration information of the formatter, such as format and datefmt |
filters | Optional. Its value is a dictionary object. The key of each element of the dictionary object is the name of the filter to be defined, and value is the dict composed of filter configuration information, such as name |
handlers | Optional. Its value is a dictionary object. The key of each element of the dictionary object is the processor name to be defined, and value is the dcit composed of processor configuration information, such as class, level, formatter and filters, where class is a required option and others are optional; Other configuration information will be passed to the constructor of the processor class specified by class, such as stream, filename, maxBytes and backupCount in the following handlers definition example |
loggers | Optional. Its value is a dictionary object. The key of each element of the dictionary object is the name of the logger to be defined, and value is the dcit composed of the configuration information of the logger, such as level, handlers, filters and propagate (yes) |
root | Optional. This is the configuration information of the root logger, and its value is also a dictionary object. Unless the propagate value is explicitly specified as no when defining other loggers, the handlers defined by root logger will be applied to other loggers |
incremental | Optional. The default value is False. The meaning of this option is whether the definitions of the objects defined here are applied to the existing objects if they already exist. A value of False indicates that existing objects will be redefined |
disable_existing_loggers | Optional. The default value is True. This option is used to specify whether to disable the existing logger loggers. If the value of incremental is True, this option will be ignored |
Example of handler definition:
handlers: console: class : logging.StreamHandler formatter: brief level : INFO filters: [allow_foo] stream : ext://sys.stdout file: class : logging.handlers.RotatingFileHandler formatter: precise filename: logconfig.log maxBytes: 1024 backupCount: 3
Add context information to log output
reference resources: https://www.cnblogs.com/yyds/p/6897964.html
1. Introduce context information by passing an extra parameter to the logging function (generally not recommended)
import logging import sys fmt = logging.Formatter("%(asctime)s - %(name)s - %(ip)s - %(username)s - %(message)s") h_console = logging.StreamHandler(sys.stdout) h_console.setFormatter(fmt) logger = logging.getLogger("myPro") logger.setLevel(logging.DEBUG) logger.addHandler(h_console) extra_dict = {"ip": "113.208.78.29", "username": "Petter"} logger.debug("User Login!", extra=extra_dict) extra_dict = {"ip": "223.190.65.139", "username": "Jerry"} logger.info("User Access!", extra=extra_dict)
2. Introducing context information using LoggerAdapters
import logging import sys # Override the process() method in LoggerAdapter to prevent the parameter extra from being overwritten by the default value class MyLoggerAdapter(logging.LoggerAdapter): def process(self, msg, kwargs): if 'extra' not in kwargs: kwargs["extra"] = self.extra return msg, kwargs if __name__ == '__main__': # Initialize a logger instance to be passed to the LoggerAdapter constructor fmt = logging.Formatter("%(asctime)s - %(name)s - %(ip)s - %(username)s - %(message)s") h_console = logging.StreamHandler(sys.stdout) h_console.setFormatter(fmt) init_logger = logging.getLogger("myPro") init_logger.setLevel(logging.DEBUG) init_logger.addHandler(h_console) # Initialize a context dictionary object to be passed to the LoggerAdapter constructor extra_dict = {"ip": "IP", "username": "USERNAME"} # Get an instance of the custom LoggerAdapter class logger = MyLoggerAdapter(init_logger, extra_dict) # Logging method call in application logger.info("User Login!") logger.info("User Login!", extra={"ip": "113.208.78.29", "username": "Petter"}) logger.info("User Login!")
3. Introducing context information using Filters
Using the custom Filter.Filter instance, modify the passed LogRecord instance in the filter(record) method, assign the context information to be added to the instance as a new attribute, and output these context information by specifying the string format of the formatter
import logging from random import choice class ContextFilter(logging.Filter): ip = 'IP' username = 'USER' def filter(self, record): record.ip = self.ip record.username = self.username return True if __name__ == '__main__': levels = (logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR, logging.CRITICAL) users = ['Tom', 'Jerry', 'Peter'] ips = ['113.108.98.34', '219.238.78.91', '43.123.99.68'] logging.basicConfig(level=logging.DEBUG, format='%(asctime)-15s %(name)-5s %(levelname)-8s %(ip)-15s %(username)-8s %(message)s') logger = logging.getLogger('myLogger') filter = ContextFilter() logger.addFilter(filter) logger.debug('A debug message') logger.info('An info message with %s', 'some parameters') for x in range(5): lvl = choice(levels) lvlname = logging.getLevelName(lvl) filter.ip = choice(ips) filter.username = choice(users) logger.log(lvl, 'A message at %s level with %d %s' , lvlname, 2, 'parameters')