2021SC@SDUSC
Starting from this article, I will formally interpret the code from bottom to top according to the relationship between function calls. This article begins to introduce the part of general.py. Due to the large amount of code, I will interpret it several times.
reference library
import contextlib import glob import logging import math import os import platform import random import re import signal import time import urllib from itertools import repeat from multiprocessing.pool import ThreadPool from pathlib import Path from subprocess import check_output import cv2 import numpy as np import pandas as pd import pkg_resources as pkg import torch import torchvision import yaml from utils.downloads import gsutil_getsize from utils.metrics import box_iou, fitness
contextlib: a utility for creating and using context manager. This module contains real-time programs for processing context manager and with statements.
Glob: This module is used to find file directories and files, and return the search results to a list. The two common methods are glob.glob() and glob.iglob(), which can be compared with the common find function.
logging: This module can print the log on the terminal and record the log to a file when the program is running according to the custom log information.
math: math Library
os: provides an interface to interact with the operating system.
Platform: This module is used to access platform related properties.
Random: generate random numbers
re: provides regular expression functionality
signal: it is responsible for processing signals inside python programs, mainly for UNIX platforms.
Time: provides the function of processing date and time.
urllib: provides the function of crawling web pages.
itertools: provides various functions that can work on iterators to produce complex iterators. The repeat() function belongs to the infinite iterator category. It gives the data and numbers, and how many times the data will be repeated. If no number is specified, it will be repeated countless times.
multiprocessing: provides processing of multiple processes and threads.
pathlib: the operation object of this module is the path used in various operating systems.
subprocess: allows you to generate new processes, connect to their input/output/error pipes, and get their returns. check_ The output() function executes the specified command. If the execution status code is 0, the execution result will be returned. Otherwise, an exception will be thrown.
cv2: openCV, which provides methods for image processing.
numpy: a common library for multidimensional data processing.
pandas: data analysis support library.
pkg_resource: provides runtime tools for finding, introspecting, activating, and using installed python distributions.
torch: pytorch library, deep learning framework.
Torch vision: visual processing in pytorch
yaml: is a special language for writing configuration files.
set up
# Settings torch.set_printoptions(linewidth=320, precision=5, profile='long') np.set_printoptions(linewidth=320, formatter={'float_kind': '{:11.5g}'.format}) # format short g, %precision=5 pd.options.display.max_columns = 10 cv2.setNumThreads(0) # prevent OpenCV from multithreading (incompatible with PyTorch DataLoader) os.environ['NUMEXPR_MAX_THREADS'] = str(min(os.cpu_count(), 8)) # NumExpr max threads
torch.set_printoptions sets some formats for output. linewidth - the length of each line of output, precision - the floating-point precision is reserved for 5 bits, and profile - modifies the default settings.
np.set_ Printing options ibid
pd.options.display.max_columns sets the maximum number of output columns to 10
cv2.setNumThreads turn off multithreading in opencv
The next sentence sets the number of threads in the program.
Profile class
class Profile(contextlib.ContextDecorator): # Usage: @Profile() decorator or 'with Profile():' context manager def __enter__(self): self.start = time.time() def __exit__(self, type, value, traceback): print(f'Profile results: {time.time() - self.start:.5f}s')
It inherits from the ContextDecorator class, which is the base class that enables the context manager to also be used as a decorator, and needs to be overridden__ enter__ And__ exit__ The function. Profile is used for timing, recording the current time when entering, and outputting the difference between the current time and start when exiting.
Timeout class
class Timeout(contextlib.ContextDecorator): # Usage: @Timeout(seconds) decorator or 'with Timeout(seconds):' context manager def __init__(self, seconds, *, timeout_msg='', suppress_timeout_errors=True): self.seconds = int(seconds) self.timeout_message = timeout_msg self.suppress = bool(suppress_timeout_errors) def _timeout_handler(self, signum, frame): raise TimeoutError(self.timeout_message) def __enter__(self): signal.signal(signal.SIGALRM, self._timeout_handler) # Set handler for SIGALRM signal.alarm(self.seconds) # start countdown for SIGALRM to be raised def __exit__(self, exc_type, exc_val, exc_tb): signal.alarm(0) # Cancel SIGALRM if it's scheduled if self.suppress and exc_type is TimeoutError: # Suppress TimeoutError return True
This class also inherits from ContextDecorator and is used to check whether it times out. seconds is the specified time, beyond which an exception will be thrown, timeout_msg is the exception output information. suppress is whether to strictly check. If strictly check, an exception will be thrown, but if not, the exception information will be output, and the program will continue to execute.
__ enter__ Method defines an alarm signal, which is initiated by signal.alarm. The parameter is seconds, that is, the time we want to guarantee. When the time exceeds the time we define, it will be called_ timeout_handler, the method throws an exception, executes_- exit__ Method__ exit__ First, turn off the alarm clock to prevent an error when the program exits normally. Next, check whether it is strictly checked. If yes, throw an exception. Otherwise, return directly and output exception information.
try_except function
def try_except(func): # try-except function. Usage: @try_except decorator def handler(*args, **kwargs): try: func(*args, **kwargs) except Exception as e: print(e) return handler
This function integrates the try and except modules. When you need to use this module, you can call this function directly instead of calling the try and ecept modules. func is the content to be executed.
methods function
def methods(instance): # Get class/instance methods return [f for f in dir(instance) if callable(getattr(instance, f)) and not f.startswith("__")]
This function returns all public methods of a class or instance into a list for later calls.
set_logging function
def set_logging(rank=-1, verbose=True): logging.basicConfig( format="%(message)s", level=logging.INFO if (verbose and rank in [-1, 0]) else logging.WARN)
Use the default formatter to create a StreamHandler and add it to the root logger to complete the basic configuration of the logging system. If there is no handler defined for the root logger, the debug(),info(),warning(),error() and critical() functions will automatically call basicConfig().
Format is that the handler uses the specified format string, and level sets the root logger level to the specified level. By default, the level of the generated root logger is logging.Warning. If it is lower than this level, it will not be output.
print_args function
def print_args(name, opt): # Print argparser arguments print(colorstr(f'{name}: ') + ', '.join(f'{k}={v}' for k, v in vars(opt).items()))
Output argparser parameters
init_seeds function
def init_seeds(seed=0): # Initialize random number generator (RNG) seeds https://pytorch.org/docs/stable/notes/randomness.html # cudnn seed 0 settings are slower and more reproducible, else faster and less reproducible import torch.backends.cudnn as cudnn random.seed(seed) np.random.seed(seed) torch.manual_seed(seed) cudnn.benchmark, cudnn.deterministic = (False, True) if seed == 0 else (True, False)
Initialize the seed of the random generator. When seed is 0, set cudnn to slower, when reusability is higher, otherwise set to faster but low reusability.
get_latest_run function
def get_latest_run(search_dir='.'): # Return path to most recent 'last.pt' in /runs (i.e. to --resume from) last_list = glob.glob(f'{search_dir}/**/last*.pt', recursive=True) return max(last_list, key=os.path.getctime) if last_list else ''
last_list is a list returned by glob. glob.glob can return a list of all matching file paths. The parameter defines the file path matching rules, that is, find all the file paths of last**.pt, * * represents all other strings. Finally, the path closest to the current time is returned according to the creation time
user_config_dir function
def user_config_dir(dir='Ultralytics', env_var='YOLOV5_CONFIG_DIR'): # Return path of user configuration directory. Prefer environment variable if exists. Make dir if required. env = os.getenv(env_var) if env: path = Path(env) # use environment variable else: cfg = {'Windows': 'AppData/Roaming', 'Linux': '.config', 'Darwin': 'Library/Application Support'} # 3 OS dirs path = Path.home() / cfg.get(platform.system(), '') # OS-specific config dir path = (path if is_writeable(path) else Path('/tmp')) / dir # GCP and AWS lambda fix, only /tmp is writeable path.mkdir(exist_ok=True) # make if required return path
Returns the path of the user configuration environment directory. If it does not exist, create one.