2021SC@SDUSC

Starting from this article, I will formally interpret the code from bottom to top according to the relationship between function calls. This article begins to introduce the part of general.py. Due to the large amount of code, I will interpret it several times.

reference library

import contextlib
import glob
import logging
import math
import os
import platform
import random
import re
import signal
import time
import urllib
from itertools import repeat
from multiprocessing.pool import ThreadPool
from pathlib import Path
from subprocess import check_output

import cv2
import numpy as np
import pandas as pd
import pkg_resources as pkg
import torch
import torchvision
import yaml

from utils.downloads import gsutil_getsize
from utils.metrics import box_iou, fitness

contextlib: a utility for creating and using context manager. This module contains real-time programs for processing context manager and with statements.

Glob: This module is used to find file directories and files, and return the search results to a list. The two common methods are glob.glob() and glob.iglob(), which can be compared with the common find function.

logging: This module can print the log on the terminal and record the log to a file when the program is running according to the custom log information.

math: math Library

os: provides an interface to interact with the operating system.

Platform: This module is used to access platform related properties.

Random: generate random numbers

re: provides regular expression functionality

signal: it is responsible for processing signals inside python programs, mainly for UNIX platforms.

Time: provides the function of processing date and time.

urllib: provides the function of crawling web pages.

itertools: provides various functions that can work on iterators to produce complex iterators. The repeat() function belongs to the infinite iterator category. It gives the data and numbers, and how many times the data will be repeated. If no number is specified, it will be repeated countless times.

multiprocessing: provides processing of multiple processes and threads.

pathlib: the operation object of this module is the path used in various operating systems.

subprocess: allows you to generate new processes, connect to their input/output/error pipes, and get their returns. check_ The output() function executes the specified command. If the execution status code is 0, the execution result will be returned. Otherwise, an exception will be thrown.

cv2: openCV, which provides methods for image processing.

numpy: a common library for multidimensional data processing.

pandas: data analysis support library.

pkg_resource: provides runtime tools for finding, introspecting, activating, and using installed python distributions.

torch: pytorch library, deep learning framework.

Torch vision: visual processing in pytorch

yaml: is a special language for writing configuration files.

set up

# Settings
torch.set_printoptions(linewidth=320, precision=5, profile='long')
np.set_printoptions(linewidth=320, formatter={'float_kind': '{:11.5g}'.format})  # format short g, %precision=5
pd.options.display.max_columns = 10
cv2.setNumThreads(0)  # prevent OpenCV from multithreading (incompatible with PyTorch DataLoader)
os.environ['NUMEXPR_MAX_THREADS'] = str(min(os.cpu_count(), 8))  # NumExpr max threads

  torch.set_printoptions sets some formats for output. linewidth - the length of each line of output, precision - the floating-point precision is reserved for 5 bits, and profile - modifies the default settings.

np.set_ Printing options ibid

pd.options.display.max_columns sets the maximum number of output columns to 10

cv2.setNumThreads turn off multithreading in opencv

The next sentence sets the number of threads in the program.

Profile class

class Profile(contextlib.ContextDecorator):
    # Usage: @Profile() decorator or 'with Profile():' context manager
    def __enter__(self):
        self.start = time.time()

    def __exit__(self, type, value, traceback):
        print(f'Profile results: {time.time() - self.start:.5f}s')

It inherits from the ContextDecorator class, which is the base class that enables the context manager to also be used as a decorator, and needs to be overridden__ enter__ And__ exit__ The function. Profile is used for timing, recording the current time when entering, and outputting the difference between the current time and start when exiting.

Timeout class

class Timeout(contextlib.ContextDecorator):
    # Usage: @Timeout(seconds) decorator or 'with Timeout(seconds):' context manager
    def __init__(self, seconds, *, timeout_msg='', suppress_timeout_errors=True):
        self.seconds = int(seconds)
        self.timeout_message = timeout_msg
        self.suppress = bool(suppress_timeout_errors)

    def _timeout_handler(self, signum, frame):
        raise TimeoutError(self.timeout_message)

    def __enter__(self):
        signal.signal(signal.SIGALRM, self._timeout_handler)  # Set handler for SIGALRM
        signal.alarm(self.seconds)  # start countdown for SIGALRM to be raised

    def __exit__(self, exc_type, exc_val, exc_tb):
        signal.alarm(0)  # Cancel SIGALRM if it's scheduled
        if self.suppress and exc_type is TimeoutError:  # Suppress TimeoutError
            return True

This class also inherits from ContextDecorator and is used to check whether it times out. seconds is the specified time, beyond which an exception will be thrown, timeout_msg is the exception output information. suppress is whether to strictly check. If strictly check, an exception will be thrown, but if not, the exception information will be output, and the program will continue to execute.

__ enter__ Method defines an alarm signal, which is initiated by signal.alarm. The parameter is seconds, that is, the time we want to guarantee. When the time exceeds the time we define, it will be called_ timeout_handler, the method throws an exception, executes_- exit__ Method__ exit__ First, turn off the alarm clock to prevent an error when the program exits normally. Next, check whether it is strictly checked. If yes, throw an exception. Otherwise, return directly and output exception information.

try_except function

def try_except(func):
    # try-except function. Usage: @try_except decorator
    def handler(*args, **kwargs):
        try:
            func(*args, **kwargs)
        except Exception as e:
            print(e)

    return handler

This function integrates the try and except modules. When you need to use this module, you can call this function directly instead of calling the try and ecept modules. func is the content to be executed.

methods function

def methods(instance):
    # Get class/instance methods
    return [f for f in dir(instance) if callable(getattr(instance, f)) and not f.startswith("__")]

This function returns all public methods of a class or instance into a list for later calls.

set_logging function

def set_logging(rank=-1, verbose=True):
    logging.basicConfig(
        format="%(message)s",
        level=logging.INFO if (verbose and rank in [-1, 0]) else logging.WARN)

Use the default formatter to create a StreamHandler and add it to the root logger to complete the basic configuration of the logging system. If there is no handler defined for the root logger, the debug(),info(),warning(),error() and critical() functions will automatically call basicConfig().

Format is that the handler uses the specified format string, and level sets the root logger level to the specified level. By default, the level of the generated root logger is logging.Warning. If it is lower than this level, it will not be output.

print_args function

def print_args(name, opt):
    # Print argparser arguments
    print(colorstr(f'{name}: ') + ', '.join(f'{k}={v}' for k, v in vars(opt).items()))

Output argparser parameters

init_seeds function

def init_seeds(seed=0):
    # Initialize random number generator (RNG) seeds https://pytorch.org/docs/stable/notes/randomness.html
    # cudnn seed 0 settings are slower and more reproducible, else faster and less reproducible
    import torch.backends.cudnn as cudnn
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    cudnn.benchmark, cudnn.deterministic = (False, True) if seed == 0 else (True, False)

Initialize the seed of the random generator. When seed is 0, set cudnn to slower, when reusability is higher, otherwise set to faster but low reusability.

get_latest_run function

def get_latest_run(search_dir='.'):
    # Return path to most recent 'last.pt' in /runs (i.e. to --resume from)
    last_list = glob.glob(f'{search_dir}/**/last*.pt', recursive=True)
    return max(last_list, key=os.path.getctime) if last_list else ''

last_list is a list returned by glob. glob.glob can return a list of all matching file paths. The parameter defines the file path matching rules, that is, find all the file paths of last**.pt, * * represents all other strings. Finally, the path closest to the current time is returned according to the creation time

user_config_dir function

def user_config_dir(dir='Ultralytics', env_var='YOLOV5_CONFIG_DIR'):
    # Return path of user configuration directory. Prefer environment variable if exists. Make dir if required.
    env = os.getenv(env_var)
    if env:
        path = Path(env)  # use environment variable
    else:
        cfg = {'Windows': 'AppData/Roaming', 'Linux': '.config', 'Darwin': 'Library/Application Support'}  # 3 OS dirs
        path = Path.home() / cfg.get(platform.system(), '')  # OS-specific config dir
        path = (path if is_writeable(path) else Path('/tmp')) / dir  # GCP and AWS lambda fix, only /tmp is writeable
    path.mkdir(exist_ok=True)  # make if required
    return path

Returns the path of the user configuration environment directory. If it does not exist, create one.