Python Advanced Software Technology

P1 sorting algorithm

1. Select Sort

1.1 algorithm steps

  1. First find the smallest (large) element in the unsorted sequence and store it at the beginning of the sort sequence
  2. Continue looking for the smallest (large) element from the remaining unsorted elements and place it at the end of the sorted sequence
  3. Repeat the second step until all elements are sorted

1.2 Code implementation

def selectionSort(arr):
    for i in range(len(arr) - 1):
        # Index of record minimum
        minIndex = i
        for j in range(i + 1, len(arr)):
            if arr[j] < arr[minIndex]:
                minIndex = j
        # If i is not the minimum, swap i with the minimum
        if i != minIndex:
            arr[i], arr[minIndex] = arr[minIndex], arr[i]
    return arr

2. Quick Sort

2.1 Algorithmic steps

  1. Pick out an element from a number of columns, called a pivot;
  2. Reorder the number columns, all elements smaller than the base value are placed in front of the base, and all elements larger than the base value are placed behind the base (the same number can be on either side). After the partition exits, the benchmark is in the middle of the column. This is called a partition operation;
  3. Recursively sorts subordinate columns of elements less than the base value and those greater than the base value

2.2 Code implementation

def quickSort(arr, left=None, right=None):
    left = 0 if not isinstance(left,(int, float)) else left
    right = len(arr)-1 if not isinstance(right,(int, float)) else right
    if left < right:
        partitionIndex = partition(arr, left, right)
        quickSort(arr, left, partitionIndex-1)
        quickSort(arr, partitionIndex+1, right)
    return arr

def partition(arr, left, right):
    pivot = left
    index = pivot+1
    i = index
    while  i <= right:
        if arr[i] < arr[pivot]:
            swap(arr, i, index)
            index+=1
        i+=1
    swap(arr,pivot,index-1)
    return index-1

def swap(arr, i, j):
    arr[i], arr[j] = arr[j], arr[i]
# Complete a round of exchange
def sub_sort(list_, low, high):
    temp = list_[low]  # Selected Baseline
    while low < high:
        while list_[high] >= temp and high > low:
            high -= 1
        list_[low] = list_[high]
        while list_[low] < temp and low < high:
            low += 1
        list_[high] = list_[low]
    list_[low] = temp
    return low


def quicksort(list_, low, high):
    """
    Quick Sort
    :param list_: List to Sort
    :param low: Index of the first element of the list
    :param high:Last element index
    :return:A sorted list
    """
    if low < high:
        key = sub_sort(list_, low, high)
        quicksort(list_, low, key - 1)
        quicksort(list_, key + 1, high)
    return list_

3. Insert Sort

3.1 Algorithmic steps

  1. Consider the first element of the first sequence to be sorted as an ordered sequence, and the second to the last element as an unsorted sequence.
  2. Unordered sequences are scanned from beginning to end, and each element scanned is inserted into the appropriate position in the ordered sequence. (If the element to be inserted is equal to an element in an ordered sequence, the element to be inserted is inserted after the equal element)

3.2 Code implementation

def insertionSort(arr):
    for i in range(len(arr)):
        preIndex = i-1
        current = arr[i]
        while preIndex >= 0 and arr[preIndex] > current:
            arr[preIndex+1] = arr[preIndex]
            preIndex-=1
        arr[preIndex+1] = current
    return arr

P2 IO Network Programming

1. Byte Strings and Strings

  1. Common ASCII encoding strings can be converted to byte strings by preceding b, for example, b'hello'
  2. String to byte string method: str.encode()
  3. Byte string to string method: bytes.decode()

2. File operations

After obtaining the file object, you can perform various operations on the file through the file object, which can call the following functions:

2.1 Read Files

  • read([size])
  • readline([size])
  • readlines([sizeint])

If the file object is read-fetched, it is an iterative object that iterates over each line of the file in a for loop

f = open('test', 'r')

while True:
    # Returns an empty string when reading to the end of the file
    data = f.read(100)  # Read up to 100 characters at a time
    if not data:
        break
    print(data)

2.2 with Operation

The with statement in python is used when accessing resources. It guarantees that no matter whether errors or exceptions occur during processing, a specified "clean up" operation will be performed to release the accessed resources, such as automatic closure of files after reading and writing, automatic acquisition and release of locks in threads, etc.

The syntax format of the with statement is as follows

with context_expression [as target(s)]:
		with-body

_With the with method, close() is not required, because the object generated by with is automatically processed after the statement block ends, so close is not necessary, but this file object can only be used within the with statement block

with open('file', 'r+') as f:
	r.read()

3. File offset

3.1 Definition

_When a file is opened for operation, a record is automatically generated that describes a series of our operations on the file. This includes the location of the file to which each operation takes place. Read and write operations start from this location.

3.2 Basic Operations

3.2.1 tell()

Function: Get file offset size

3.2.2 seek(offset[,whence])

_Function: Move file offset location
_parameter: offset represents the number of bytes moved relative to a location. Negative numbers mean moving forward, positive numbers mean moving backward. When is the base location, the default value is 0, which means counting from the beginning of the file, 1 from the current location, and 2 from the end of the file.
_Note: The base position can only be 1 or 2 when the file must be opened binary

4. File descriptors

4.1 Definition

Each IO operation in the system is assigned an integer as the number, which is the file descriptor of the IO operation.

4.2 Get the file descriptor

fileno():
_Get the corresponding file descriptor from the IO object

4.3 File Management Functions

  1. Get file size
    os.path.getsize(file)
  2. View File List
    os.listdir(dir)
  3. See if the file exists
    os.path.exists(file)
  4. Determine File Type
    os.path.isfile(file)
  5. Delete Files
    os.remove(file)

5. Network Model

5.1 OSI Seven Layer Model

Formulate organization
_ISO (International Organization for Standardization)
Effect
_Standardize network communication workflow

Application layer: Provide user services, specific functions implemented by applications
Representation layer: data compression optimizes encryption
Session Layer: Establish user-level connections and select appropriate transport services
Transport Layer: Provide transport services
Network layer: routing, network connectivity
Link Layer: Conduct data exchange to control the sending of specific data
Physical Layer: Provides hardware assurance for data transmission, network card interface, transmission media

Advantage:

  1. A unified workflow has been established
  2. Clear division, clear division of duties, clear division of work in each step
  3. Reduce coupling between modules for easy development

5.2 TCP/IP Four-Layer Model

  • application layer
  • transport layer
  • Internet Layer
  • network interface

Network Protocol

In the network data transmission, rules are followed, including what kind of data structure to establish, what kind of special marks, and so on.

5.3 Basic Network Concepts

IP Address

  • Function: Determine a host's network routing location
  • View local network address command: ifconfig

domain name

  • Definition: Name the address of the network server
  • Function: Easy to remember, express certain meaning
  • ping[ip]: test whether a host is connected

Port number

  • Role: Ports are part of a network address used to distinguish between different network applications on a host
  • Features: Application listening ports in a system cannot be duplicated
  • Value range: 1 - 65535
  • 1-1023 System Application or Popular Program Listening Port
  • 1024-65535 self-use port

5.4 Transport Layer Service

5.4.1 Connection-oriented Transport Service (TCP-based data transfer)

Transmission characteristics:

  • Provides reliable data transmission, reliability index data transmission process without loss, disorder, error, duplication

Means of implementation:

  • Data connection needs to be established before communication ends and disconnects normally

Three handshakes (make connection)

  1. Client sends message message message to server requesting connection
  2. After the server receives the request, the reply confirms that it can connect
  3. Client receives reply, sends final message connection establishment

Four wave (disconnect)

  1. The active party sends a message requesting disconnection
  2. When the passive party receives the message, it immediately replies, indicating that it is ready to disconnect
  3. The passive party is ready to send the message again to indicate that it can be disconnected
  4. The active party receives the confirmation, sends the final message and finishes disconnecting

5.4.2 Connectionless Transport Services (UDP-based data transfer)

Transmission characteristics:

  • The reliability of transmission is not guaranteed, the transmission process is not connected or disconnected, and the data is sent and received freely.

Where applicable:

  • The network is poor and the reliability of transmission is not high. Examples: web video, group chat, radio

5.5 socket socket programming

Introduction to 5.5.1 sockets

Socket:

  • A Technical Mean to Implement Network Programming for Data Transfer

Python implements socket programming:

  • import socket
  • from socket import socket

Socket Classification

  1. Streaming socket (SOCK_STREAM): Transmit data in byte stream for tcp network transmission
  2. Datagram socket: Transmit data as datagram to implement udp network transmission scheme.

5.5.2 tcp socket

Service-side process:
  socket–>bind–>listen–>accept–>send/recv–>close
1. Create sockets

sockfd = socket.socket(socket_family = AF_INET, socket_type = SOCK_STREAM, proto=0)
# Function: Create socket
# Parameter socket_family 	 Network Address Type 	 AF_INET stands for ipv4
# Parameter socket_type 	 socket type 	 SOCK_ STREAM (Streaming)
# proto 	 Usually 0 	 Select Subprotocol

2. Binding Address

  • Local address: localhost 127.0.0.1
  • Network address: 172.40.91.185
  • Automatic Address Acquisition: 0.0.0.0
sockfd.bind(addr)
# Function: Bind local network address
# Parameter: Binary tuple (ip, port) 	 ('0.0.0.0', 8888)


3. Set up monitoring

sockfd.listen(n)
# Function: Set the socket as a listening socket, determine the size of the listening queue
# Parameter: listening queue size

4. Waiting to process client connection requests

connfd, addr = sockfd.accept()
# Function: Block waiting to process client requests
# Return value: connfd 	 Client Connection Socket
# addr 	 Connected Client Address

5. Message Receiving and Sending

data = connfd.recv(buffersize)
#Function: Receive client messages
#Parameter: Size of the maximum message received at a time
#Return value: Received content

n = connfd.send(data)
#Function: Send message
#Parameter: What to send 	 bytes format
#Return value: number of bytes sent

6. Close the socket

sockfd.close()
# Function: close socket

Code Samples

import socket

# Create Socket Object
import unicodedata

sockfd = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# Binding Address
sockfd.bind(('127.0.0.1', 8888))

# Set Listening Socket
sockfd.listen(5)

while True:

    # Block waiting for client connection
    print("Waiting for connect")
    try:
        connfd, addr = sockfd.accept()
        print("Connect from", addr)
    except KeyboardInterrupt:
        print("Server Exit")
        break
    except Exception as e:
        print(e)
        continue

    # Send and receive messages
    while True:
        data = connfd.recv(1024)
        if not data:  # Empty data indicates client exit
            break
        print("Received:", data.decode())
        n = connfd.send(b'Thanks')
        print("Send out%d byte" % n)

    # Close
    connfd.close()
sockfd.close()

Client Process
_socket ->bind < optional > (generally not written) - connect ->send/recv ->close

1. Create sockets

  • Note: Only sockets of the same type can communicate

2. Connect to the server

sockfd.connect((server_addr,port))
# Function: Connect to server
# Parameter: Tuple 	 server address

3. Send and receive messages

  • Note: to prevent blocking at both ends, recv send should work together

4. Close the socket

Code Samples

"""
tcp_client.py   tcp Client Process
"""

from socket import *

# Create tcp socket
sockfd = socket()  # Use default parameter-->tcp socket

# Connect to Server
server_addr = ('127.0.0.1', 8888)

sockfd.connect(server_addr)

# Send and receive messages
while True:
    data = input("Msg>>")
    if not data:
        break
    sockfd.send(data.encode())
    data = sockfd.recv(1024)
    print("Server:", data.decode())  # Print Received Content

# Close
sockfd.close()

5.5.3 tcp socket data transmission characteristics

_tcp connection when one end exits and the other is blocked in recv, recv immediately returns an empty string
BrokenPipeError in tcp connection if one end no longer exists and still attempts to send via send
A listening socket can connect multiple clients at the same time, or it can be connected repeatedly

5.5.4 Network Transceiver Buffer

  1. Network buffers effectively coordinate the sending and receiving speed of messages
  2. Send and recv actually send receive messages to the buffer, and recv will not block if the buffer is not empty

5.5.5 tcp sticker

Reason:

  • tcp is transmitted as a byte stream without message boundaries. Messages sent multiple times are received once, and a sticky packet is formed

Influence:

  • If each sent content has a separate meaning, the glue will have an impact when the receiver needs to resolve it independently

Processing method:

  1. Artificial Add Message Boundary
  2. Control Send Speed

5.5.6 UDP Sockets

Service-side process
  socket–>bind–>recvfrom–>sendto–>close

1. Create a datagram socket

sockfd = sock(AF_INET, SOCK_DGRAM)

2. Binding Address

sockfd.bind(addr)

3. Message Receiving and Receiving

data, addr = sockfd.recvfrom(buffersize)
# Function: Receive UDP messages
# Parameter: Maximum number of bytes received at a time
# Return value:data 	 Received content; 	 addr 	 Message sender address

n = sockfd.sendto(data, addr)
#Function: Send UDP message
#Parameters:data 	 Sent content (bytes format); 	 addr destination address
#Return value: number of bytes sent

4. Close the socket

sockfd.close()

Service-side code example

"""
udp_server.py   udp Socket server process
"""

import socket

# Create UDP socket
sockfd = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

# Binding Address
server_addr = ('127.0.0.1', 8888)
sockfd.bind(server_addr)

# Receive and receive messages in a loop
while True:
    data, addr = sockfd.recvfrom(1024)
    print("Received message:", data.decode())
    sockfd.sendto(b'Thanks', addr)

# Close
socket.close()

udp client code example

from socket import *

# server address
ADDR = ('127.0.0.1', 8888)

# Create Socket
sockfd = socket(AF_INET, SOCK_DGRAM)

# Receive and receive messages in a loop
while True:
    data = input("Msg>>")
    if not data:
        break
    # Send a message to the server
    sockfd.sendto(data.encode(), ADDR)
    # receive messages
    msg, addr = sockfd.recvfrom(1024)
    # Print received messages
    print("From server:", msg.decode())

sockfd.close()

Differences between 5.5.7 tcp sockets and udp sockets

  1. Streaming sockets transfer data as byte streams, and datagram sockets transfer data as datagrams
  2. tcp sockets will stick, udp sockets will not stick if they have message boundaries
  3. tcp sockets guarantee message integrity, udp sockets do not
  4. tcp sockets rely on listen accept to establish a connection to send and receive messages, udp sockets do not need
  5. tcp socket uses send, recv to send and receive messages, udp socket uses sendto, recvfrom

5.5.8 socket socket properties

  1. sockfd.type socket type
  2. sockfd.family socket address type
  3. sockfd.getsockname() Gets the socket binding address
  4. sockfd.fileno() Gets the file descriptor of the socket
  5. sockfd.getpeername() Gets the connection socket client address
  6. sockfd.setsockopt(level,option,value)
    _Function: Setting socket options
    _Parameter: level Option Category SOL_SOCKET
    Selection Specific Options
    Value option value
  7. sockfd.getsockopt(level, option)
    _Function: Get socket option value

Code Samples

"""
Introduction to socket properties
"""

from socket import *

# Create Socket
s = socket()

# Set the port for immediate reuse
s.setsockopt(SOL_SOCKET, SO_REUSEADDR, True)    

s.listen(3)

c, addr = s.accept()
print("Connection End Address:",c.getpeername())

print("Address type:",s.family)
print("Socket type:",s.type)
print("Binding Address:",s.getsockname())
print("File Descriptor:",s.fileno())

5.5.8 UDP Radio

  1. Broadcast Definition: One End Sends Multipoint Receive
  2. Broadcast Address: The maximum address of each network is the address where the broadcast is sent, and when sent to that address, all hosts in the network segment can receive it.

Code Samples

"""
Broadcast reception	broadcast_recv.py
	1.Establish udp socket
	2.Set up a socket to send and receive broadcasts ( setsockopt)
	3.Select the port to receive
	4.Receive radio
"""

from socket import *

s = socket(AF_INET, SOCK_DGRAM)

# Set up socket to receive broadcasts
s.setsockopt(SOL_SOCKET, SO_BROADCAST, True)  # Set up a socket to receive broadcasts

s.bind(('0.0.0.0', 9999))

msg, addr = s.recvfrom(1024)
print(msg.decode())
"""
Broadcast Send	broadcast_send.py
	1.Establish udp socket
	2.Set up to send broadcasts
	3.Loop to broadcast address
"""

from socket import *
from time import sleep

# Address of broadcast address ifconfig
dest = ('192.168.0.161', 9999)

# Create Datagram Socket
s = socket(AF_INET, SOCK_DGRAM)

# Set up to send broadcasts
s.setsockopt(SOL_SOCKET, SO_BROADCAST, 1)

data = """
    ******************
    Beijing 11.17   Early winter
    Temperature: 7
    Status: No four five chicks
    ******************
"""

sleep(2)  # Send every two seconds
s.sendto(data.encode(), dest)  # The destination address is the broadcast address

HTTP Transport for 5.5.9 TCP Sockets

1. HTTP Protocol (Hypertext Transfer Protocol)
  1. Purpose: Web page acquisition, data transfer
  2. Characteristic:
    _Application Layer Protocol, Transport Layer uses tcp transport
    _Simple, flexible, many languages have HTTP-specific interfaces
    _Stateless, protocol does not record transmission content
    _http1.1 supports persistent connections, enriching request types
2. HTTP request
  • Request line: specific request category and content

 GET      /      HTTP/1.1
Request Category_Request Content_Protocol Version

  • Request Header: A further explanation and description of the request in the form of one key-value pair per line

Accept-Encoding: gzip

  • Blank Line
  • Requestor: Request parameters or submissions
3. HTTP response
  • Response line: Feedback basic response

HTTP/1.1    200    OK
Version Information_Response Code_Additional Information

  • Response Header: Description of response content

Content-Type: text/html

  • Blank Line
  • Response body: The body content information of the response

Code examples for http requests and responses

"""
httpserver v1.0
 Basic requirements:
    Get requests from browsers
    Determine if the content of the request is/,take index.html Response to Client
    Response 404 if something else is returned
"""

from socket import *


# Client (Browser) Processing
def request(connfd):
    # Get the request and extract the content of the request

    data = connfd.recv(4096)  # Receive requests
    if not data:  # Prevent browser from exiting abnormally
        return
    # print(data.decode(), type(data.decode()))

    request_line = data.decode().split('\n')[0]  # Get Request Line
    # print(request_line, type(request_line))   # Print Request Line

    info = request_line.split(" ")[1]  # Get Request Content
    # print(info)  # Print Request Content

    # Judgment Yes/Returns index.html, not/Returns 404
    if info == "/":  # Yes/then return index.html
        with open('index.html') as f:  # Open index.html file with with
            response = "HTTP/1.1 200 OK\r\n"  # Response line
            response += "Content-Type:text/html\r\n"  # Response Header (Writable)
            response += "\r\n"  # Blank Line
            response += f.read()  # Response Body
    else:  # No/Return 404
        response = "HTTP/1.1 404 Not Found\r\n"  # Response line
        response += "Content-Type:text/html\r\n"  # Response Header (Writable)
        response += "\r\n"  # Blank Line
        response += "<h1>Sorry...</h1>"  # Response Body

    # Send response content to browser
    connfd.send(response.encode())


# Set up tcp network
sockfd = socket()

# Set the port for immediate use
sockfd.setsockopt(SOL_SOCKET, SO_REUSEADDR, True)

# Binding Address
sockfd.bind(('0.0.0.0', 8000))

# Set socket to listen on socket
sockfd.listen(3)

while True:
    connfd, addr = sockfd.accept()
    request(connfd)  # Processing client requests

Use of 5.5.10 struct module

  1. Principle: Package a set of simple data and send it in bytes format. Or parse a set of bytes formatted data
  2. Interface use
import struct
st = struct.Struct(fmt)
# Function: Generate structured objects
# Parameter: fmt 	 Customized data structure


st.pack(v1, v2, v3...)
# Function: Converts a set of data packaged in a specified format to bytes
# Parameter: Data to be packaged
# Return value: bytes byte string


st.unpack(bytes_data)
# Function: parse bytes string in specified format
# Parameter: Byte string to parse
# Return value: parsed content

struct.pack(fmt, v1, v2, v3,...)
struct.unpack(fmt, bytes_data)
# Note: You can use the struct module to call pack unpack directly. In this case, the first parameter of the two functions is passed into fmt. Other usage functions are the same
# Note: What format (fmt) is packaged when packaging and what format is parsed when parsing

Demonstrate in terminal:

In [1]: import struct

In [2]: st = struct.Struct('i4sf')

In [3]: data = st.pack(1, b'Lily', 1.65)

In [4]: data
Out[4]: b'\x01\x00\x00\x00Lily33\xd3?'

In [5]: st.unpack(data)
Out[5]: (1, b'Lily', 1.649999976158142)

Exercises for struct module

Complete with udp

  • Enroll student information on the client side and send it to the server
  • Write student information into a file on the server side, one line for each student
  • Information format: id(int) name(str) age(int) score(float)
"""
struct_send.py
"""

from socket import *
import struct

# Define the data format first
st = struct.Struct('i32sif')

# udp socket
s = socket(AF_INET, SOCK_DGRAM)
ADDR = ('127.0.0.1', 8888)

while True:
    print("=====================")
    id = int(input("ID:"))
    name = input("Name:").encode()
    age = int(input("Age:"))
    score = float(input("Score:"))
    # Data Packaging Send
    data = st.pack(id, name, age, score)
    s.sendto(data, ADDR)
"""
struct_recv.py
"""

from socket import *
import struct

# Define the same data format on both sides
st = struct.Struct('i32sf')

# Create udp socket
s = socket(AF_INET, SOCK_DGRAM)

# binding
s.bind(('127.0.0.1', 8888))

# Open File
f = open('student.txt', 'a')

while True:
    data, addr = s.recvfrom(1024)
    data = st.unpack(data)

    # write file
    info = "%d  %-10s   %d  %.1f\n" % data
    f.write(info)
    f.flush()

6. Network Concurrency

6.1 Multitask Programming

  1. Significance: Make full use of computer multi-core resources to improve program efficiency
  2. Implementation scenario: multi-process, multi-threaded
  3. Parallel and Concurrent
    _Concurrent: Processing multiple tasks at the same time, the kernel continuously switches between tasks to achieve the effect of as if multiple tasks were executed at the same time, in fact, only one task occupies the kernel at a time
    _Parallel: Multiple tasks are executed simultaneously using computer multi-core resources, where the relationship between multiple tasks is parallel

6.2 process

1. Definition: A program is run once on a computer
_Program is an executable file, it is a static occupying disk
_Process is a dynamic description of the process, occupies computer running resources, has a certain life cycle

2. How a process is generated in the system
[1] User space initiates requests by calling program interfaces or commands
[2] The operating system accepts user requests and starts creating processes
[3] The operating system allocates computer resources, determines process status, etc.
[4] The operating system provides the processes created for the user to use

3. Basic concepts of processes

_cpu time slice: If a process occupies the CPU core, it is said that the process is on the CPU time slice

_PCB (Process Control Block): A space in memory that stores basic information about a process and is also used for system lookups to identify processes

_Process ID(PID): The system assigns an integer greater than 0 to each process as the process ID. Do not duplicate each process ID_Linux view process ID:ps-aux

Parent-Child Processes: Each process in the system (except the system initialization process) has a unique parent process and can have zero or more child processes. Parent-child process relationships are easy to manage. View the process tree: pstree

_Process state (three states):
_Ready state: Process is ready to execute, waiting to allocate CPU resources
Runtime: Processes holding CPU time slices are running
Waiting State: Process is temporarily stopped and CPU is relinquished

_five states (new and terminated on the basis of three states):
New: Create a process, get resources
Termination: The process that ends, the process that releases resources

Status View Command: ps-aux -->STAT Column
Waiting S tate
_R_Runtime
Waiting State
Waiting State
Z ombies

<has higher priority
Low priority
Foreground process
Session Group Leader
Multithreaded

2. Running characteristics of processes
[1] Processes can use computer multi-core resources
[2] Process is the smallest unit of computer resources allocated
[3] The processes run independently of each other.
[4] Each process has its own space and uses its own space resources

6.3 fork-based multi-process programming

"""fork Create Process Demo"""

import os

pid = os.fork()
# Function: Create a new process
# Return value: Integer, returns a negative number if the creation process fails, returns the PID of the new process in the original process if successful, and returns 0 in the new process

if pid < 0:
    print("Create process failed")

#Part of subprocess execution
elif pid == 0:
    print("The new progress")

# Part of parent process execution
else:
    print("The old process")
    
    
# Both parent and child processes execute
print("Fork test over")

Be careful:

  • The child process copies all the memory space of the parent process, starting with the next sentence of fork
  • Parent-child processes run independently, in varying order
  • Using the difference of parent-child process fork return value, it is almost fixed to match if structure to allow parent-child process to perform different content
  • Parent-child processes have their own unique features such as PID, PCB, command set, etc.
  • Spatial and child processes opened up before the parent process fork also have the same ownership, and parent-child processes do not influence each other's operations in their own space

6.4 Process-related functions

os.getpid()
# Function: Get the PID value of a process
# Return value: Returns the PID of the current process

os.getppid()
# Function: Get the PID of the parent process
# Return value: Returns the PID of the parent process

os._exit(status)
# Function: End a process
# Parameter: Termination state of the process

sys.exit([status])
# Function: Exit process
# Parameter: Integer denotes exit status
# If the parameter passes a string: indicating what to print on exit

Code Samples

import os, sys

pid = os.fork()

if pid < 0:
    print("Error")
elif pid == 0:
    print("Child PID:", os.getpid())  # SubPID
    print("Parent PID:", os.getppid())  # Parent PID
else:
    print("Get Child PID:", pid)  # SubPID
    print("Parent PID:", os.getpid())  # Parent PID

os._exit(0)  # Exit the process, the following statements will not be executed
sys.exit("Sign out")  # Exit the process and print the passed string

print("Exit")  # Will not execute

6.5 Orphans and Zombies

1. Orphan process: The parent process exits before the child process, when the child process becomes an orphan process
Features: The orphan process will be adopted by the system process, at which time the system process will become the new parent of the orphan process, the orphan process will be automatically processed to exit the process

2. Zombie process: When a child process exits before the parent process, and the parent process does not handle the exit state of the child process, the child process is called a zombie process.
Features: Although the zombie process ends, it will retain part of the PCB in memory, a large number of zombie processes will waste the system's memory resources

3. How to Avoid Zombie Processes

  1. Use wait function to handle child process exit
pid, status = os.wait()
# Function: Block in parent process waiting for processing child process to exit
# Return value: pid of the child process that pid exits; Status 	 Subprocess Exit Status

Code Samples

import os, sys

pid = os.fork()
if pid < 0:
    print("Error")
elif pid == 0:
    print("Child PID:", os.getpid())
    sys.exit("Subprocess Exit")
else:
    """
    os.wait() Dealing with zombie processes
    """
    pid, status = os.wait()
    print("pid:", pid)
    print("status:", status)  # The printed status is the exit status of the child process multiplied by 256

    while True:  # Let the parent process not exit
        pass
  1. Create a secondary subprocess to handle the zombies
    _Parent process creates child process, waiting to recycle child process
    _Subprocess Creates a secondary subprocess and exits
    _Second-level child processes are called orphans and execute events with the parent of first-level child processes

  2. Exit by signal processing subprocess
    Principle: When a child process exits, it sends a signal to the parent process. If the parent process ignores the child process signal, the system automatically processes the child process exit
    _Method: Use the signal module to write the following statement before a parent process creates a child process

import signal
signal.signal(signal.SIGCHLD,signal.SIG_IGN)

Sample code

"""Signal Processing Zombies"""
import os, sys
import signal

# When a child process exits, the parent process ignores the exit behavior and the child process is handled by the system
signal.signal((signal.SIGCHLD, signal.SIG_IGN))

pid = os.fork()
if pid < 0:
    print("Error")
elif pid == 0:
    print("Child PID:", os.getpid())
    sys.exit(2)
else:
    while True:  # Parent process does not exit
        pass

6.6 Group Chat Room

Requirements:

  1. When someone enters a chat room, they need to enter their name. The name cannot be repeated.
  2. When someone enters the chat room, others will be notified that XXX enters the chat room
  3. One person sends a message, others receive: XXX:xxxxxxxxxxx
  4. When someone exits the chat room, others will be notified that XXX exits the chat room
  5. Extension: Server can send announcements to all users: Administrator Message: XXXX

code implementation

Client Code

"""chat_client.py"""
import os
import sys
from socket import *

# server address
ADDR = ('127.0.0.1', 8888)


# send message
def send_msg(s, name):
    while True:
        try:
            text = input("Speeches:")
        except KeyboardInterrupt:
            text = 'quit'
        if text.strip() == 'quit':
            msg = 'Q' + ' ' + name
            s.sendto(msg.encode(), ADDR)
            sys.exit("Exit Chat Room")
        msg = 'C %s %s' % (name, text)
        s.sendto(msg.encode(), ADDR)


# receive messages
def recv_msg(s):
    while True:
        try:
            data, addr = s.recvfrom(4096)
        except KeyboardInterrupt:
            sys.exit()
        if data.decode() == 'EXIT':
            sys.exit()
        print(data.decode() + '\n Speeches:', end='')


# Client startup function
def main():
    s = socket(AF_INET, SOCK_DGRAM)

    # Enter the chat room
    while True:
        name = input("Please enter your name:")
        msg = 'L' + ' ' + name
        s.sendto(msg.encode(), ADDR)
        # Receive feedback from the server
        data, addr = s.recvfrom(128)
        if data.decode() == 'OK':
            print("You have entered the chat room")
            break
        else:
            print(data.decode())
    # Exit the loop to indicate that you have entered the chat room
    pid = os.fork()
    if pid < 0:
        sys.exit("Error!")
    elif pid == 0:
        send_msg(s, name)  # Subprocess sends message
    else:
        recv_msg(s)  # Parent process receives messages


main()

Service-side code

"""chat_server"""

from socket import *
import os, sys

# server address
ADDR = ('0.0.0.0', 8888)

# Store user information
user = {}


# Sign in
def do_login(s, name, addr):
    if name in user:
        s.sendto("The user already exists".encode(), addr)
        return
    s.sendto(b'OK', addr)
    # Notify Others
    msg = "\n Welcome%s Enter the chat room" % name
    for i in user:
        s.sendto(msg.encode(), user[i])
    user[name] = addr


# chat
def do_chat(s, name, text):
    msg = "\n%s:  %s" % (name, text)
    for i in user:
        if i != name:
            s.sendto(msg.encode(), user[i])


# Sign out
def do_quit(s, name):
    msg = "\n%s Exit Chat Room" % name
    for i in user:
        if i != name:
            s.sendto(msg.encode(), user[i])
        else:
            s.sendto(b'EXIT', user[i])
    del user[name]


# Processing Request
def do_request(s):
    while True:
        data, addr = s.recvfrom(1024)
        temp = data.decode().split(' ')
        if temp[0] == 'L':
            do_login(s, temp[1], addr)
        elif temp[0] == 'C':
            text = ' '.join(temp[2:])
            do_chat(s, temp[1], text)
        elif temp[0] == 'Q':
            do_quit(s, temp[1])


# Set up a network
def main():
    # udp server
    s = socket(AF_INET, SOCK_DGRAM)
    s.setsockopt(SOL_SOCKET, SO_REUSEADDR, True)
    s.bind(ADDR)

    pid = os.fork()
    if pid == 0:
        while True:
            msg = input("Administrator message:")
            msg = "C Administrators " + msg
            s.sendto(msg.encode(), ADDR)

    # Request Handling Function
    do_request(s)


main()

6.7 multiprocessing module creation process

6.7.1 Process Characteristics

  1. Encapsulate events that require subprocesses to execute as functions
  2. Create process objects, associated functions, from the module's Process class
  3. Process information and properties can be set through process objects
  4. start a process by calling it from a process object
  5. Call join recycling process through process object

6.7.2 Basic interface use


Be careful:

  • Start the process and the target binding function starts executing, which executes as a child process, when the process is actually created

    Be careful:
  1. Creating a process with multiprocessing is also a child process that copies the parent process space code snippet, and parent-child processes run independently of each other
  2. The child process only runs the function part of the target binding, the rest of which is the parent process execution
  3. A parent process in multiprocessing is often used only to create a child process to recycle a child process, and the specific event is completed by the child process
  4. Standard input cannot be used in a subprocess created by multiprocessing (input cannot be used in a subprocess)

6.7.3 multiprocessing module creation process

"""
multiprocessing Module Creation Process
1.Writing process functions
2.Generate process objects
3.Start process
4.Recycling process
"""

import multiprocessing as mp
from time import sleep


# Create process functions
def fun():
    print("Start a process")
    sleep(5)
    print("End of child process")


# Create Process Object
p = mp.Process(target=fun)  # p This object can represent a process
p.start()  # Start process

# Parent Process Event
sleep(3)
print("Part of parent process execution is written in start and join between")

p.join()  # Recycling process

6.7.4 multiprocessing module to create multiple processes

"""
multiprocessing Create multiple processes
"""
from multiprocessing import Process
from time import sleep
import os


def th1():
    sleep(3)
    print("Having dinner")
    print(os.getppid(), '--', os.getpid())


def th2():
    sleep(2)
    print("Sleep")
    print(os.getppid(), '--', os.getpid())


def th3():
    sleep(4)
    print("Bean Bean")
    print(os.getppid(), '--', os.getpid())


things = [th1, th2, th3]
jobs = []

for th in things:
    p = Process(target=th)
    jobs.append(p)  # Save process objects from a list
    p.start()

# Recycle all subprocesses together
for i in jobs:
    i.join()

Result of running the code above

6.7.5 Pass-on to process functions

"""
Process Pass-through to process function
"""

from multiprocessing import Process
from time import sleep


# Process functions with parameters
def worker(sec, name):
    for i in range(3):
        sleep(sec)
        print("I'm %s" % name)
        print("I'm working...")


# p = Process(target=worker, args=(2, 'Baron'))  # Pass by location: 2 to sec, Baron to name
# p = Process(target=worker, kwargs={'sec': 2, 'name': 'Baron'})  # Pass by keyword
p = Process(target=worker, args=(2,), kwargs={'name': 'Baron'})

p.start()
p.join()

6.7.6 Process Object Properties

  • p.name process name
  • The PID number of the p.pid subprocess
  • p.is_alive() to see if a child process is in the lifecycle
  • p.daemon sets parent-child process exit relationships
    _If set to True, the child process ends with the parent process exiting
    _Requires that must be set before start
    _If daemon is set to True, join() will not normally be used
"""
Process Object Properties
"""
import time
from multiprocessing import Process


def tm():
    for i in range(3):
        time.sleep(2)
        print(time.ctime())


p = Process(target=tm, name="pname")

p.daemon = True  # Child process exits when parent process exits

p.start()
print("Process name:", p.name)  # Default name Process-1, assigned pname
print("process ID: ", p.pid)  # Get the PID of the corresponding subprocess
print("is alive: ", p.is_alive())  # Check to see if the process is in the lifecycle, True is not in the lifecycle False

6.8 Process Pool

6.8.1 Necessity

_The process of creation and destruction consumes more resources, and when there are many tasks and each task is completed in a very short time, frequent creation and destruction processes are required. At this time, the pressure on the computer is high, and the process pool technology solves the above problems well.

6.8.2 Principle

_Create a number of processes to handle events. When the process is finished, it does not exit but continues to process other events until all events have been processed and destroyed uniformly. Increase process reuse and reduce resource consumption

Implementation of 6.8.3 Process Pool

  1. Create process pool objects and place appropriate processes
from multiprocessing import Pool
Pool(processes)
# Function: Create process pool object
# Parameters: Specify the number of processes, automatically determined by default based on the system
  1. Queue events to process pool for execution
pool.apply_async(func,args,kwargs)
# Function: Use process pool to execute func events
# Parameter: func event function
# Parameter: args tuple passes positional arguments to func
# Parameters: The kwargs dictionary passes keywords to func
# Return value: Return function event object
  1. Close process pool
pool.close()
# Function: Close process pool
  1. Recycle processes in process pool
pool.join()
# Function: Recycle processes in the process pool

6.8.4 Code Samples

"""
Process pool usage example
"""

from multiprocessing import Pool
from time import ctime, sleep


# Process pool events
def worker(msg):
    sleep(2)
    print(ctime(), '--', msg)


# Create process pool
pool = Pool(4)  # If no parameters are written, the number of processes in the process pool is determined by the system by default

# Add event to process pool queue
for i in range(10):
    msg = "Tedu %d" % i
    pool.apply_async(func=worker, args=(msg,))

# Close process pool
pool.close()

# Recycle process pool
pool.join()

Run Results

6.9 Interprocess Communication (IPC)

1. Necessity:
_Inter-process space is independent, resources are not shared, at this time, when data transfer between processes is required, specific means of data communication are needed.
2. Common methods of interprocess communication:
Pipeline_Message Queue_Shared Memory_Signal_Semaphore_Socket

6.9.1 Pipeline Communication

  1. Communication principle
    _Open up pipeline space in memory, generate pipeline operation objects, multiple processes can use the same pipeline object to read and write to achieve communication
  2. Implementation Method
from multiprocessing import Pipe
fd1,fd2 = Pipe(duplex=True)
# Function: Create Pipeline
# Parameter: Default for two-way pipe, if False for one-way pipe
# Return value: A read-write object representing both ends of the pipe
        #Readable and Writable in Bidirectional Representation
        #If it is a one-way pipe fd1 read-only fd2 write-only

fd.recv()
# Function: Get content from pipeline

fd.send(data)
# Function: Write content to pipe
# Parameter: Data to be written 
  1. Code Samples
"""
pipe.py pipeline communication
 Be careful:
    1.multiprocessing Medium pipeline communication can only be used in relational processes
    2.Pipeline objects are created in the parent process, and child processes are acquired through the parent process
"""

from multiprocessing import Process, Pipe

# Create Pipeline
fd1, fd2 = Pipe()


def app1():
    print("start-up app1,Please login")
    print("request app2 To grant authorization")
    fd1.send("app1 Request login")  # Write Pipeline
    data = fd1.recv()
    if data:
        print("Logon success:", data)


def app2():
    data = fd2.recv()  # Block waiting to read pipe contents
    print(data)
    fd2.send(('Dave', '123'))  # You can send any type of Python type data, where tuples are sent


p1 = Process(target=app1)
p2 = Process(target=app2)
p1.start()
p2.start()
p1.join()
p2.join()

Run Screenshot

6.9.2 Message Queue

  1. Communication principle
    _Build a queue model in memory where processes either store messages in or take them out to complete interprocess communication
  2. Implementation Method
from multiprocessing import Queue

q = Queue(maxsize=0)
# Function: Create Queue Objects
# Parameter: Maximum number of messages to store
# Return value: Queue object

q.put(data,[block,timeout])
# Function: Save messages to queue
# Parameter: whether the content block settings of the data to be stored are blocked (False is non-blocked) timeout detection

q.get([block,timeout])
# Function: Remove messages from queue
# Parameter: block setting whether blocking (False is non-blocking) timeout detection
# Return value: Return what you get

q.full()    #Determine if the queue is full
q.empty()   #Determine if the queue is empty
q.qsize()   #Determine the number of messages in the queue
q.close()   #Close Queue 
  1. Code Samples
"""
queue_0.py  Message Queue Demo
 Be careful:
    Message queue complies with FIFO principle
"""

from multiprocessing import Queue, Process
from time import sleep
from random import randint

# Create Message Queue
q = Queue(5)  # Store up to five messages


def handle():
    for i in range(6):
        x = randint(1, 33)
        q.put(x)  # Message Entry
    q.put(randint(1, 16))


def request():
    list_ = []
    for i in range(6):
        list_.append(q.get())
    list_.sort()
    list_.append(q.get())
    print(list_)


p1 = Process(target=handle)
p2 = Process(target=request)
p1.start()
p2.start()
p1.join()
p2.join()

6.9.3 Shared Memory

  1. Communication principle
    _Opens up a space in memory where processes can write and read content to complete communication, but each write will overwrite the previous content
  2. Implementation Method
from multiprocessing import Value, Array

obj = Value(ctype, data)
# Function: Open up shared memory
# Parameter: ctype denotes shared memory space type'i''f''c'
      # Data shared memory space initial data  
# Return value: Shared memory object

obj.value  # Viewing modifications to this property reads and writes to shared memory

obj.Array(ctype, data)
# Function: Open up shared memory space
# Parameter: ctype represents shared memory data type
      # The data integer represents the size of the open space
# Return value: Shared memory object
"""
value.py    Open up a single shared memory space
 Note: Shared memory can only have one value
"""

from multiprocessing import Process, Value
import time
import random

# Create Shared Memory
money = Value('i', 5000)


# Operational shared memory
def man():
    for i in range(30):
        time.sleep(0.2)
        money.value += random.randint(1, 1000)


def girl():
    for i in range(30):
        time.sleep(0.15)
        money.value -= random.randint(100, 800)


p1 = Process(target=man)
p2 = Process(target=girl)
p1.start()
p2.start()
p1.join()
p2.join()

# Get Shared Memory
print("One month balance", money.value)
"""
array.py    Shared memory holds a set of data
"""

from multiprocessing import Process, Array

# Create Shared Memory
# shm = Array('i', [1, 2, 3, 4])
# shm = Array('i', 5)  # Initially open up five reshaping spaces
shm = Array('c', b'hello')


def func():
    # Shared memory objects created by array can be iterated
    for i in shm:
        print(i)


p = Process(target=func)
p.start()
p.join()

6.9.4 semaphores

  1. Communication principle
    Given a number, it is visible to multiple processes. Multiple processes can operate on the increase or decrease of this number and determine their own behavior based on the value of the number.
  2. Implementation Method
from multiprocessing import Semaphore

sem = Semaphore(num)
# Function: Create semaphore object
# Parameter: Initial value of semaphore
# Return value: semaphore object

sem.acquire()	# Reduce semaphore by 1 Blocking when semaphore is 0
sem.release()	# Add semaphore to 1
sem.get_value()	# Get the number of semaphores
  1. Code Samples
"""
sem.py  Semaphore demonstration
 Idea: The number of semaphores is equivalent to resources, so resources must be consumed to perform tasks
"""

from multiprocessing import Semaphore, Process
from time import sleep
import os

# Create semaphores (up to 3 tasks allowed to execute simultaneously)
sem = Semaphore(3)


# Task Functions
def handle():
    sem.acquire()  # A semaphore must be consumed to execute
    print("%s Execute Tasks" % os.getpid())
    sleep(2)
    print("%s Task Execution Completed" % os.getpid())
    sem.release()  # Return semaphore


# 10 tasks to perform
for i in range(10):
    p = Process(target=handle)
    p.start()

7. Thread Programming

7.1 Thread Basic Concepts

7.1.1 What is a thread

  1. Threads are called lightweight processes
  2. Threads can also use computer multicore resources, which are multitask programming
  3. Threads are the smallest unit of system allocation kernel
  4. Threads can be understood as branch tasks of a process

7.1.2 Thread Characteristics

  1. A process can contain multiple threads
  2. Threads are also a running behavior, consuming computer resources
  3. All threads in a process share the resources of that process
  4. Running between threads does not affect each other's operation
  5. Thread creation and destruction consume much less resources than process
  6. Each thread also has its own ID and other features

7.2 threading module create thread

  1. Create Thread Object
from threading import Thread

t = Thread()
# Function: Create Thread Object
# Parameter: target bound thread function
	 # args tuple passes parameter to thread function location
	 # The kwargs dictionary passes keys to thread functions
  1. Start Thread
t.start()
  1. Recycle Threads
t.join([timeout])

Code Samples

"""
thread.py   Thread Basic Usage
 Steps:
    1.Encapsulate Thread Function
    2.Create Thread Object
    3.Start Thread
    4.Recycle Threads
"""

import threading
from time import sleep
import os


# Thread Functions
def music():
    for i in range(3):
        sleep(2)
        print(os.getpid(), "Play: Yellow River Chorus")


# Create Thread Object
t = threading.Thread(target=music)
# Start Thread
t.start()

for i in range(4):
    sleep(1)
    print(os.getpid(), "Play: Cucurbita")

# Recycle Threads
t.join()
"""
thread  Thread Function Parameter Demo
"""

from threading import Thread
from time import sleep


# Thread function with parameters
def func(sec, name):
    print("Thread Function Parameters")
    sleep(sec)
    print("%s completion of enforcement" % name)


# Create plated multiple threads
jobs = []
for i in range(5):
    t = Thread(target=func, args=(2,), kwargs={'name': 'T%d' % i})
    jobs.append(t)
    t.start()

for i in jobs:
    i.join()

7.3 Thread Object Properties

  • t.name thread name
  • t.setName() Sets the thread name
  • t.getName() Gets the thread name
  • t.is_alive() to see if a thread is in its lifecycle
  • t.daemon sets the exit relationship between main and branch threads
  • t.setDaemon() Sets the daemon property value
  • t.isDaemon() View daemon attribute values
    The main thread exits the branch thread when daemon is True. To set it before start, it is usually not used with join
    Code Samples
"""
Thread Properties
"""

from threading import Thread
from time import sleep


def func():
    sleep(3)
    print("Thread Property Test")


t = Thread(target=func, name='よう')

t.setDaemon(True)  # Main thread exits branch thread also exits

t.start()

print("name:", t.getName())

t.setName('Hello')
print("name:", t.getName())

print("is alive:", t.is_alive())

print("daemon:", t.isDaemon())  # Print the value of daemon

7.4 Custom Thread Class

7.4.1 Creation steps

  1. Inherit Thread Class
  2. Rewrite_u init_u Method adds its own property, using super to load parent property
  3. Override run method

7.4.2 Usage

  1. Instantiate Object
  2. Call start to automatically execute run method
  3. Call join recycle thread

Code Samples

"""
Custom Thread Class Example
"""

from threading import Thread


# Custom Thread Class
class ThreadClass(Thread):
    # Override parent init
    def __init__(self, *args, **kwargs):
        self.attr = args[0]
        super().__init__()  # Load parent init

    # Assume there are many steps to complete
    def f1(self):
        print("step1")

    def f2(self):
        print("step2")

    # Override run logical call
    def run(self):
        self.f1()
        self.f2()


t = ThreadClass('abc')
t.start()
t.join()
from threading import Thread
from time import sleep, ctime


class MyThread(Thread):
    def __init__(self, target=None, args=(), kwargs=None):
        super().__init__()  # Passwords are not allowed on this line
        self.target = target
        self.args = args
        self.kwargs = kwargs

    def run(self):
        self.target(*self.args, **self.kwargs)


def player(sec, song):
    for i in range(3):
        print("Playing %s : %s" % (song, ctime()))
        sleep(sec)


t = MyThread(target=player, args=(3,), kwargs={'song': 'Cool'})
t.start()
t.join()

7.5 Interthread Communication

7.5.1 Communication Method

Communication between threads using global variables

7.5.2 Shared Resource Competition

Shared resources: Resources that can be manipulated by multiple processes or threads are called shared resources. Operational code snippets for shared resources are called critical zones
Impact: Unordered operations on shared resources can lead to data confusion or operational errors. It is often necessary to synchronize mutexes to coordinate the order of operations.

7.5.3 Synchronization Mutual Exclusion Mechanism

Synchronization: Synchronization is a collaborative relationship in which multiple processes or threads form a coordination and perform operations in an orderly manner according to the necessary steps to complete an operation.

Mutual exclusion: Mutual exclusion is a restriction. When a process or thread occupies a resource, it is locked, and other processes or threads cannot operate the resource until it is unlocked.

7.5.4 Thread Synchronization Mutual Exclusion Method

  1. Thread Event
from threading import Event

e = Event()	# Create Thread Evet Object

e.wait([timeout])	# Block waiting for e to be set

e.set()	# Set e to end the wait blocking

e.clear()	# Return e to an unset state

e.is_set()	# Check if the current e is set

  1. thread lock
from threading import Lock

lock = Lock()   # Create Lock Object
lock.acquire()  # Lock Up Blocking if lock is already locked and then called
lock.release()  # Unlock

with lock:  # Uplock
    ...
    ...

# Automatically unlock when the with block ends    

Code Samples

"""
thread_lock.py  thread lock
"""

from threading import Thread, Lock

a = b = 0
lock = Lock()  # Create Lock


def value():
    while True:
        lock.acquire()  # Uplock
        if a != b:
            print("a = %d,b=%d" % (a, b))
        lock.release()  # Unlock


t = Thread(target=value)
t.start()
while True:
    with lock:
        a += 1
        b += 1
        # Unlock automatically after the with statement ends

Tags: Python Back-end Software development

Posted on Mon, 22 Nov 2021 14:56:05 -0500 by MiCR0