1, Implementation of multi process
Method 1
# Method wrapper multiprocess
from multiprocessing import Process
from time import sleep
def func1(arg):
print(f'{arg}start...')
sleep(2)
print(f'{arg}end...')
if __name__ == "__main__":
p1 = Process(target=func1,args=('p1',))
p2 = Process(target=func1,args=('p2',))
p1.start()
p2.start()Method 2:
Class packaging ----Multi process
from multiprocessing import Process
from time import sleep
import os
class MyProcess(Process):
def __init__(self,name):
Process.__init__(self)
self.name = name
def run(self):
# print('current process number: {} '. format(os.getpid()))
# print('parent process No.: {} '. format(os.getppid()))
print(f'{self.name}start...')
sleep(2)
print(f'{self.name}end...')
if __name__ == "__main__":
# print('current process number: {} '. format(os.getpid()))
# print('parent process No.: {} '. format(os.getppid()))
p1 = MyProcess('p1')
p2 = MyProcess('p2')
p1.start()
p2.start()2, Advantages and disadvantages of using process
1. Advantages
The multi-core computer can be used for the concurrent execution of tasks to improve the execution efficiency
The operation is not affected by other processes and is easy to create
Space independence, data security
2. Shortcomings
Process creation and deletion consume more system resources
3, Process communication
Python provides a variety of mechanisms to realize inter process communication, mainly including the following two types:
1. The Queue class under the python multiprocessing module provides many ways to realize communication between multiple processes
method
2. Pipe, also known as "pipe", is often used to realize the communication between two processes, which are located in the pipe respectively
Both ends of the road
Pipe literally means "pipe" or "pipe". It is a way to realize multi process programming, which is similar to that in real life
The Pipe (Pipe) is very similar. Usually, the pipeline has two ports, and Pipe is often used to realize the combination of two processes
The two processes are located at both ends of the pipeline. One end is used to send data and the other end is used to receive data
- send(obj)
Send an obj to the other end of the pipe, and the other end uses the recv() method to receive it. It should be noted that this obj must be
Must be serializable. If the object exceeds 32MB after serialization, it is likely to throw ValueError exception
- recv()
Receive the data sent by the other end through the send() method
- close()
Close connection
- poll([timeout])
Returns whether there is still data in the connection that can be read
- end_bytes(buffer[, offset[, size]])
Send byte data. If the offset and size parameters are not specified, the total number of buffer byte strings will be sent by default
According to; If the offset and size parameters are specified, only the buffer byte string starting from offset and having a length of size will be sent
Byte data. Data sent through this method should use recv_bytes() or recv_bytes_into method reception
- recv_bytes([maxlength])
Receive via send_ The data sent by the bytes () method. maxlength specifies the maximum number of bytes received. The method returns
Loopback received byte data
- recv_bytes_into(buffer[, offset])
Function and recv_ The bytes () method is similar, except that it puts the received data in the buffer
1. Queue implements interprocess communication
from multiprocessing import Process,current_process,Queue # current_process refers to the current process
# from queue import Queue
import os
def func(name,mq):
print('process ID {} Data obtained:{}'.format(os.getpid(),mq.get()))
mq.put('shiyi')
if __name__ == "__main__":
# print('process ID:{}'.format(current_process().pid))
# print('process ID:{}'.format(os.getpid()))
mq = Queue()
mq.put('yangyang')
p1 = Process(target=func,args=('p1',mq))
p1.start()
p1.join()
print(mq.get())2. Pipe realizes inter process communication (send(obj) and receive (obj) at the same time)
from multiprocessing import Process,current_process,Pipe
import os
def func(name,con):
print('process ID {} Data obtained:{}'.format(os.getpid(),con.recv()))
con.send('Hello!')
if __name__ == "__main__":
# print('process ID:{}'.format(current_process().pid))
con1,con2 = Pipe()
p1 = Process(target=func,args=('p1',con1))
p1.start()
con2.send("hello!")
p1.join()
print(con2.recv())4, Manager Manager
Manager provides a way to create shared data so that it can be shared in different processes
from multiprocessing import Process,current_process
import os
from multiprocessing import Manager
def func(name,m_list,m_dict):
print('Subprocess ID {} Data obtained:{}'.format(os.getpid(),m_list))
print('Subprocess ID {} Data obtained:{}'.format(os.getpid(),m_dict))
m_list.append('Hello')
m_dict['name'] = 'shiyi'
if __name__ == "__main__":
print('Main process ID:{}'.format(current_process().pid))
with Manager() as mgr:
m_list = mgr.list()
m_dict = mgr.dict()
m_list.append('Hello!!')
p1 = Process(target=func,args=('p1',m_list,m_dict))
p1.start()
p1.join()
print(m_list)
print(m_dict)5, Process pool
Python provides a better way to manage multiple processes by using process pools.
The process pool can provide a specified number of processes to users, that is, when a new request is submitted to the process pool, if the pool
If it is not full, a new process will be created to execute the request; Conversely, if the number of processes in the pool has reached the specified maximum
Value, the request will wait. As long as there are processes in the pool idle, the request can be executed.
Advantages of using process pools
1. Improve efficiency and save time for developing process, developing memory space and destroying process
2. Save memory space
| Class / method | function | parameter |
|
Pool(processes)
|
Create process pool object
|
processes represents a process pool
How many processes are there in the
|
|
pool.apply_async(func,a
rgs,kwds)
|
Asynchronous execution; Put events into the process pool queue
|
func event function
args is given as a tuple
func pass parameter
kwds is given in dictionary form
func parameter return value:
Returns a pair representing process pool events
Like, through the get method that returns a value
You can get the return value of the event function
|
|
pool.apply(func,args,kw
ds)
|
Synchronous execution; Put events into the process pool queue
|
func event functionargs
Pass parameters to func in tuple form
kwds gives func in dictionary form
Transmission parameter
|
|
pool.close()
|
Close process pool
| |
|
pool.join()
|
Recycle process pool
| |
|
pool.map(func,iter)
|
map function similar to python
Count and put the events to be done into the process pool
|
func function to execute
iter iteration object
|
from multiprocessing import Pool
import os
from time import sleep
def func1(name):
print(f"Of the current process ID:{os.getpid()},{name}")
sleep(2)
return name
def func2(args):
print(args)
if __name__ == "__main__":
pool = Pool(5)
pool.apply_async(func = func1,args=('t1',),callback=func2)
pool.apply_async(func = func1,args=('t2',),callback=func2)
pool.apply_async(func = func1,args=('t3',),callback=func2)
pool.apply_async(func = func1,args=('t4',))
pool.apply_async(func = func1,args=('t5',))
pool.apply_async(func = func1,args=('t6',))
pool.close()
pool.join()
from multiprocessing import Pool
import os
from time import sleep
def func1(name):
print(f"Of the current process ID:{os.getpid()},{name}")
sleep(2)
return name
if __name__ == "__main__":
with Pool(5) as pool:
args = pool.map(func1,('t1,','t2,','t3,','t4,','t5,','t6,','t7,','t8,'))
for a in args:
print(a)