1, Introduction
Multithreading and thread pool are commonly used in Java development to improve program running efficiency and machine utilization. Python multithreading uses Parallel class and Multiprocessing class. In addition, there are _ Thread, threading and many other thread related classes can cooperate with os, sys, subprocess and other tool classes to realize complex operations. The following Demo introduces several multithreading implementation methods through the example of sum.
2, Parallel no Lock
Parallel under the joblib library implements the function of improving efficiency in parallel. The function to be executed by multiple threads is executed at the delayed position. njobs specifies the number of parallel core s, and the parenthesis at the end of delayed is responsible for passing the function add_ Parameters used by sum.
#!/usr/bin/python # -*- coding: UTF-8 -*- from joblib import Parallel, delayed import numpy as np def add_sum(_numList): return sum(_numList) if __name__ == '__main__': numList = [[1, 2, 3, 4], [2, 3, 4, 5], [3, 4, 5, 6], [4, 5, 6, 7]] cores = 4 results = Parallel(n_jobs=cores)(delayed(add_sum)(num) for num in numList) print(sum(results))
3, MultiProcessing without Lock
Thread pool is commonly used in Java, mainly using ThreadPoolExecutor class development with Python multiprocessing can also implement thread pooling.
#!/usr/bin/python # -*- coding: UTF-8 -*- import multiprocessing def add_sum(nums): return sum(nums) if __name__ == '__main__': pool = multiprocessing.Pool(processes=2) numLists = [[1, 2, 3, 4], [2, 3, 4, 5], [3, 4, 5, 6], [4, 5, 6, 7]] sub_process = [] for i in range(len(numLists)): data = numLists[i] process = pool.apply_async(add_sum, (data,)) sub_process.append(process) pool.close() pool.join() result = 0 for process in sub_process: result += process.get() print(result)
Tips
->The. Pool method can implement the thread pool, and processes represents the parallelism of the thread pool
-> apply_ Async implements asynchronous submission. If you pass only one parameter, you need to add a comma after the parameter, otherwise the function will recognize an error
->. close closes the thread pool. The. join method waits for all tasks in the thread pool to finish executing and starts executing the following code, so avoid writing dead loops and bugs
->. get can get the results returned after the thread processing. You need to aggregate the results of multiple threads yourself
4, MultiProcessing has Lock
In addition to parallel execution, multithreading is often designed to solve the problem of synchronization in java synchronized locking controls the reading and writing of unified memory variables. multiprocessing implements locking through lock class to solve the problem of thread synchronization.
#!/usr/bin/python # -*- coding: UTF-8 -*- import multiprocessing import time import numpy as np import sys def add_sum(_initial, _numList, _lock): while True: try: _lock.acquire() value = _numList.__next__() _initial.value = _initial.value + value _lock.release() except StopIteration: _lock.release() sys.exit() if __name__ == "__main__": lock = multiprocessing.Lock() initial = multiprocessing.Value('i', 0) work_nums = 4 sub_process = [] # A collection of processes that process data numList = [[1, 2, 3, 4], [2, 3, 4, 5], [3, 4, 5, 6], [4, 5, 6, 7]] st = time.time() for i in range(work_nums): sub_process_tmp = multiprocessing.Process(target=add_sum, args=(initial, iter(numList[i]), lock)) sub_process.append(sub_process_tmp) for process in sub_process: process.start() for process in sub_process: process.join() end = time.time() # Comparison of results print("Add Sum Cost :", (end - st)) realSum = np.array(numList).sum() print("Real Result: ", realSum, " Multi Process Result: ", initial.value)
Tips
-> lock = multiprocessing.Lock() obtains the lock of the program running, which can be used in the thread
-> initial = multiprocessing.Value('i', 0) represents a global variable, similar to spark's longAccumulator, whose value can be obtained through the. Value method, 'i' represents the type, generally including the following types:
Type | C Type | Python Type | Minimum bytes |
---|---|---|---|
c | char | char | 1 |
b | signed char | int | 1 |
B | unsigned char | int | 1 |
u | unicode | unicode char | 2 |
h | signed short | int | 2 |
H | unsigned short | int | 2 |
i | signed int | int | 2 |
I | unsigned int | long | 2 |
l | signed long | int | 4 |
L | unsigned long | long | 4 |
f | float | float | 4 |
d | double | float | 8 |
If you share a string, you can execute the following code to c_char_p as the type indication of shared variable Value, you can check ctypes. Since numerical accumulation is common, it will not be repeated here
from ctypes import c_char_p
-> target is the objective function executed by multiple threads, args is the parameter corresponding to the objective function, and tasks can be evenly allocated to each thread through the parameter
->Start starts the thread, which is similar to the Runnable class of java. join waits for the thread to finish running
->Lock has acquire() acquisition and release() release methods, and the synchronization logic can be placed between them to ensure the consistency of the results
->Remember to add the lock.release() method in excpet, otherwise the lock cannot be released and the program will get stuck
Operation result without lock:
Add Sum Cost : 0.1731090545654297 Real Result: 64 Multi Process Result: 39
Operation result with lock:
Add Sum Cost : 0.18215608596801758 Real Result: 64 Multi Process Result: 64
The program running results in the unlocked state are inconsistent, which may be 39, the correct result 64, or other values. The values after locking are consistent.
5, Summary
Through the simple demo of array summation, various schemes of lock parallel and lock free parallel are realized. The above code can be modified according to your own needs. For more specific parameters and implementation details, please refer to the official Api explanation.