Three ways to implement python multithreading concurrent processing

Tag: python's spoof technique

Optimal number of threads

  • Ncpu = number of CPUs
  • Ucpu = target CPU usage
  • W/C = ratio of waiting time to calculation time

In order to maintain the expected utilization of the processor, the optimal thread pool size is equal to
$$Nthreads=Ncpu*Ucpu*(1+W/C$$

  • cpu intensive tasks, that is, $W < < C $, then $W/C ≈ 0 $, then $Nthreads=Ncpu*Ucpu$

If you want 100% CPU utilization, $Nthreads=Ncpu$

  • IO intensive tasks, that is, most of the time the system interacts with I/O, and this time thread does not occupy the CPU for processing, that is, in this time range, other threads can use the CPU, so some more threads can be configured.
  • Hybrid tasks, both of which take up a certain amount of time

Line city

For the program with increasing number of tasks, each task will generate a thread, which will eventually cause the number of threads out of control. For programs with increasing tasks, a fixed number of threads pool is necessary.

Method 1: use threadpool module

threadpool is an older module, supporting py2 and py3.

import threadpool
import time

def sayhello (a):
    print("hello: "+a)
    time.sleep(2)

def main():
    global result
    seed=["a","b","c"]
    start=time.time()
    task_pool=threadpool.ThreadPool(5)
    requests=threadpool.makeRequests(sayhello,seed)
    for req in requests:
        task_pool.putRequest(req)
    task_pool.wait()
    end=time.time()
    time_m = end-start
    print("time: "+str(time_m))
    start1=time.time()
    for each in seed:
        sayhello(each)
    end1=time.time()
    print("time1: "+str(end1-start1))

if __name__ == '__main__':
    main(

Method 2: use the concurrent.futures module

from concurrent.futures import ThreadPoolExecutor
import time

def sayhello(a):
    print("hello: "+a)
    time.sleep(2)

def main():
    seed=["a","b","c"]
    start1=time.time()
    for each in seed:
        sayhello(each)
    end1=time.time()
    print("time1: "+str(end1-start1))
    start2=time.time()
    with ThreadPoolExecutor(3) as executor:
        for each in seed:
            executor.submit(sayhello,each)
    end2=time.time()
    print("time2: "+str(end2-start2))
    start3=time.time()
    with ThreadPoolExecutor(3) as executor1:
        executor1.map(sayhello,seed)
    end3=time.time()
    print("time3: "+str(end3-start3))

if __name__ == '__main__':
    main()

Method 3: use the vthread module

Reference resources: https://pypi.org/project/vthr...

demo1

import vthread
 
@vthread.pool(6)
def some(a,b,c):
    import time;time.sleep(1)
    print(a+b+c)
 
for i in range(10):
    some(i,i,i)

demo2: grouping thread pool

import vthread
pool_1 = vthread.pool(5,gqueue=1) # open a threadpool with 5 threads named 1
pool_2 = vthread.pool(2,gqueue=2) # open a threadpool with 2 threads named 2

@pool_1
def foolfunc1(num):
    time.sleep(1)
    print(f"foolstring1, test3 foolnumb1:{num}")

@pool_2
def foolfunc2(num):
    time.sleep(1)
    print(f"foolstring2, test3 foolnumb2:{num}")

@pool_2
def foolfunc3(num):
    time.sleep(1)
    print(f"foolstring3, test3 foolnumb3:{num}")

for i in range(10): foolfunc1(i)
for i in range(4): foolfunc2(i)
for i in range(2): foolfunc3(i)

Tags: Python

Posted on Tue, 03 Dec 2019 05:53:09 -0500 by x_maras