Tag: python's spoof technique
Optimal number of threads
- Ncpu = number of CPUs
- Ucpu = target CPU usage
- W/C = ratio of waiting time to calculation time
In order to maintain the expected utilization of the processor, the optimal thread pool size is equal to
$$Nthreads=Ncpu*Ucpu*(1+W/C$$
- cpu intensive tasks, that is, $W < < C $, then $W/C ≈ 0 $, then $Nthreads=Ncpu*Ucpu$
If you want 100% CPU utilization, $Nthreads=Ncpu$
- IO intensive tasks, that is, most of the time the system interacts with I/O, and this time thread does not occupy the CPU for processing, that is, in this time range, other threads can use the CPU, so some more threads can be configured.
- Hybrid tasks, both of which take up a certain amount of time
Line city
For the program with increasing number of tasks, each task will generate a thread, which will eventually cause the number of threads out of control. For programs with increasing tasks, a fixed number of threads pool is necessary.
Method 1: use threadpool module
threadpool is an older module, supporting py2 and py3.
import threadpool import time def sayhello (a): print("hello: "+a) time.sleep(2) def main(): global result seed=["a","b","c"] start=time.time() task_pool=threadpool.ThreadPool(5) requests=threadpool.makeRequests(sayhello,seed) for req in requests: task_pool.putRequest(req) task_pool.wait() end=time.time() time_m = end-start print("time: "+str(time_m)) start1=time.time() for each in seed: sayhello(each) end1=time.time() print("time1: "+str(end1-start1)) if __name__ == '__main__': main(
Method 2: use the concurrent.futures module
from concurrent.futures import ThreadPoolExecutor import time def sayhello(a): print("hello: "+a) time.sleep(2) def main(): seed=["a","b","c"] start1=time.time() for each in seed: sayhello(each) end1=time.time() print("time1: "+str(end1-start1)) start2=time.time() with ThreadPoolExecutor(3) as executor: for each in seed: executor.submit(sayhello,each) end2=time.time() print("time2: "+str(end2-start2)) start3=time.time() with ThreadPoolExecutor(3) as executor1: executor1.map(sayhello,seed) end3=time.time() print("time3: "+str(end3-start3)) if __name__ == '__main__': main()
Method 3: use the vthread module
Reference resources: https://pypi.org/project/vthr...
demo1
import vthread @vthread.pool(6) def some(a,b,c): import time;time.sleep(1) print(a+b+c) for i in range(10): some(i,i,i)
demo2: grouping thread pool
import vthread pool_1 = vthread.pool(5,gqueue=1) # open a threadpool with 5 threads named 1 pool_2 = vthread.pool(2,gqueue=2) # open a threadpool with 2 threads named 2 @pool_1 def foolfunc1(num): time.sleep(1) print(f"foolstring1, test3 foolnumb1:") @pool_2 def foolfunc2(num): time.sleep(1) print(f"foolstring2, test3 foolnumb2:") @pool_2 def foolfunc3(num): time.sleep(1) print(f"foolstring3, test3 foolnumb3:") for i in range(10): foolfunc1(i) for i in range(4): foolfunc2(i) for i in range(2): foolfunc3(i)