Distributed task? Master / task? Worker


In Thread and Process, Process should be preferred, because Process is more stable, and Process can be distributed to multiple machines, while Thread can only be distributed to multiple CPU s of the same machine at most.

Python's multiprocessing module supports not only multiprocessing, but also the managers sub module to distribute multiprocessing to multiple machines.

A service process can act as a dispatcher (master), distribute tasks to other processes, and rely on network communication. Due to the good encapsulation of managers module, it is easy to write distributed multiprocess program without knowing the details of network communication.


The master/worker method has encapsulated the socket module, and most of the code in other uses is directly used... (Python is highly unified in ideas and methods)



# tast_master.py
//Let's first look at the service process. The service process is responsible for starting the Queue, registering the Queue on the network, and then writing tasks to the Queue
import random, time, queue
from multiprocessing import freeze_support
from multiprocessing.managers import BaseManager

# Send task queue:
task_queue = queue.Queue()
# Queue to receive results:
result_queue = queue.Queue()

# from BaseManager Inherited QueueManager:
class QueueManager(BaseManager):

def return_task_queue():
    global task_queue
    return task_queue
def return_result_queue():
    global result_queue
    return result_queue
# Take two Queue All registered on the network, callable Parameter is associated with Queue object:
if __name__=="__main__":
    #If your machine is win32,Run code for process object if this in not the main process
    //Please note that when we write multi process programs on one machine, the created Queue can be used directly,
    //However, in a distributed multiprocessing environment, adding tasks to a Queue cannot directly affect the original
    task_queue Do it, and it's bypassed QueueManager Must pass
    manager.get_task_queue()Acquired Queue Interface add.
    QueueManager.register('get_task_queue', callable=return_task_queue)
    QueueManager.register('get_result_queue', callable=return_result_queue)
    # Bind port 5000, Set verification code'abc':
    manager = QueueManager(address=('', 5000), authkey=b'abc')
    # start-up Queue:
    # Get access to Queue object:
    task = manager.get_task_queue()
    result = manager.get_result_queue()
    # Put some tasks in:
    for i in range(10):
        n = random.randint(0, 10000)
        print('Put task %d...' % n)
    # from result Queue read results:
    print('Try get results...')
    for i in range(10):
        r = result.get(timeout=20)
        print('Result: %s' % r)
    # Close:
    print('master exit.')


# tast_worker.py

import time, sys, queue

from multiprocessing.managers import BaseManager

# Create similar QueueManager:
class QueueManager(BaseManager):

# Because of this QueueManager Get only from the network Queue,So only name is provided when registering:

# Connect to the server, that is, run task_master.py 's machine:
server_addr = ''
print('Connect to server %s...' % server_addr)
# Keep the port and verification code in line with the task_master.py The settings are exactly the same:
m = QueueManager(address=(server_addr, 5000), authkey=b'abc')
# Connect from network:
# Obtain Queue Object of:
task = m.get_task_queue()
result = m.get_result_queue()
# from task Queue fetching task,And write the result result queue:
for i in range(10):
        n = task.get(timeout=1)
        print('run task %d * %d...' % (n, n))
        r = '%d * %d = %d' % (n, n, n*n)
    except Queue.Empty:
        print('task queue is empty.')
# End of processing:
print('worker exit.')

Tags: Python network socket

Posted on Mon, 04 May 2020 03:56:45 -0400 by forums