from C/C + + Programming: ZeroMQ architecture As you can see, inter thread communication includes two types:

• One is used to send and receive commands to tell the object what methods to call and what to do. The command structure is determined by command_t structure determination;
• The other is socket_ base_ The instance of T communicates with the message of session. The message structure is determined by msg_t OK.

Commands are sent and stored through mailbox_t, the message is sent and stored through pipe_t implementation. Limit the sending and receiving commands between processes.

Threads in zeromq can be divided into two categories:

• One is Io thread, like reaper_t,io_thread_t all belong to this category. The characteristic of this kind of thread is that it contains a poller and malibox_t. Poller can monitor the signal of activating mailbox
• The other is zmq socket, all sockets_ base_ T instantiated objects can be regarded as a separate thread. This kind of thread does not call poller, but also contains a mailbox_t. It can be used to send and receive commands. Since poller is not included, socket can only be used every time_ base_ T instance, first handle the mailbox_t. Check whether there are commands to be processed. Each time, call the following function to accept and process the commands:

int zmq::socket_base_t::process_commands (int timeout_, bool throttle_)

In addition, the two types of threads send commands in the same way. Next, let's talk in detail about the command structure, how to send commands, and how the two types of threads receive commands

command

Let's take a look at the command structure (see the source code Command.hpp for the detailed structure):

//  This structure defines the commands that can be sent between threads.
    struct command_t
    {
        //  Object to process the command.
        zmq::object_t *destination;

        enum type_t
        {
           ...
        } type;

        union {
           ...
        } args;
    };

As you can see, the command consists of three parts: destination, command type, and command parameter args. The so-called command is that an object tells another object to do something. To put it bluntly, it tells an object which method to call. The sender of the command is an object and the receiver is a thread. After receiving the command, the thread sends it to the corresponding object for processing according to the destination. You can see that the destination attribute of the command is object_t type. When introducing the class hierarchy diagram in the previous section, we talked about object_t and its subclasses have the function of sending and processing commands (without the function of receiving commands), so it is necessary to find out one thing, objects and objects_ t. poller, thread, mailbox_t. What is the relationship between commands?

• In zmq, each thread will have a mailbox, and the bottom function of command sending and receiving is realized by the mailbox
• zmq provides an object_ Class T, which is used to send commands using thread mailbox (object_class has other functions), object_t also has the function of processing commands.
• There is also a poller in the io thread to listen for the activation of mailbox_ After receiving the activation signal, the thread will go to the mailbox_ Read the command in T, and then give the command to the object_T treatment

Issue orders

If an object wants to use the command function of the thread, its class must inherit from object_t (the source code is in Object.hpp/.cpp):

class object_t
    {
    public:
        object_t (zmq::ctx_t *ctx_, uint32_t tid_);
        void process_command (zmq::command_t &cmd_);
        ...
    protected:
        ...
    private:
        zmq::ctx_t *ctx;//  Context provides access to the global state.
        uint32_t tid;//  Thread ID of the thread the object belongs to.
        void send_command (command_t &cmd_);
    }

You can see:

• object_t contains a tid, which means the object_ Which thread's mailbox does the t object use_ t.
• About zmq::ctx_t. In zmq, it is called context. Context is simply the living environment of zmq, which stores some global objects that can be used by all threads in zmq.

Mailbox in zmq thread_ T objects are zmq stored in ctx_t object. zmq's approach is to use A container slots to load the thread's mailbox in the context. When creating A new thread, assign A thread flag TID and mailbox to the thread, and put the mailbox in the TID position of the container. For the code, slots[tid]=mailbox. With this foundation, when thread A sends A command to thread B, it is only necessary to write A command to slots[B.tid]:

void zmq::object_t::send_command (command_t &cmd_)
{
    ctx->send_command (cmd_.destination->get_tid (), cmd_);
}
void zmq::ctx_t::send_command (uint32_t tid_, const command_t &command_)
{
    slots [tid_]->send (command_);
}
void zmq::mailbox_t::send (const command_t &cmd_)
{
    sync.lock();
    cpipe.write (cmd_, false);
    bool ok = cpipe.flush ();
    sync.unlock ();
    if (!ok)
        signaler.send ();
}

io thread receive command

As mentioned earlier, each IO thread contains a poller. The IO thread structure is as follows (the source code is Io_thread_t.hpp/.cpp):

class io_thread_t : public object_t, public i_poll_events
    {
    public:
        io_thread_t (zmq::ctx_t *ctx_, uint32_t tid_);
        ~io_thread_t ();
        void start (); //  Launch the physical thread.
        void stop ();//  Ask underlying thread to stop.
        ...
    private:
        mailbox_t mailbox;//  I/O thread accesses incoming commands via this mailbox.
        poller_t::handle_t mailbox_handle;//  Handle associated with mailbox' file descriptor.
        poller_t *poller;//  I/O multiplexing is performed using a poller object.
    }

zmq::io_thread_t::io_thread_t (ctx_t *ctx_, uint32_t tid_) :
    object_t (ctx_, tid_)
{
    poller = new (std::nothrow) poller_t;
    alloc_assert (poller);

    mailbox_handle = poller->add_fd (mailbox.get_fd (), this);
    poller->set_pollin (mailbox_handle);
}

Mailbox in constructor_ T handle is put into poller to let poller listen to its read events. Therefore, if a signal is sent, poller will wake up and call io_ thread_ In of T_ event:

void zmq::io_thread_t::in_event ()
{
    //  TODO: Do we want to limit number of commands I/O thread can
    //  process in a single go?

    command_t cmd;
    int rc = mailbox.recv (&cmd, 0);

    while (rc == 0 || errno == EINTR) {//If there is content in the read pipeline or it is interrupted while waiting for a signal, it will be read all the time
        if (rc == 0)
            cmd.destination->process_command (cmd);
        rc = mailbox.recv (&cmd, 0);
    }

    errno_assert (rc != 0 && errno == EAGAIN);
}

As you can see, in_event uses mailbox_t's function of receiving commands. After receiving the command, invoke the function of destination to process commands to process commands.

socket_base_t thread receive command

I talked about socket before_ base_ Each instance of t can be regarded as a zmq thread, but it is special. poller is used to detect whether there are unprocessed commands when using the following methods of socket:

int zmq::socket_base_t::getsockopt (int option_, void *optval_,size_t *optvallen_)
int zmq::socket_base_t::bind (const char *addr_)
int zmq::socket_base_t::connect (const char *addr_)
int zmq::socket_base_t::term_endpoint (const char *addr_)
int zmq::socket_base_t::send (msg_t *msg_, int flags_)
int zmq::socket_base_t::recv (msg_t *msg_, int flags_)
void zmq::socket_base_t::in_event ()//This function is only used when destroying socke, which will be described later ZMQ_ I'll say it when I close

The way to check is to call process_commands method:

int zmq::socket_base_t::process_commands (int timeout_, bool throttle_)
{
    int rc;
    command_t cmd;
    if (timeout_ != 0) {
        //  If we are asked to wait, simply ask mailbox to wait.
        rc = mailbox.recv (&cmd, timeout_);
    }
    else {
        some code
        rc = mailbox.recv (&cmd, 0);
    }
    //  Process all available commands.
    while (rc == 0) {
        cmd.destination->process_command (cmd);
        rc = mailbox.recv (&cmd, 0);
    }
    some code
}

It can be seen that mailbox is used in the end_ T's function of receiving commands.

Here is a question worth thinking about, why socket_base_t instance this thread does not use poller? Isn't it troublesome to check every time you use the above methods?

It may not be correct to say personal understanding. socket_base_t instance is considered a special thread because it is associated with io_ thread_ Like t, it has the function of sending and receiving commands. (for this, you can see the source code of io_thread_t, and you can find that its main function is sending and receiving commands). However, socket_ base_ The T instance is created by the user thread, that is, it is attached to the user thread. All communications in zmq are asynchronous, so the user thread cannot be blocked. Once poller is used, the thread will be blocked, which is contrary to the original design intention.

mailbox_t

As mentioned above, sending and receiving commands between threads are through mailbox_t. now let's take a look at mailbox_ How is t implemented, mailbox_ The declaration of T is as follows (the source code is located in Mailbox.hpp/.cpp)

    class mailbox_t
    {
    public:
        mailbox_t ();
        ~mailbox_t ();
        fd_t get_fd ();
        void send (const command_t &cmd_);
        int recv (command_t *cmd_, int timeout_);
    private:
        typedef ypipe_t <command_t, command_pipe_granularity> cpipe_t; //  The pipe to store actual commands.
        cpipe_t cpipe;
        signaler_t signaler;//  The semaphore passes the signal from the write thread to the read thread.


        mutex_t sync;//Only one thread receives messages from the mailbox, but a large number of threads will send messages to the mailbox. Since ypipe needs to access both ends synchronously, we must synchronize the sender
        bool active; //  True if the underlying pipeline is active, which allows us to read commands from it.

        //  Disable copying of mailbox_t object.
        mailbox_t (const mailbox_t&);
        const mailbox_t &operator = (const mailbox_t&);
    };

mailbox_ There are several key attributes in T, which need to be mentioned

• Cpipe, which may later be called pipe type_ T type, in the implementation of zmq, type_ T is a single producer and single consumer lockless queue. It is thread safe when there is only one read command thread and one write command thread. ypipe_ The security of T is the responsibility of who uses it. Commands are stored in cpipe
• sync, due to mailbox_t the bottom layer uses type_ t. It is common for multiple threads to send commands to one thread, so they should mutually exclusive type_ T both ends
• signaler, notify the command recipient. Now there are commands in the mailbox. You can read them. From the perspective of code, it is to notify the recipient mailbox_t set active to true. The bottom layer of the signal has different implementations according to different platforms. In essence, it can be regarded as a socket pair. This thing is more important. It should be done first. I won't talk about it here.
• active, whether there is a command readable in the pipeline

Let's think about a question first. Since the signaler can be used as a signal notification, why should we use the active attribute? Then look at the source code with questions

Now, how does thread th1 send commands to thread th2? In zmq, th1 writes the command to the pipeline cpipe of th2, then refreshes the pipeline of th2, and then sends a signal to th2 using the signaler to tell th2 that I wrote a command to your pipeline, and you can read the command from the pipeline.

void zmq::mailbox_t::send (const command_t &cmd_)
{
    sync.lock();//Mutually exclusive write command side
    //The detailed implementation of cpipe will be introduced in the next article. Now you only need to know the function
    cpipe.write (cmd_, false);//Send mailbox to recipient_ T pipeline write command. The receiver cannot see this command before calling flush
    bool ok = cpipe.flush ();//Refresh the pipeline. At this time, the receiver can see the command just now
    sync.unlock ();
    if (!ok)
        signaler.send ();//Send a signal to the party receiving the command
}

Besides th2 read command, if th2 is a socket_base_t instance thread, call process first_ commands，process_commands will cycle through the process_ The recv function of commands exits the loop until no command is readable; If th2 is io_thread_t such information will have the arrival of the poller monitoring signal, and then call the in_ of the thread. event，in_event will call mailbox again_ The recv function of T exits the loop until there is no command readable, sleeps, and waits to be awakened by the signal again. It should be noted that the signals sent by these two types of threads are in the mailbox_ Processed in the recv function of T. Now let's take a look at mailbox_ How t receives commands:

int zmq::mailbox_t::recv (command_t *cmd_, int timeout_)
{
    //  Try to get the command straight away.
    if (active) {
        bool ok = cpipe.read (cmd_);
        if (ok)
            return 0;

        //  If there are no more commands available, switch into passive state.
        //  If there is no command to read, set the mailbox to inactive state, indicating that there is no command to read, and then process the signal sent by the other party to activate the mailbox (no special processing, just accept it)
        active = false;
        signaler.recv ();
    }

    //  Wait for signal from the command sender.
    int rc = signaler.wait (timeout_);//There are three return values of signaler.wait: ① the wait function returns - 1 in case of error and sets errno=EINTR; ② returns - 1 and errno=EAGAIN, indicating that the signal did not wait for the signal; ③ waits for the signal.
    if (rc != 0 && (errno == EAGAIN || errno == EINTR))//This corresponds to the first two cases of wait
        return -1;

    //  We've got the signal. Now we can switch into active state.
    active = true;//Wait until the signal to activate the mailbox, activate the mailbox

    //  Get a command.
    errno_assert (rc == 0);
    bool ok = cpipe.read (cmd_);
    zmq_assert (ok);
    return 0;
}

From the code point of view, recv works like this. First check whether the mailbox is activated. If it has been activated, directly read the command to exit; If it is not activated, wait for the activation signal first, wait until the reading command exits, and exit directly before waiting. It should be noted that the functions calling recv wrap a while on the recv, probably in the form of while (true) {mailbox. Recv();} (you can see how the above source code calls recv), that is, the caller will call the recv read command until the command cannot be read, then take away the activation signal and set the mailbox to inactive state. This is the process of receiving commands.

Therefore, active and signaler cooperate in this way: each time the write command thread writes a command, first check whether the read command thread is blocked. If it is blocked, it will call the read command thread mailbox_t to send an activated read thread mailbox_t. after receiving this command, the reading thread sets activ to true in the recv function. At this time, when the reading thread circularly calls recv and finds that active is true, it will continue to read the command until there is no command to read. It sets active to false and waits for the next signal.

Now you can answer the above question. Is active redundant?

Think about it first. If you don't use active, you must send a signal to read the thread every time you write a command. In the case of large concurrency, this is also a consumption. Using active, you only need to send a signal to wake up the reading thread when the reading thread is sleeping (io_thread_t will sleep when there is no command to read, socket_base_t instance thread is special and will not sleep), which can save a lot of resources.