Open source project SMSS Development Guide -- thread pool based on libevent

Libevent is a set of lightweight network library, which is based on event driven development. It can realize multi-threaded multiplexing and register event response. This article introduces the basic functions of libevent and how to develop a thread pool with libevent.

1, Use guide

Listen for service and register connection events

Libevent is an event driven network library, which completes thread multiplexing by registering different events on an event loop. Because libevent is developed in c language, we can encapsulate its function in c + + through object-oriented design pattern for convenience. The following is a detailed introduction to common functions:

(1) Event base: create (initialize) event base

Event base represents an event loop context. All events that need to be based on this event loop need to be registered on it. If the creation is successful, you need to use event_base_free() to release the resource.

(2) Evconnlistener new bind(): bind a local port and register network listening events

Parameter Description:

  • struct event_base* base the base created earlier will be associated with this event loop
  • Callback triggered by evconnlistener ABCD CB event
  • The parameter of the void *ptr callback function, which can be specified by the user arbitrarily, is convenient for use in the callback function
  • Additional ID of the unsigned flags event, representing the event operation
  • int backlog network cache size
  • const struct sockaddr *sa socket address
  • Int socket address length

The function returns a new evconnlistener. If the creation is successful, you need to use evconnlistener_free() to release the resource.

(3) Event base dispatch(): start event cycle and event distribution

This function blocks the current thread, and the user can interrupt it in the event callback function through event_base_loopbreak(). You can also use the event? Base? Loop() function if you do not want the current thread to be blocked. Be careful not to clean up the event base in the callback function.

Code example:

// Create event loop
ev_base_ = event_base_new();
if (!ev_base_)
{
    return false;
}
sockaddr_in sin;
memset(&sin, 0, sizeof(sin));
sin.sin_family = AF_INET;
sin.sin_port = htons(port_);
// Establish socket Link callback
ev_listener_ = evconnlistener_new_bind(
    ev_base_,
    SConnListenerCb,
    this,
    LEV_OPT_REUSEABLE | LEV_OPT_CLOSE_ON_FREE,
    this->backlog_,
    (sockaddr *)&sin,
    sizeof(sin));
if (!ev_listener_)
{
    return false;
}
while (!quit_)
{
    event_base_loop(ev_base_, EVLOOP_NONBLOCK);
    this_thread::sleep_for(chrono::milliseconds(1));
}
evconnlistener_free(ev_listener_);
event_base_free(ev_base_);
static void SConnListenerCb(struct evconnlistener *listen, evutil_socket_t sock, struct sockaddr *addr, int len, void *ctx)
{
    // Resolving clients ip
    char ip[16] = {0};
    sockaddr_in *addr_in = (sockaddr_in *)addr;
    evutil_inet_ntop(AF_INET, &addr_in->sin_addr, ip, sizeof(ip));
    stringstream ss;
    ss << ip << ":" << addr_in->sin_port << " Connection completed...";
    LOG4CPLUS_INFO(SimpleLogger::Get()->LoggerRef(), ss.str());
    SmsServer *server = (SmsServer *)ctx;
    int s = sock;

    server->ConnListener(s);
}

Create connection and register read, write and event listening

(1) Bufferevent? Socket? New(): create an event with socket cache

Buffer event indicates an event cache. Whenever there is data to be read, it will first take the data out of the kernel state and then notify the user. By the way, libevent supports two modes for event triggering: (ET) edge triggering and (LT) horizontal triggering. If you set the level trigger, but read the message through bufferevent, no matter whether you receive the message or not, the callback will not be triggered repeatedly. Therefore, when using buffer event to receive messages, we need to pay special attention to TCP sticky packets and long packets.

(2) bufferevent_setcb(): set the callback function of bufferevent

Parameter Description:

  • Struct bufferevent * buffev Association object
  • Buffer event data readcb read callback function prototype void (* buffer event data CB) (struct buffer event * Bev, void * CTX)
  • bufferevent_data_cb writecb write callback function prototype (same as above)
  • Buffer event ABCD event callback function prototype void (* buffer event ABCD CB) (struct buffer event * Bev, short what, void * CTX)
  • The last parameter of the void *cbarg callback function, specified by the user

As the name implies, read callback is a function that will trigger when there is data, but when will write callback trigger? Interested friends can test it by themselves. Special attention should be paid to the event callback function. All events that can be triggered include: Bev? Event? Reading, Bev? Event? Writing, Bev? Event? EOF, Bev? Event? Error, Bev? Event? Timeout, Bev? Event? Connected. If you are in the development server, the Bev event connected event will not be triggered, because the connection event is generated before the creation of bufferevent. Bev | event | reading | Bev | event | timeout can be used to indicate read data timeout. Through this event, you can detect that the heartbeat represents that the last read data has timed out. Bev event write timeout can indicate write timeout, but this event will only be triggered when there is data to be sent but the timeout is not sent successfully.

In addition, when a timeout event occurs, the related read and write operations will be removed from the buffer event. If you want to continue with the previous operation, you need to re register read / write.

(3) Bufferevent set timeout(): set read / write timeout

Only after the read / write timeout is set through this function will Bev event timeout take effect in the event callback function.

Code example:

bufferevent *buff_ev_ = bufferevent_socket_new(ev_base_, socket_, BEV_OPT_CLOSE_ON_FREE);
if (!buff_ev_)
{
    return false;
}
// Specified parameters
bufferevent_setcb(buff_ev_, SReadCb, SWriteCb, SEventCb, this);
bufferevent_enable(buff_ev_, EV_READ | EV_WRITE);
timeval tv = {timeout_, 0};
bufferevent_set_timeouts(buff_ev_, &tv, NULL);
return true;

Read and write data

(1) bufferevent_read(): receive data from cache

It is usually used in read callback to determine whether there is data in the cache through the return value

(2) bufferevent_write(): write data to the buffer to send through socket

The return value indicates how much data has been written into the kernel

Create pipe based events

libevent can be used not only on the network, but also in combination with pipe to generate pipe events.

(1) Event config new(): create an event configuration object

Event config can be used to create a non default event loop. This function is usually used in conjunction with event base new with config() to create event base. Finally, you need to use event config free () to release resources.

(2) event_new(): create a read / write event

Unlike the creation of bufferevent, event_new() only creates a matching event. If the user does not process the data in the event, the callback will be triggered all the time.

Code example:

// Initializing a pair of pipes can only be done in linux Use under the system
int pipefd[2];
if (pipe(pipefd))
{
    return false;
}
// pipefd[0]Read pipe pipefd[1]Transmitting pipeline
this->pipe_endpoint_ = pipefd[1];
// Create pipe event
event_config *ev_conf = event_config_new();
event_config_set_flag(ev_conf, EVENT_BASE_FLAG_NOLOCK);
this->ev_base_ = event_base_new_with_config(ev_conf);
event_config_free(ev_conf);
if (!ev_base_)
{
    return false;
}

pipe_ev_ = event_new(this->ev_base_, pipefd[0], EV_READ | EV_PERSIST, SEventCb, this);
event_add(pipe_ev_, 0);

2. Implement thread pool

Implementation principle of thread pool

libevent can realize multiplexing of threads, so we can read and write to multiple clients at the same time in one thread. This can make the most of the system resources, but it can't give full play to the multithreading ability of cpu. We should still consider starting multiple threads to process data when developing high availability and high load server. The key is how to distribute events to different threads to keep the load balance of multiple threads.

  1. When the service starts, create N threads first. Each thread corresponds to an event cycle event base.
  2. The main thread is responsible for listening to the specified port and processing the processing of the new connection socket in the connection callback function.
  3. When there is a new client connection, the main thread will first save the socket in a queue. Scan the current processing capacity of all threads, and select the thread with the least load to send a signal ('c ') through the pipeline. The event loop of the corresponding thread obtains the socket from the queue in the read event of the pipeline, and establishes the corresponding buffer event for processing. Current thread load + 1.
  4. After the client disconnects, it informs the thread of bufferevent to reduce the load by one.

smss source reading

The related source files are SMS server, work group, work thread and socket manager

Service initialization, register connection listening event and initialize thread group

bool SmsServer::Init()
{
    // Create event loop
    ev_base_ = event_base_new();
    if (!ev_base_)
    {
        return false;
    }
    sockaddr_in sin;
    memset(&sin, 0, sizeof(sin));
    sin.sin_family = AF_INET;
    sin.sin_port = htons(port_);
    // Establish socket Link callback
    ev_listener_ = evconnlistener_new_bind(
        ev_base_,
        SConnListenerCb,
        this,
        LEV_OPT_REUSEABLE | LEV_OPT_CLOSE_ON_FREE,
        this->backlog_,
        (sockaddr *)&sin,
        sizeof(sin));
    if (!ev_listener_)
    {
        return false;
    }
    // Create thread group management class
    boss_ = new WorkGroup(thread_num_);
    boss_->Init();
    return true;
}

The thread group is responsible for managing all threads

bool WorkGroup::Init()
{
    // Directly initializes the specified worker thread
    for (int i = 0; i < num_; i++)
    {
        int id = group_.size() + 1;
        WorkThread *work = new WorkThread(this, id, net_bus_);
        if (!work->Init())
        {
            return false;
        }
        work->Start(); // thread start...
        group_.push_back(work);
        // Register the currently initialized worker thread to the message bus
        net_bus_->Regist(work); // regist thread to netbus
    }
    return true;
}

Each thread will create a pipeline during initialization and register the corresponding read callback on its own event loop. The Notify method is exposed externally to activate the event

bool WorkThread::Init()
{
    // Initializing a pair of pipes can only be done in linux Use under the system
    int pipefd[2];
    if (pipe(pipefd))
    {
        return false;
    }
    // pipefd[0]Read pipe pipefd[1]Transmitting pipeline
    this->pipe_endpoint_ = pipefd[1];
    // Create pipe event
    event_config *ev_conf = event_config_new();
    event_config_set_flag(ev_conf, EVENT_BASE_FLAG_NOLOCK);
    this->ev_base_ = event_base_new_with_config(ev_conf);
    event_config_free(ev_conf);
    if (!ev_base_)
    {
        return false;
    }

    pipe_ev_ = event_new(this->ev_base_, pipefd[0], EV_READ | EV_PERSIST, SEventCb, this);
    event_add(pipe_ev_, 0);
    return true;
}

void WorkThread::Notify(const char *sign)
{
    // activation
    int re = write(this->pipe_endpoint_, sign, 1);
    if (re <= 0)
    {
        LOG4CPLUS_ERROR(SimpleLogger::Get()->LoggerRef(), "Pipeline activation failed");
    }
}

Get socket in read callback and create connection management object SocketManager

void WorkThread::Activated(int fd)
{
    char buf[2] = {0};
    int re = read(fd, buf, 1);
    socket_list_mtx_.lock();
    if (strcmp(buf, "c") == 0) // Notify of new client connections
    {
        // new client connect, create SocketManager
        if (socket_list_.empty())
        {
            socket_list_mtx_.unlock();
            return;
        }
        // Read a socket
        int client_sock = socket_list_.front();
        socket_list_.pop_front();
        // Establish socketManager
        SocketManager *manager = new SocketManager(this, ev_base_, client_sock, AppContext::Get()->client_timeout());
        manager->Init();
        sock_manager_list_.push_back(manager);
        stringstream ss;
        ss << "SocketManager:" << client_sock << " Create completion";
        LOG4CPLUS_DEBUG(SimpleLogger::Get()->LoggerRef(), ss.str());
    }

    socket_list_mtx_.unlock();
}

After the client connects, the socket created will be handed over to the thread with the least load for processing

void WorkGroup::CreateConnection(int sock)
{
    int min = -1;
    WorkThread *work = nullptr;
    // Traverse to find the thread with the least load
    for (auto it = group_.begin(); it != group_.end(); it++)
    {
        if (min == -1)
        {
            min = (*it)->connect_num();
            work = (*it);
        }
        else if ((*it)->connect_num() < min)
        {
            min = (*it)->connect_num();
            work = (*it);
        }
    }
    // Add one socket fd Queued and pipeline activated
    work->AddSocket(sock);
    work->Notify("c");
}

 

The full source code has been released in Code cloud Up.

Related articles: SMSS Development Guide for open source projects

Tags: C++ socket network Linux Load Balance

Posted on Sat, 11 Jan 2020 03:11:24 -0500 by tendrousbeastie