executive summary
In this experiment, we try to use the IO multiplex select interface.
stay [network programming] TCP socket single machine communication experiment A demo of socket communication between a client thread and a server thread is implemented in. For the server thread, it not only listens and accepts connection requests, but also receives, sends and processes the data interacting with the client. A server thread can only handle one client at a time. When processing the data of one client, it cannot accept the connection request of another client. For multi client scenarios, such a server has little practical value.
An idea to solve the problem is: one thread / process listens to connection requests, and multiple other threads / processes connect to each client respectively. In this way, the server can process data interaction with multiple clients at the same time and accept connection requests from new clients in time:
However, if there are many connection requests, the server cannot afford the resource consumption caused by one client and one processing thread, and thread switching also requires cost. In fact, most of the time, there is no data interaction between client and server. In view of this, IO multiplexing (select, poll, epoll) can be considered: IO multiplexing can realize one thread to monitor multiple file handles; Once a file handle is ready, it can notify the application to read and write accordingly; When no file handle is ready, the application will be blocked and the cpu will be surrendered. Multiplexing refers to multiple network connections, and multiplexing refers to multiplexing the same thread. In essence, IO multiplexing centralizes the work of accept, recv, send blocking and waiting for event readiness to the kernel for management.
Main process structure
stay [network programming] TCP socket single machine communication experiment On the basis of, the number of client threads is increased to 3, and a single server thread is divided into a listening thread and a working thread. Listen to the port in the listening thread, establish a connection with the client, and pass the connection handle to the working thread; In the working thread, use select to monitor whether all connected handles are readable. If readable, send and receive data.
experimental result
select interface
Function prototype
int select (int maxfdp,
fd_set *readset,
fd_set *writeset,
fd_set *exceptest,
struct timeval *timeout);parameter maxfdp
First parameter maxfdp Is the maximum value of all file descriptors to be monitored plus 1. Note: it is not the number of file descriptors to be monitored plus 1. File descriptors are essentially integers. File descriptors with values greater than maxfdp will not be monitored by select.
If 3 + 1 is passed to maxfdp (the number of file descriptors to be monitored plus 1):
select does not monitor the connection handles of three clients:
The data sent by the three clients are accumulated in the receiving window of the server and will not be read by the application layer of the server:
The parameter points to fd_ Pointer to set
The second, third and fourth parameters all point to fd_ Pointer to set. Second parameter readset corresponds to the descriptor set of readable events; Third parameter writeset corresponds to the descriptor set of writable events; Fourth parameter exceptest is the descriptor set corresponding to the exception event.
typedef struct {
unsigned long fds_bits[__FDSET_LONGS];
} fd_set;fd_set is essentially an array of unsigned long shapes. The size of the array__ FDSET_ Dragons is a macro definition. The value is 1024/(8 * sizeof(unsigned long)). The actual space occupied by the array is 1024 / 8 = 128 Byte = 1024 bit.
fd_set represents the set of file descriptors in the form of bitmap: a bit corresponds to a file descriptor, and 0 or 1 of the bit of the corresponding position indicates whether the file descriptor is monitored or whether the file descriptor is ready.
Because of this array size and representation, select can only monitor file descriptors of 0 ~ 1023 at most, a total of 1024 - the data structure FD for storing file descriptors_ set This limits the maximum number of file descriptors that a select can listen to to to 1024.
select provides some macro It can be used to operate the file descriptor set, that is, to change the 0 and 1 values of the corresponding bits:
FD_ZERO(fd_set* fds);
Clear the file descriptor set fds, that is, set all bits to 0
FD_SET(int fd, fd_set* fds);
Add the file descriptor fd to the set fds, that is, the corresponding bit position is 1
FD_ISSET(int fd, fd_set* fds);
Judge whether the file descriptor fd is in the set fds, that is, judge whether the corresponding bit is 1
FD_CLR(int fd, fd_set* fds);
File descriptor fd From collection fds Delete, that is, set the corresponding bit position to 0
In terms of usage mode, call FD first_ Zero will fd_set cleared, and then called FD_. Set adds the file descriptor to be monitored to fd_set, and then call the function select to test FD_ For all file descriptors in set, the macro FD is used after the select function returns_ Isset checks whether the bit corresponding to a file descriptor is still 1. If it is 1, it indicates that an event is ready, and if it is 0, it is vice versa.
Parameter timeout
Last parameter Timeout is the timeout time, which indicates how long to wait before giving up the wait. Passing NULL means waiting for an infinite time. It continues to block until an event is ready.
timeout points to The timeval structure has two members: seconds and microseconds:
struct timeval
{
__time_t tv_sec; /* Seconds. */
__suseconds_t tv_usec; /* Microseconds. */
};When using timeval, just assign values to the members directly, eg
struct timeval timeout = {0};
timeout.tv_sec = 1;
timeout.tv_usec = 0;Return value
If select fails, return - 1; If timeout, return 0; If there are ready descriptors, the number of ready descriptors is returned.
select disadvantages
select has several major disadvantages, which are similar to the file descriptor set fd_set about:
- Every time you call select, you need to set fd_set is copied from the user state to the kernel state. When the select detects a ready event or a timeout return, it will send FD again_ Set is copied back from the kernel state to the user state, which is expensive;
- select monitors file descriptors by polling. It performs linear scanning on the file descriptor set over and over again to check whether there are events ready, which is inefficient;
- select can only monitor 1024 file descriptors at most, and the number is limited;
Complete code implementation
Header file
#include <stdio.h> #include <unistd.h> #include <errno.h> #include <string.h> #include <pthread.h> #include <sys/types.h> #include <sys/socket.h> #include <netinet/in.h> #include <arpa/inet.h> #include <sys/syscall.h>
Macro definition
#define LOCAL_IP_ADDR "127.0.0.1" #define SERVER_LISTEN_PORT 5197 #define MAX_LISTEN_EVENTS 16 #define NET_MSG_BUF_LEN 128 #define CLINET_SEND_MSG "Hello Server~" #define SERVER_SEND_MSG "Hello Client~"
Function to get thread ID
pid_t gettid(void) {
// The header file sys/syscall.h needs to be imported
return syscall(SYS_gettid);
}Client thread entry function
void* client(void* param) {
int iRes = 0, iConnFd = 0, iNetMsgLen = 0;
pthread_t thdId = gettid();
char szNetMsg[NET_MSG_BUF_LEN] = {0};
struct sockaddr_in stServAddr;
iConnFd = socket(AF_INET, SOCK_STREAM, 0);
if (-1 == iConnFd) {
printf("Client[%u] failed to create socket, err[%s]\n",
thdId, strerror(errno));
return NULL;
}
// Fill in the target address structure and specify the protocol family, target port and target host IP address
stServAddr.sin_family = AF_INET;
stServAddr.sin_port = htons(SERVER_LISTEN_PORT);
stServAddr.sin_addr.s_addr = inet_addr(LOCAL_IP_ADDR);
// 1. Pass the socket handle, 2. Pass the pointer of the target address structure to be connected, and 3. Pass the size of the address structure
while (1) {
iRes = connect(iConnFd, (struct sockaddr *)&stServAddr,
sizeof(stServAddr));
if (0 != iRes) {
printf("Client[%u] failed to connect to[%s:%u], err[%s]\n",
thdId, LOCAL_IP_ADDR, SERVER_LISTEN_PORT,
strerror(errno));
sleep(2);
continue;
} else {
printf("Client[%u] succeeded to connect to[%s:%u]\n",
thdId, LOCAL_IP_ADDR, SERVER_LISTEN_PORT);
break;
}
}
iNetMsgLen = send(iConnFd, CLINET_SEND_MSG,
strlen(CLINET_SEND_MSG), 0);
if (iNetMsgLen < 0) {
printf("Client[%u] failed to send msg to server, err[%s]\n",
thdId, strerror(errno));
close(iConnFd);
return NULL;
}
iNetMsgLen = recv(iConnFd, szNetMsg, sizeof(szNetMsg), 0);
if (iNetMsgLen < 0) {
printf("Client[%u] failed to read from network, err[%s]\n",
thdId, strerror(errno));
} else {
printf("Client[%u] recv reply[%s]\n", thdId, szNetMsg);
}
close(iConnFd);
return NULL;
}Function functions handle readable events
int eventProc(int *piConnFdSet, fd_set *pfsReadSet) {
int iIndex = 0, iConnFd = 0, iNetMsgLen = 0;
char szNetMsg[NET_MSG_BUF_LEN] = {0};
for (iIndex = 0; iIndex < MAX_LISTEN_EVENTS; iIndex++) {
if (!FD_ISSET(piConnFdSet[iIndex], pfsReadSet)) {
continue;
}
// Simplify code with temporary variables
iConnFd = piConnFdSet[iIndex];
// Receive client message
iNetMsgLen = recv(iConnFd, szNetMsg, sizeof(szNetMsg), 0);
if (iNetMsgLen < 0) {
printf("Server failed to recv from network, err[%s]\n",
strerror(errno));
close(iConnFd);
return -1;
}
printf("Server recv msg[%s]\n", szNetMsg);
// Reply to client
iNetMsgLen = send(iConnFd, SERVER_SEND_MSG,
strlen(SERVER_SEND_MSG), 0);
if (iNetMsgLen < 0) {
printf("Server failed to reply client, err[%s]\n",
strerror(errno));
close(iConnFd);
return -1;
}
piConnFdSet[iIndex] = 0;
close(iConnFd);
}
return 0;
}Server worker thread entry function
void* serverWork(void* param) {
int iRes = 0, iIndex = 0, iMaxFd = 0, iEventNum = 0;
int *piConnFdSet = (int *)param;
fd_set fsReadSet;
struct timeval timeout = {0};
// monitor
while (1) {
timeout.tv_sec = 1;
timeout.tv_usec = 0;
FD_ZERO(&fsReadSet);
#if 0
iMaxFd = 0;
for (iIndex = 0; iIndex < MAX_LISTEN_EVENTS; iIndex++) {
FD_SET(piConnFdSet[iIndex], &fsReadSet);
iMaxFd = piConnFdSet[iIndex] > iMaxFd ?
piConnFdSet[iIndex] : iMaxFd;
}
#else
iMaxFd = 3;
#endif
iEventNum = select(iMaxFd + 1, &fsReadSet, NULL, NULL, &timeout);
if (-1 == iEventNum) {
printf("Server failed to select event.\n");
break;
}
printf("Server select get [%u] event\n", iEventNum);
if (0 == iEventNum) {
continue;
}
iRes = eventProc(piConnFdSet, &fsReadSet);
if (0 != iRes) {
printf("Server failed to proc event.\n");
break;
}
}
return NULL;
}Server listening thread entry function
void* serverLsn(void* param) {
int iRes = 0, iIndex = 0;
int iLsnFd = 0, iConnFd = 0, iReusePort = 0, iSockAddrLen = 0;
int *paiConnFdSet = (int *)param;
struct sockaddr_in stLsnAddr;
struct sockaddr_in stCliAddr;
// Create socket
iLsnFd = socket(AF_INET, SOCK_STREAM, 0);
if (-1 == iLsnFd) {
printf("Server failed to create socket, err[%s]\n",
strerror(errno));
return NULL;
}
// Set port multiplexing
iReusePort = 1;
iRes = setsockopt(iLsnFd, SOL_SOCKET, SO_REUSEPORT, &iReusePort,
sizeof (iReusePort));
if (-1 == iRes) {
printf("Server failed set reuse attr, err[%s]\n",
strerror(errno));
close(iLsnFd);
return NULL;
}
stLsnAddr.sin_family = AF_INET;
stLsnAddr.sin_port = htons(SERVER_LISTEN_PORT);
stLsnAddr.sin_addr.s_addr = INADDR_ANY;
// Binding port
iRes = bind(iLsnFd, (struct sockaddr*)&stLsnAddr,
sizeof(stLsnAddr));
if (-1 == iRes) {
printf("Server failed to bind port[%u], err[%s]\n",
SERVER_LISTEN_PORT, strerror(errno));
close(iLsnFd);
return NULL;
} else {
printf("Server succeeded to bind port[%u], start listen.\n",
SERVER_LISTEN_PORT);
}
// monitor
iRes = listen(iLsnFd, MAX_LISTEN_EVENTS);
if (-1 == iRes) {
printf("Server failed to listen port[%u], err[%s]\n",
SERVER_LISTEN_PORT, strerror(errno));
close(iLsnFd);
return NULL;
}
while (iIndex < MAX_LISTEN_EVENTS) {
iSockAddrLen = sizeof(stCliAddr);
// 1. Pass in the listening handle, 2. Pass in the address structure pointer to receive the client's address
// 3 parameter incoming address structure size
iConnFd = accept(iLsnFd, (struct sockaddr*)&stCliAddr,
&iSockAddrLen);
if (-1 == iConnFd) {
printf("Server failed to accept connect request, err[%s]\n",
strerror(errno));
break;
} else {
printf("Server accept connect request from[%s:%u]\n",
inet_ntoa(stCliAddr.sin_addr),
ntohs(stCliAddr.sin_port));
paiConnFdSet[iIndex] = iConnFd;
iIndex++;
}
}
close(iLsnFd);
return NULL;
}Main function
int main() {
// Thread ID, which is essentially an unsigned long integer
pthread_t thdServerWork = 101;
pthread_t thdServerLsn = 102;
pthread_t thdClient1 = 1;
pthread_t thdClient2 = 2;
pthread_t thdClient3 = 3;
// Set of socket file descriptors to monitor for readability
int aiConnFdSet[MAX_LISTEN_EVENTS] = {0};
// 1 reference thread ID, 2 reference thread attribute,
// Parameter 3 specifies the thread entry function, and parameter 4 specifies the parameters passed to the entry function
pthread_create(&thdServerWork, NULL, serverWork, &aiConnFdSet[0]);
pthread_create(&thdServerLsn, NULL, serverLsn, &aiConnFdSet[0]);
pthread_create(&thdClient1, NULL, client, NULL);
pthread_create(&thdClient2, NULL, client, NULL);
pthread_create(&thdClient3, NULL, client, NULL);
// The 1 Parameter passes in the thread ID, and the 2 parameter is used to receive the return value of the thread entry function. If the return value is not required, set it to NULL
pthread_join(thdServerWork, NULL);
pthread_join(thdServerLsn, NULL);
pthread_join(thdClient1, NULL);
pthread_join(thdClient2, NULL);
pthread_join(thdClient3, NULL);
return 0;
}