RocketMQ source code analysis RPC communication

The essence of message queue lies in sending, storing and receiving messages. So, for a message queue, how to send and receive messages efficiently is the key point

1, Overview of Remoting communication module in RocketMQ

The overall deployment architecture of RocketMQ message queue is as follows:

First, let's talk about several roles in RocketMQ message queue cluster:

(1) NameServer: in MQ cluster, we do naming service, update and route discovery broker service;

(2) Broker master: broker message host server;

(3) Broker Slave: broker message slave server;

(4) Producer: Message producer;

(5) Consumer: message consumer;

Part of the communication of RocketMQ cluster is as follows:

(1) After the Broker starts, it needs to complete the operation of registering itself to the NameServer, and then report the Topic routing information to the NameServer regularly every 30s;

(2) When a message Producer sends a message as a client, it needs to obtain routing information from the locally cached Topic publishinfotable according to the Topic of Msg. If not, the updated routing information will be pulled again from the NameServer;

(3) The message Producer selects a message queue to send messages according to the route information obtained in (2); the Broker receives messages as the receiver of the messages and stores them on the disk; it can be seen from (1) - (3) above that in the message Producer, There will be communication between Broker and NameServer (only part of MQ communication is mentioned here), so how to design a good network communication module is very important in MQ. It will determine the overall message transmission capacity and final performance of RocketMQ cluster. RocketMQ remoting module is responsible for network communication in RocketMQ message queue. It is relied on and referenced by almost all other modules that need network communication (such as RocketMQ client, RocketMQ server, RocketMQ namesrv). In order to achieve efficient data request and reception between client and server, RocketMQ message queue defines communication protocol and expands communication module on the basis of netty. ps: since the communication module of RocketMQ is built on the basis of netty, before reading the source code of RocketMQ, readers should have a certain understanding of the multithreaded model and JAVA NIO model of netty, so that they can understand the source code of RocketMQ faster. The RocketMQ version read by the author is 4.2.0, and the netty version relied on is 4.0.42.final. The code structure of RocketMQ is as follows:

The source part can be divided into

Modules such as rocketmq broker, rocketmq client, rocketmq common, rocketmq filtersrv, rocketmq namesrv and rocketmq Remoting

, the communication framework is encapsulated in the RocketMQ remoting module. This paper mainly discusses RocketMQ protocol format, message encoding and decoding, communication mode (synchronous / asynchronous / one-way) and specific communication process of sending / receiving messages.

2, Implementation of Remoting communication module in RocketMQ

1. Class structure of Remoting communication module

From the perspective of class hierarchy:

(1)RemotingService

: three methods are provided for the top-level interface:

void start();
void shutdown();
void registerRPCHook(RPCHook rpcHook);

(2) Remotingclient / RemotingServer: the two interfaces inherit the top-level interface RemotingService, and provide the necessary methods for the Client and Server respectively. The methods of RemotingServer are listed below:

/**
     * Same as remoting client
     *
     * @param requestCode
     * @param processor
     * @param executor
     */
    void registerProcessor(final int requestCode, final NettyRequestProcessor processor,
        final ExecutorService executor);

    /**
     * Register default processor
     *
     * @param processor
     * @param executor
     */
    void registerDefaultProcessor(final NettyRequestProcessor processor, final ExecutorService executor);

    int localListenPort();

    /**
     * Get different processing pairs according to the request code
     *
     * @param requestCode
     * @return
     */
    Pair<NettyRequestProcessor, ExecutorService> getProcessorPair(final int requestCode);

    /**
     * The same as the RemotingClient, synchronous communication returns RemotingCommand
     * @param channel
     * @param request
     * @param timeoutMillis
     * @return
     * @throws InterruptedException
     * @throws RemotingSendRequestException
     * @throws RemotingTimeoutException
     */
    RemotingCommand invokeSync(final Channel channel, final RemotingCommand request,
        final long timeoutMillis) throws InterruptedException, RemotingSendRequestException,
        RemotingTimeoutException;

    /**
     * The same as RemotingClient, asynchronous communication, no RemotingCommand returned
     *
     * @param channel
     * @param request
     * @param timeoutMillis
     * @param invokeCallback
     * @throws InterruptedException
     * @throws RemotingTooMuchRequestException
     * @throws RemotingTimeoutException
     * @throws RemotingSendRequestException
     */
    void invokeAsync(final Channel channel, final RemotingCommand request, final long timeoutMillis,
        final InvokeCallback invokeCallback) throws InterruptedException,
        RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException;

    /**
     * Like the remoting client, one-way communication, such as heartbeat packet
     *
     * @param channel
     * @param request
     * @param timeoutMillis
     * @throws InterruptedException
     * @throws RemotingTooMuchRequestException
     * @throws RemotingTimeoutException
     * @throws RemotingSendRequestException
     */
    void invokeOneway(final Channel channel, final RemotingCommand request, final long timeoutMillis)
        throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException,
        RemotingSendRequestException;

(3) NettyRemotingAbstract: the abstract class of Netty communication processing, defines and encapsulates the public processing methods of Netty processing; (4) NettyRemotingClient/NettyRemotingServer: implements RemotingClient and RemotingServer respectively, and inherits the NettyRemotingAbstract abstract class. Other components in RocketMQ (such as client, nameServer and broker, both of which are used for sending and receiving messages)

2. Protocol design and coding and decoding of messages

When a message is sent between the Client and the Server, a protocol is needed for the sent message. Therefore, it is necessary to customize RocketMQ's message protocol. At the same time, in order to efficiently transmit and read messages in the network, it is necessary to encode and decode messages. In RocketMQ, RemotingCommand encapsulates all data content in the process of message transmission, including not only all data structures, but also encoding and decoding operations. Some member variables of the RemotingCommand class are as follows:

Header field type Request description Response description
code int Request operation code. The responder processes different services according to different request codes Response code. 0 for success, non-0 for errors
language LanguageCode Language implemented by requester Language implemented by the responder
version int Version of requester program Version of responder program
opaque int Equivalent to reqeustId, different request identification codes on the same connection correspond to those in the response message Reply without modification
flag int Flag to distinguish between normal RPC and oneway RPC Flag to distinguish between normal RPC and oneway RPC
remark String Transfer custom text information Transfer custom text information
extFields HashMap<String, String> Request custom extension information Respond to custom extension information

Here, Broker sends a heartbeat registration message to NameServer:

[
code=103,//The code corresponding to 103 here is that broker registers its own message with nameserver
language=JAVA,
version=137,
opaque=58,//This is the requestId
flag(B)=0,
remark=null,
extFields={
    brokerId=0,
    clusterName=DefaultCluster,
    brokerAddr=ip1: 10911,
    haServerAddr=ip1: 10912,
    brokerName=LAPTOP-SMF2CKDN
},
serializeTypeCurrentRPC=JSON

Here is the format of RocketMQ communication protocol:

Visible transmission content can be divided into the following four parts:

(1) Message length

: total length, four bytes storage, occupying one int type;

(2) Serialization type & header length

: it also occupies an int type. The first byte represents the serialization type, and the last three bytes represent the header length;

(3) Header data

: serialized header data;

(4) Message body data

: the binary byte data content of the message body; the encoding and decoding of the message are respectively completed in the encode and decode methods of the RemotingCommand class. The following is the specific implementation of the message encoding encode method:

public ByteBuffer encode() {
    // 1> header length size
    int length = 4;    //Total message length

    // 2> header data length
    //Encode message header as byte []
    byte[] headerData = this.headerEncode();
    //Calculate head length
    length += headerData.length;

    // 3> body data length
    if (this.body != null) {
        //Message body length
        length += body.length;
    }
    //Allocate ByteBuffer, add 4 here,
    //This is because 4 bytes of the storage header length are not included in the calculation of the total message length
    ByteBuffer result = ByteBuffer.allocate(4 + length);

    // length
    //Put the total message length into ByteBuffer
    result.putInt(length);

    // header length
    //Put header length into ByteBuffer
    result.put(markProtocolType(headerData.length, serializeTypeCurrentRPC));

    // header data
    //Put header data into ByteBuffer
    result.put(headerData);

    // body data;
    if (this.body != null) {
        //Put message body in ByteBuffer
        result.put(this.body);
    }
    //Reset the position of ByteBuffer
    result.flip();

    return result;
}

    /**
     * markProtocolType The method is to put RPC type and headerData length code into a byte[4] array
     *
     * @param source
     * @param type
     * @return
     */
    public static byte[] markProtocolType(int source, SerializeType type) {
        byte[] result = new byte[4];

        result[0] = type.getCode();
        //Move 16 bits to the right and then add 255 and - > 16-24 bits
        result[1] = (byte) ((source >> 16) & 0xFF);
        //Move 8 bits to the right and then add 255 and - > 8-16 bits
        result[2] = (byte) ((source >> 8) & 0xFF);
        //Move 0 bits to the right and then add 255 and - > 8-0 bits
        result[3] = (byte) (source & 0xFF);
        return result;
    }

The decode method of message decoding is the reverse process of encoding. Its specific implementation is as follows:

public static RemotingCommand decode(final ByteBuffer byteBuffer) {
        //Get the total length of byteBuffer
        int length = byteBuffer.limit();

        //Get the first 4 bytes, assemble int type, the length is the total length
        int oriHeaderLen = byteBuffer.getInt();

        //Get the length of the message header, here and 0xFFFFFF do and operation, the length of the encoding is 24 bits
        int headerLength = getHeaderLength(oriHeaderLen);

        byte[] headerData = new byte[headerLength];
        byteBuffer.get(headerData);

        RemotingCommand cmd = headerDecode(headerData, getProtocolType(oriHeaderLen));

        int bodyLength = length - 4 - headerLength;
        byte[] bodyData = null;
        if (bodyLength > 0) {
            bodyData = new byte[bodyLength];
            byteBuffer.get(bodyData);
        }
        cmd.body = bodyData;

        return cmd;
    }

3. Communication mode and flow of messages

There are three main ways to support communication in RocketMQ message queue:

(1) Sync (sync)

(2) Async (asynchronous)

(3) One way

The "synchronous" communication mode is relatively simple, which is generally used in the scenario of sending heartbeat packets without paying attention to its Response. This paper mainly introduces the asynchronous communication process of RocketMQ (limited to space, readers can analyze the synchronous communication process according to the same mode). The following is the overall flow chart of RocketMQ asynchronous communication:

The following two sections mainly introduce the concrete implementation of sending request message on Client side and receiving message on Server side, and briefly analyze the callback on Client side.

3.1. Specific implementation of Client sending request message

When the client calls the asynchronous communication interface, invokeAsync, first, the RemotingClient implementation class - NettyRemotingClient gets the corresponding channel according to addr (if the local cache is not created), then calls the invokeAsyncImpl method to transfer the data to the abstract class NettyRemotingAbstract processing. Class). The specific source code to send the request message is as follows:

/**
 * invokeAsync(Asynchronous call)
 *
 */
public void invokeAsyncImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis,
    final InvokeCallback invokeCallback)
    throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException {
    //Equivalent to request ID, RemotingCommand will generate a request ID for each request, starting from 0 and adding 1 each time

    final int opaque = request.getOpaque();
    boolean acquired = this.semaphoreAsync.tryAcquire(timeoutMillis, TimeUnit.MILLISECONDS);
    if (acquired) {
        final SemaphoreReleaseOnlyOnce once = new SemaphoreReleaseOnlyOnce(this.semaphoreAsync);
        //Build ResponseFuture based on request ID
        final ResponseFuture responseFuture = new ResponseFuture(opaque, timeoutMillis, invokeCallback, once);
        //Put ResponseFuture in responseTable
        this.responseTable.put(opaque, responseFuture);
        try {
            //Using Netty's channel to send request data
            channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
                //Execute after message sending
                @Override
                public void operationComplete(ChannelFuture f) throws Exception {
                    if (f.isSuccess()) {
                        //If the message is sent to the Server successfully, Set and return directly here
                        responseFuture.setSendRequestOK(true);
                        return;
                    } else {
                        responseFuture.setSendRequestOK(false);
                    }

                    responseFuture.putResponse(null);
                    responseTable.remove(opaque);
                    try {
                        //Execute callback
                        executeInvokeCallback(responseFuture);
                    } catch (Throwable e) {
                        log.warn("excute callback in writeAndFlush addListener, and callback throw", e);
                    } finally {
                        //Release semaphore
                        responseFuture.release();
                    }

                    log.warn("send a request command to channel <{}> failed.", RemotingHelper.parseChannelRemoteAddr(channel));
                }
            });
        } catch (Exception e) {
            //exception handling
            responseFuture.release();
            log.warn("send a request command to channel <" + RemotingHelper.parseChannelRemoteAddr(channel) + "> Exception", e);
            throw new RemotingSendRequestException(RemotingHelper.parseChannelRemoteAddr(channel), e);
        }
    } else {
        if (timeoutMillis <= 0) {
            throw new RemotingTooMuchRequestException("invokeAsyncImpl invoke too fast");
        } else {
            String info =
                String.format("invokeAsyncImpl tryAcquire semaphore timeout, %dms, waiting thread nums: %d semaphoreAsyncValue: %d",
                    timeoutMillis,
                    this.semaphoreAsync.getQueueLength(),
                    this.semaphoreAsync.availablePermits()
                );
            log.warn(info);
            throw new RemotingTimeoutException(info);
        }
    }
}

When sending request messages on the Client side, there is an important data structure to note:

(1) responseTable - save request code and response association mapping

protected final ConcurrentHashMap<Integer /* opaque */, ResponseFuture> responseTable

opaque means that the initiator of the request has different request identification codes on the same connection. Each time a message is sent, the synchronous blocking / asynchronous non blocking mode can be selected. Regardless of the communication mode, the request opcode will be saved in the Map map Map responseTable of ResponseFuture.

(2) ResponseFuture - save the return response (including callback execution method and semaphore)

public ResponseFuture(int opaque, long timeoutMillis, InvokeCallback invokeCallback,
        SemaphoreReleaseOnlyOnce once) {
        this.opaque = opaque;
        this.timeoutMillis = timeoutMillis;
        this.invokeCallback = invokeCallback;
        this.once = once;
    }

For synchronous communication, the third and fourth parameters are null; for asynchronous communication, invokeCallback can find the callback execution method corresponding to the request code according to the responseTable when receiving the message response, and the semaphore parameter is used as flow control. When multiple threads write data to one connection at the same time, the number of permission to write at the same time can be controlled through the semaphore.

(3) Exception sending process processing - regularly scan the local cache of the responseTable. When sending messages, if there is an exception (for example, the server does not return the response to the client or the response is lost due to the network), the local cache Map of the responseTable described above will be stacked. At this time, a timing task is needed to specifically clean and recycle the responseTable. When the client / server of RocketMQ starts, a scheduled task called once every 1s will be generated to check all the responseFuture variables in the responseTable cache, judge whether they have been returned, and carry out corresponding processing.

public void scanResponseTable() {
        final List<ResponseFuture> rfList = new LinkedList<ResponseFuture>();
        Iterator<Entry<Integer, ResponseFuture>> it = this.responseTable.entrySet().iterator();
        while (it.hasNext()) {
            Entry<Integer, ResponseFuture> next = it.next();
            ResponseFuture rep = next.getValue();

            if ((rep.getBeginTimestamp() + rep.getTimeoutMillis() + 1000) <= System.currentTimeMillis()) {
                rep.release();
                it.remove();
                rfList.add(rep);
                log.warn("remove timeout request, " + rep);
            }
        }

        for (ResponseFuture rf : rfList) {
            try {
                executeInvokeCallback(rf);
            } catch (Throwable e) {
                log.warn("scanResponseTable, operationComplete Exception", e);
            }
        }
    }

3.2. Specific implementation of receiving and processing messages on the Server side

The Server port receives the processing entrance of the message in the NettyServerHandler class channelRead0 method, which calls the processMessageReceived method (here omitted most of the flow and logic of the Netty service side message flow). The most important request processing method of the server is as follows:

public void processRequestCommand(final ChannelHandlerContext ctx, final RemotingCommand cmd) {
    //Get processor and ExecutorService according to code in RemotingCommand
    final Pair<NettyRequestProcessor, ExecutorService> matched = this.processorTable.get(cmd.getCode());
    final Pair<NettyRequestProcessor, ExecutorService> pair = null == matched ? this.defaultRequestProcessor : matched;
    final int opaque = cmd.getOpaque();

    if (pair != null) {
        Runnable run = new Runnable() {
            @Override
            public void run() {
                try {
                    //rpc hook
                    RPCHook rpcHook = NettyRemotingAbstract.this.getRPCHook();
                    if (rpcHook != null) {
                        rpcHook.doBeforeRequest(RemotingHelper.parseChannelRemoteAddr(ctx.channel()), cmd);
                    }
                    //processor processes requests
                    final RemotingCommand response = pair.getObject1().processRequest(ctx, cmd);
                    //rpc hook
                    if (rpcHook != null) {
                        rpcHook.doAfterResponse(RemotingHelper.parseChannelRemoteAddr(ctx.channel()), cmd, response);
                    }

                    if (!cmd.isOnewayRPC()) {
                        if (response != null) {
                            response.setOpaque(opaque);
                            response.markResponseType();
                            try {
                                ctx.writeAndFlush(response);
                            } catch (Throwable e) {
                                PLOG.error("process request over, but response failed", e);
                                PLOG.error(cmd.toString());
                                PLOG.error(response.toString());
                            }
                        } else {

                        }
                    }
                } catch (Throwable e) {
                    if (!"com.aliyun.openservices.ons.api.impl.authority.exception.AuthenticationException"
                        .equals(e.getClass().getCanonicalName())) {
                        PLOG.error("process request exception", e);
                        PLOG.error(cmd.toString());
                    }

                    if (!cmd.isOnewayRPC()) {
                        final RemotingCommand response = RemotingCommand.createResponseCommand(RemotingSysResponseCode.SYSTEM_ERROR, //
                            RemotingHelper.exceptionSimpleDesc(e));
                        response.setOpaque(opaque);
                        ctx.writeAndFlush(response);
                    }
                }
            }
        };

        if (pair.getObject1().rejectRequest()) {
            final RemotingCommand response = RemotingCommand.createResponseCommand(RemotingSysResponseCode.SYSTEM_BUSY,
                "[REJECTREQUEST]system busy, start flow control for a while");
            response.setOpaque(opaque);
            ctx.writeAndFlush(response);
            return;
        }

        try {
            //Encapsulate the requestTask
            final RequestTask requestTask = new RequestTask(run, ctx.channel(), cmd);
            //Submit requestTask to thread pool
            pair.getObject2().submit(requestTask);
        } catch (RejectedExecutionException e) {
            if ((System.currentTimeMillis() % 10000) == 0) {
                PLOG.warn(RemotingHelper.parseChannelRemoteAddr(ctx.channel()) //
                    + ", too many requests and system thread pool busy, RejectedExecutionException " //
                    + pair.getObject2().toString() //
                    + " request code: " + cmd.getCode());
            }

            if (!cmd.isOnewayRPC()) {
                final RemotingCommand response = RemotingCommand.createResponseCommand(RemotingSysResponseCode.SYSTEM_BUSY,
                    "[OVERLOAD]system busy, start flow control for a while");
                response.setOpaque(opaque);
                ctx.writeAndFlush(response);
            }
        }
    } else {
        String error = " request type " + cmd.getCode() + " not supported";
        //Build response
        final RemotingCommand response =
            RemotingCommand.createResponseCommand(RemotingSysResponseCode.REQUEST_CODE_NOT_SUPPORTED, error);
        response.setOpaque(opaque);
        ctx.writeAndFlush(response);
        PLOG.error(RemotingHelper.parseChannelRemoteAddr(ctx.channel()) + error);
    }
}

In the above request processing method, the request business code of RemotingCommand is matched to the corresponding business processor; then a new thread is generated and submitted to the corresponding business thread pool for asynchronous processing.

(1) processorTable -- mapping variables of request business code, business processing and business thread pool

protected final HashMap<Integer/* request code */, Pair<NettyRequestProcessor, ExecutorService>> processorTable =
    new HashMap<Integer, Pair<NettyRequestProcessor, ExecutorService>>(64);

I think RocketMQ's purpose is to specify different Processor processing for different types of request business codes. At the same time, the actual processing of messages is not in the current thread, but is encapsulated as task s and put into the corresponding thread pool of the business Processor to complete asynchronous execution. (in RocketMQ, we can see that this kind of processing is used in many places. This kind of design can guarantee the maximum asynchrony and ensure that each thread focuses on what it is responsible for.)

3.3. Implementation analysis of asynchronous callback execution on Client side

You can see that some students may wonder where the asynchronous callback on the Client end is executed? From the above "overall sequence diagram of RocketMQ asynchronous communication", the callback execution process is indeed completed on the Client side, while the RocketMQ remoting communication module only provides an interface for asynchronous callback processing. Here you can see part of the code of asynchronous message sending by RocketMQ Client module (only part of the code of asynchronous callback execution is listed for space):

private void sendMessageAsync(
        final String addr,
        final String brokerName,
        final Message msg,
        final long timeoutMillis,
        final RemotingCommand request,
        final SendCallback sendCallback,
        final TopicPublishInfo topicPublishInfo,
        final MQClientInstance instance,
        final int retryTimesWhenSendFailed,
        final AtomicInteger times,
        final SendMessageContext context,
        final DefaultMQProducerImpl producer
    ) throws InterruptedException, RemotingException {
        this.remotingClient.invokeAsync(addr, request, timeoutMillis, new InvokeCallback() {
            @Override
            public void operationComplete(ResponseFuture responseFuture) {
                //First, get the value of RemotingCommand from the responseFuture variable returned by the Server
                RemotingCommand response = responseFuture.getResponseCommand();
              if (null == sendCallback && response != null) {

                    try {
                        //The Client side processes the Reponse return of the sent message (including decoding the header of the message return body,
                        //Get "topic", "BrokerName", "QueueId" equivalent)
                        //Then build the sendResult object and set it in the Context
                        SendResult sendResult = MQClientAPIImpl.this.processSendResponse(brokerName, msg, response);
                        if (context != null && sendResult != null) {
                            context.setSendResult(sendResult);
                            context.getProducer().executeSendMessageHookAfter(context);
                        }
                    } catch (Throwable e) {
                    }

                    producer.updateFaultItem(brokerName, System.currentTimeMillis() - responseFuture.getBeginTimestamp(), false);
                    return;
                }
            //Omit other codes
            //......
}

Here, we need to combine the content of Section 3.1 with the processResponseCommand method of the NettyRemotingAbstract abstract class to understand the general process of implementing asynchronous callback on the Client side. When the Client sends asynchronous messages (when the rocketmq Client module finally calls the sendMessageAsync method), the interface of InvokeCallback will be injected. When the asynchronous thread on the Server is actually executed by the business thread pool mentioned above, the execution will be triggered when the response is returned to the Client. The specific code of processResponseCommand method of NettyRemotingAbstract abstract class is as follows:

public void processResponseCommand(ChannelHandlerContext ctx, RemotingCommand cmd) {
        //Get the opaque value from RemotingCommand
        final int opaque = cmd.getOpaque();'
        //Take the ResponseFuture variable corresponding to the asynchronous communication connection from the local cached responseTable Map
        final ResponseFuture responseFuture = responseTable.get(opaque);
        if (responseFuture != null) {
            responseFuture.setResponseCommand(cmd);

            responseTable.remove(opaque);

            if (responseFuture.getInvokeCallback() != null) {
                //The asynchronous callback method injected by the Client is actually executed here
                executeInvokeCallback(responseFuture);
            } else {
                //Otherwise, release the responseFuture variable
                responseFuture.putResponse(cmd);
                responseFuture.release();
            }
        } else {
            log.warn("receive response, but not matched any request, " + RemotingHelper.parseChannelRemoteAddr(ctx.channel()));
            log.warn(cmd.toString());
        }
    }

3, Summary

At the beginning, the RocketMQ source code RPC communication module may feel a little complex, but as long as it can grasp the process of sending request messages on the Client side, receiving and processing messages on the Server side and callback process to analyze and sort out, it is not complex as a whole. RPC communication part is also one of the most important parts of RocketMQ source code. If you want to have a deeper understanding of the whole process and details, you need to Debug and analyze the corresponding logs in the local environment. At the same time, in view of the limited space, this paper has not yet introduced RocketMQ's Netty multithreading model, which will be introduced in detail in the message middleware - RocketMQ's RPC communication (2). Here, I'll call myself a Call. Interested friends can pay attention to my personal official account: "a unique blog". Articles on Java concurrency, Spring, database and message queue will be published on this official account. Welcome to exchange and discuss.

 

Tags: Netty encoding network Java

Posted on Sat, 06 Jun 2020 05:51:09 -0400 by fsumba