Handwritten RPC core module network protocol module -- user defined protocol

Custom protocol?
When I first heard this term, I felt so tall! After learning this class later, I found that the so-called user-defined protocol is to define a set of data transmission rules. So you don't necessarily understand

We know that only binary can be transmitted in the network, so before RPC requests are sent to the network, they need to convert the request parameters of method calls into binary. However, during the transmission process, RPC does not send all binary data of the request parameters to the opposite machine at once, and may be split into several packets, Other requested packets may also be merged (the premise of merging is the same)
As for how to split and merge, the details will involve the system parameter configuration and TCP window size. For the service provider application, it will receive a lot of binary data from the TCP channel. At this time, how to identify which binary is the first request? So here comes the first essential element of a custom protocol.

The message length identifies the length of the message. When the server receives the message, it first reads the data at a fixed location to obtain the message length, and then knows where to start reading the message and how much to read. The simplest message protocol is as follows

But that's not enough. Why?
As we mentioned earlier, our may support multiple serialization methods. If there is no serialization method identification in the protocol, we can't deserialize the data correctly even if we get it. Therefore, the serialization method identification should also be added to the protocol header.
Since it is a custom protocol, should we have a fixed ID? This message is based on our protocol, just like looking at the type of a file should not just look at the suffix of the file. For different types of documents, we will use a fixed number or number at the beginning of the document. This sign has a unified magic number.
You think that's what a custom protocol looks like?

A good agreement should consider not only the present, but also the future!
Our protocol also needs a protocol version. We may encode messages in different ways at different times. In order to ensure the scalability of the protocol, we can also define the length of a protocol header to facilitate the extension of the content of the protocol header in the future. It's not difficult to write code, but it's not easy to write good code.
Based on the principle of simplicity, our custom protocol is as long as this

Having said so much, some people may want to ask, is the http protocol not good enough? Do you think you can write better than http protocol?

  1. Compared with the use of HTTP, RPC is more responsible for the communication between applications, so
    The performance requirements are relatively higher.
  2. The packet size of HTTP protocol is much larger than the request data itself, and needs to be added
    A lot of useless content, such as line feed, carriage return, etc.
  3. Another more important reason is that HTTP protocol belongs to stateless protocol, and the client cannot associate the request and response. Each request needs to re-establish the connection, and then close the connection after the response is completed.

Having said that, we can implement this custom protocol. A complete custom protocol contains the protocol header and message body. Therefore, the custom protocol is composed of the protocol header and message content.

package com.info.protocol.netty.core;

import lombok.Getter;
import lombok.Setter;

public class Header {

        |Magic number 16bit | protocol version 8bit | serialization method 8bit | protocol header length 16bit | message length 32bit | message type (request or response) 2bit|

    private short magic;

    private byte protocolVersion;

    private byte serializeType;

    private short protocolHeaderLength;

    private int messageLength;

    private byte messageType;

package com.info.protocol.netty.core;

import lombok.Getter;
import lombok.Setter;

public class Protocol<T> {
    private Header header;
    private T content;

Define an enumeration to facilitate the management of message types

package com.info.protocol.enums;

import lombok.AllArgsConstructor;
import lombok.Getter;

import java.util.Arrays;

public enum MessageTypeEnum {
    REQUEST((byte) 1),

    RESPONSE((byte) 2),

    heart_beat((byte) 3);

    private byte code;

    public static MessageTypeEnum getMessageTypeEnumByCode(int code) {
        return Arrays.stream(MessageTypeEnum.values())
                .filter(e -> e.getCode() == code)

That's all for this section. If you want to know what will happen next, listen to the decomposition of the next time (shooting Xingmu)!

Tags: network rpc Middleware Network Protocol

Posted on Sun, 05 Dec 2021 08:24:20 -0500 by wmguk