Custom Codec for Netty Source Analysis

Protocol resolution is essential in daily network development. Although length-based, delimiter-based codecs are built into Netty, in most scenarios we use custom protocols, so Netty provides MessageToByteEncoder<I> and ByteToMessageDecoder Two abstract classes that encode and decode private protocols by inheriting overrides of their encode and decode methods.In this article, we will practice and analyze the custom codec in Netty.

I. Use of codecs

Below is a simple example of the use of MessageToByteEncoder and ByteToMessageDecoder, with no specific protocol coding involved.

Create a sever-side service

EventLoopGroup bossGroup = new NioEventLoopGroup(1);
        EventLoopGroup workerGroup = new NioEventLoopGroup();
        final CodecHandler codecHandler = new CodecHandler();
        try {
            ServerBootstrap b = new ServerBootstrap();
            b.group(bossGroup, workerGroup).channel(NioServerSocketChannel.class).option(ChannelOption.SO_BACKLOG, 100)
                    .handler(new LoggingHandler(LogLevel.INFO)).childHandler(new ChannelInitializer<SocketChannel>() {
                        @Override
                        public void initChannel(SocketChannel ch) throws Exception {
                            ChannelPipeline p = ch.pipeline();
                            if (sslCtx != null) {
                                p.addLast(sslCtx.newHandler(ch.alloc()));
                            }
                            //Add Codec handler
                            p.addLast(new MessagePacketDecoder(),new MessagePacketEncoder());
                            //Add Customization handler
                            p.addLast(codecHandler);
                        }
                    });

            // Start the server.
            ChannelFuture f = b.bind(PORT).sync();

Inherit MessageToByteEncoder and override the encode method for encoding

public class MessagePacketEncoder extends MessageToByteEncoder<byte[]> {

    @Override
    protected void encode(ChannelHandlerContext ctx, byte[] bytes, ByteBuf out) throws Exception {
        //Processing specific encoding Print the byte array here
        System.out.println("Encoder receives data:"+BytesUtils.toHexString(bytes));
        //Write and transfer data
        out.writeBytes(bytes);
    }
}

Inherit ByteToMessageDecoder and override decode method for decoding

public class MessagePacketDecoder extends ByteToMessageDecoder {
    @Override
    protected void decode(ChannelHandlerContext ctx, ByteBuf buffer, List<Object> out){
        try {
            if (buffer.readableBytes() > 0) {
                // Message Packets to Process
                byte[] bytesReady = new byte[buffer.readableBytes()];
                buffer.readBytes(bytesReady);
                //Conduct specific decoding
                System.out.println("Decoder receives data:"+ByteUtils.toHexString(bytesReady));
                //Don't overdo it here, just put the received message in the list and pass it back
                out.add(bytesReady);
            
            }
        }catch(Exception ex) {
            
        }

    }

}

Implement a custom message processing handler, where you actually get the coded data

public class CodecHandler extends ChannelInboundHandlerAdapter{
    @Override
    public void channelRead(ChannelHandlerContext ctx, Object msg) {
        System.out.println("CodecHandler Data received:"+ByteUtils.toHexString((byte[])msg));
        byte[] sendBytes = new byte[] {0x7E,0x01,0x02,0x7e};
        ctx.write(sendBytes);
    }

    @Override
    public void channelReadComplete(ChannelHandlerContext ctx) {
        ctx.flush();
    }

    @Override
    public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
        // Close the connection when an exception is raised.
        cause.printStackTrace();
        ctx.close();
    }
}

Run a client simulation to send bytes 0x01,0x02 to see the output execution results

Decoder receives data: 0102
 CodecHandler received data: 0102
 Encoder receives data: 7E01027E

Based on the output, you can see that the inbound and outbound messages are delivered in a sequence customized in the pipeline, and the specific protocol codec operations we need are implemented by rewriting the encode and decode methods.

2. Source Code Analysis

From the example above, you can see that MessageToByteEncoder <I> and ByteToMessageDecoder inherit ChannelInboundHandlerAdapter and ChannelOutboundHandlerAdapter respectively, so they are also implementations of channelHandler.And is added to the pipeline when sever is created.At the same time, for our convenience, netty has built-in and encapsulated some of its operations in these two abstract classes; outbound and inbound messages trigger both the write and channelRead event methods, respectively, so the encode and decode methods we override in the above example are also called in the write and channelRead methods of the parent class. Let's start with these two methods separately and edit the whole programThe decoding process is sorted out and analyzed.

1,MessageToByteEncoder

Encoding requires outbound data, so our overridden encode implementation is invoked in the write method of MesageToByteEncoder to encode our internally defined message entity as the final byte stream data to be sent.

    @Override
    public void write(ChannelHandlerContext ctx, Object msg, ChannelPromise promise) throws Exception {
        ByteBuf buf = null;
        try {
            if (acceptOutboundMessage(msg)) {//Determine incoming msg Is it consistent with the type you defined
                @SuppressWarnings("unchecked")
                I cast = (I) msg;//Convert to message type defined by you
                buf = allocateBuffer(ctx, cast, preferDirect);//Package as one ByteBuf
                try {
                    encode(ctx, cast, buf);//Incoming declarative ByteBuf,Perform specific encoding operations
                } finally {
                    /**
                     * If the type you define is ByteBuf, this can help you free resources without having to do it yourself
                     * If you define a message type that contains ByteBuf, this is not useful and you need to release it yourself
                     */
                    ReferenceCountUtil.release(cast);//Release your incoming resources
                }

                //Send out buf
                if (buf.isReadable()) {
                    ctx.write(buf, promise);
                } else {
                    buf.release();
                    ctx.write(Unpooled.EMPTY_BUFFER, promise);
                }
                buf = null;
            } else {
                //Send directly without execution if type is inconsistent encode Method, so here's a note that if you pass a message that doesn't match the generic type, it won't actually execute
                ctx.write(msg, promise);
            }
        } catch (EncoderException e) {
            throw e;
        } catch (Throwable e) {
            throw new EncoderException(e);
        } finally {
            if (buf != null) {
                buf.release();//Release Resources
            }
        }
    }

The MessageToByteEncoder's write method achieves the simple function of converting and sending your incoming data types; here are two points to note:

  • Typically, you need to override the encode method to convert the generic type you defined to the ByteBuf type, and the write method automatically performs a pass or send operation for you.
  • Although passed in code ReferenceCountUtil.release(cast) Release the type resource you defined, but if the defined message class contains a ByteBuf object, you still need to actively release the object resource;

2,ByteToMessageDecoder

As you can see from the naming, the ByteToMessageDecoder is used to convert the byte stream data encoding to the data format we need

As an inbound event, the entry to the decoding operation is naturally the channelRead method

 @Override
    public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
        if (msg instanceof ByteBuf) {//If the message is bytebuff
            CodecOutputList out = CodecOutputList.newInstance();//Instantiate a list of chains
            try {
                ByteBuf data = (ByteBuf) msg;
                first = cumulation == null;
                if (first) {
                    cumulation = data;
                } else {
                    cumulation = cumulator.cumulate(ctx.alloc(), cumulation, data);
                }
                callDecode(ctx, cumulation, out);//Start decoding
            } catch (DecoderException e) {
                throw e;
            } catch (Exception e) {
                throw new DecoderException(e);
            } finally {
                if (cumulation != null && !cumulation.isReadable()) {//Not empty and no readable data, freeing up resources
                    numReads = 0;
                    cumulation.release();
                    cumulation = null;
                } else if (++ numReads >= discardAfterReads) {
                    // We did enough reads already try to discard some bytes so we not risk to see a OOME.
                    // See https://github.com/netty/netty/issues/4275
                    numReads = 0;
                    discardSomeReadBytes();
                }

                int size = out.size();
                decodeWasNull = !out.insertSinceRecycled();
                fireChannelRead(ctx, out, size);//Send message down
                out.recycle();
            }
        } else {
            ctx.fireChannelRead(msg);
        }
    }

The callDecode method decodes ByteBuf data internally through a while loop until there is no readable data in it

    protected void callDecode(ChannelHandlerContext ctx, ByteBuf in, List<Object> out) {
        try {
            while (in.isReadable()) {//judge ByteBuf Yes, there is still readable data
                int outSize = out.size();//Get record list size

                if (outSize > 0) {//Determine if there is already data in the list
                    fireChannelRead(ctx, out, outSize);//If any data continues to be passed down
                    out.clear();//Empty Chain List

                    // Check if this handler was removed before continuing with decoding.
                    // If it was removed, it is not safe to continue to operate on the buffer.
                    //
                    // See:
                    // - https://github.com/netty/netty/issues/4635
                    if (ctx.isRemoved()) {
                        break;
                    }
                    outSize = 0;
                }

                int oldInputLength = in.readableBytes();
                decodeRemovalReentryProtection(ctx, in, out);//Start Call decode Method

                // Check if this handler was removed before continuing the loop.
                // If it was removed, it is not safe to continue to operate on the buffer.
                //
                // See https://github.com/netty/netty/issues/1664
                if (ctx.isRemoved()) {
                    break;
                }

                //Here if the list is empty and bytebuf Jump out of the loop without readable data
                if (outSize == out.size()) {
                    if (oldInputLength == in.readableBytes()) {
                        break;
                    } else {//Readable data to continue reading
                        continue;
                    }
                }

                if (oldInputLength == in.readableBytes()) {//beytebuf Not read, but decoded
                    throw new DecoderException(
                            StringUtil.simpleClassName(getClass()) +
                                    ".decode() did not read anything but decoded a message.");
                }

                if (isSingleDecode()) {//Whether to set each inbound data to be decoded only once, default false
                    break;
                }
            }
        } catch (DecoderException e) {
            throw e;
        } catch (Exception cause) {
            throw new DecoderException(cause);
        }
    }

The decodeRemovalReentryProtection method calls our overridden decode implementation internally

    final void decodeRemovalReentryProtection(ChannelHandlerContext ctx, ByteBuf in, List<Object> out)
            throws Exception {
        decodeState = STATE_CALLING_CHILD_DECODE;//Tag Status
        try {
            decode(ctx, in, out);//Call us to override decode Decoding implementation
        } finally {
            boolean removePending = decodeState == STATE_HANDLER_REMOVED_PENDING;
            decodeState = STATE_INIT;
            if (removePending) {//Judge markers here to prevent handlerRemoved Events conflict with decoding operations
                handlerRemoved(ctx);
            }
        }
    }

The channelRead method accepts data through a series of logical processing and eventually invokes our overridden decode method to implement specific decoding functions; in the decode method, we only need the data of type ByteBuf to be parsed into the data format we need to be placed directly in the List <Object> out chain table, and ByteToMessageDecoder will automatically help you pass messages down.

3. Summary

From the explanations above, we can learn a little about the built-in custom codecs MessageToByteEncoder and ByteToMessageDecoder in Netty, which are essentially a set of channelHandler implementation classes encapsulated by Netty specifically for custom coding.In the actual development, the implementation based on these two abstract classes is very practical, so we will make a little analysis here, in which some deficiencies and inaccuracies are expected to point out and culvert.

 

Watch the WeChat Public Number for more technical articles.

 

 

Reproduction instructions: Unauthorized not to reproduce, after authorization must indicate the source (Note: from public number: architecture space, author: generic)

Tags: Java Netty encoding codec github

Posted on Mon, 22 Jun 2020 22:05:16 -0400 by jalapena