Golang net/rpc package learning

golang provides an out of the box RPC service, which is simple but not simple.

RPC introduction

Remote Procedure Call (RPC) is a kind of computer communication protocol. It allows programs running on another computer to call subroutines in another address space (generally a computer in an open network), while programmers are just like calling local programs without any additional interactive programming. RPC is a mode of CS (client server) architecture, which interacts information by sending request and receiving response.

There are many widely used RPC frameworks, such as gRPC, Thrift, Dubbo, brpc, etc. Some of the RPC frameworks here implement cross language calls, some implement service registration discovery, etc. It is much more widely used than the official RPC package we introduced today. However, through the study of net/rpc, we can make a basic understanding of an RPC framework.

The implementation of Golang

RPC is a cs architecture, so it has both client and server. Next, we first analyze the coding of communication, and then analyze the implementation of RPC from the perspective of server and client.

Communication code

In rpc implementation, golang abstracts the protocol layer. We can customize the protocol to implement our own interface. The interface of the protocol is as follows:

// Server
type ServerCodec interface {
  ReadRequestHeader(*Request) error
  ReadRequestBody(interface{}) error
  WriteResponse(*Response, interface{}) error

  // Close can be called multiple times and must be idempotent.
  Close() error
// client
type ClientCodec interface {
  WriteRequest(*Request, interface{}) error
  ReadResponseHeader(*Response) error
  ReadResponseBody(interface{}) error

  Close() error

The package provides the implementation of encoding and decoding based on gob binary coding. Of course, we can also realize the encoding and decoding methods we want.

Server side implementation

Structure definition

type Server struct {
  serviceMap sync.Map   // Save Service
  reqLock    sync.Mutex // Read requested lock
  freeReq    *Request
  respLock   sync.Mutex // Write response lock
  freeResp   *Response

The server supports concurrent execution by means of mutex. Since each request and response needs to define a Request/Response object, in order to reduce memory allocation, a freeReq/freeResp linked list is used to implement two object pools.
When a Request object is needed, it is obtained from the freeReq linked list and put back into the linked list after use.

Registration of services

Service is saved in the serviceMap of the Server. The information of each service is as follows:

type service struct {
  name   string                 // service name
  rcvr   reflect.Value          // service object 
  typ    reflect.Type           // Service type
  method map[string]*methodType // Registration method

As you can see from the above, a type and multiple methods of that type can be registered as a Service. When registering a Service, save the Service in the serviceMap through the following methods.

// Use object method name by default
func (server *Server) Register(rcvr interface{}) error {}
// Specify method name
func (server *Server) RegisterName(name string, rcvr interface{}) error {}

Service call

First, the start of rpc service. Consistent with most network applications, after accept ing a connection, a collaboration will be started for message processing. The code is as follows:

for {
  conn, err := lis.Accept()
  if err != nil {
    log.Print("rpc.Serve: accept:", err.Error())
  go server.ServeConn(conn)

Secondly, for each connection, the server will continuously obtain the request and send the response asynchronously. The code is as follows:

for {
  // Read request
  service, mtype, req, argv, replyv, keepReading, err := server.readRequest(codec)
  if err != nil {
    if debugLog && err != io.EOF {
      log.Println("rpc:", err)
    if !keepReading {
    if req != nil {
      // Send request
      server.sendResponse(sending, req, invalidRequest, codec, err.Error())
      server.freeRequest(req)  // Release req object
  // Process each request concurrently
  go service.call(server, sending, wg, mtype, req, argv, replyv, codec)

Finally, because requests are sent asynchronously, the order of the requests and the order of the responses are not necessarily the same. Therefore, in the response message, the seq (serial number) of the request message will be carried to ensure the consistency of the message.
In addition, in order to be compatible with http services, net/rpc package also converts http protocol into rpc Protocol through the Hijack method implemented by http package. The code is as follows:

func (server *Server) ServeHTTP(w http.ResponseWriter, req *http.Request) {
  // The client connects through the CONNECT method

  // Get the tcp connection through Hijack
  conn, _, err := w.(http.Hijacker).Hijack()
  if err != nil {
    log.Print("rpc hijacking ", req.RemoteAddr, ": ", err.Error())
  // Send client, support RPC Protocol
  io.WriteString(conn, "HTTP/1.0 "+connected+"\n\n")

  // Start request response for RPC

Client side implementation

The connection of the client is simpler than that of the server. We learn from three aspects: initiating connection, sending request and reading response.

RPC connections

Since this RPC supports HTTP protocol for connection upgrade, there are several connection modes.

  1. Use the tcp protocol directly.

    func Dial(network, address string) (*Client, error) {}
  2. Use the http protocol. The http protocol can specify a path or use the default rpc path.

    // Default path "/_ goRPC_"
    func DialHTTP(network, address string) (*Client, error) {}
    // Use default path
    func DialHTTPPath(network, address, path string) (*Client, error) {}

Sending of request

The sending of RPC requests provides synchronous and asynchronous interface calls as follows:

// asynchronous
func (client *Client) Go(serviceMethod string, args interface{}, reply interface{}, done chan *Call) *Call {}
// synchronization
func (client *Client) Call(serviceMethod string, args interface{}, reply interface{}) error{}

From the internal implementation, it can be seen that the return data is obtained asynchronously through Go.

Next, let's see how to send the request internally:

func (client *Client) send(call *Call) {
  // Under normal conditions of client
  seq := client.seq
  client.seq++  // Requested serial number
  client.pending[seq] = call

    // Code the request, including the request method and parameters.
  // Encode and send the request.
  client.request.Seq = seq
  client.request.ServiceMethod = call.ServiceMethod

  // client can initiate Request concurrently and wait for Done asynchronously
    err := client.codec.WriteRequest(&client.request, call.Args)
    // Whether there is a sending failure. If the sending is successful, it will be saved in the pending map and wait for the request result.
  if err != nil {
    call = client.pending[seq]
    delete(client.pending, seq)
    if call != nil {
      call.Error = err

As you can see from the above, for a client, you can send multiple requests at the same time, and then wait for the response asynchronously.

Read response

After the rpc connection is successful, a connection will be established for reading the response.

for err == nil {
  response = Response{}
  err = client.codec.ReadResponseHeader(&response)
  if err != nil {
  seq := response.Seq
  call := client.pending[seq] // Remove from the pending list
  delete(client.pending, seq)
  // Decode body
  // There are many kinds of judgments here to judge whether there is any abnormality
  // Finally, the asynchronous waiting request is notified, and the call is completed

The response header is read circularly to respond to the body, and the read result is notified to the asynchronous request calling rpc to complete the read of the response once.

Simple example

Here is a simple example provided by our government to summarize rpc package learning.


type Args struct {  // Request parameters
  A, B int

type Quotient struct {  // Type of a response
  Quo, Rem int

type Arith int

// Multiplication and division are defined
func (t *Arith) Multiply(args *Args, reply *int) error {
  *reply = args.A * args.B
  return nil

func (t *Arith) Divide(args *Args, quo *Quotient) error {
  if args.B == 0 {
    return errors.New("divide by zero")
  quo.Quo = args.A / args.B
  quo.Rem = args.A % args.B
  return nil

func main() {
  serv := rpc.NewServer()
  arith := new(Arith)
  serv.Register(arith)  // Service registration

    // Listen through http, and then do protocol conversion
  http.ListenAndServe("", serv)


func main() {
  client, err := rpc.DialHTTP("tcp", "")
  if err != nil {
    log.Fatal("dialing:", err)

  dones := make([]chan *rpc.Call, 0, 10)

  // Start the request synchronously first
  for i := 0; i < 10; i++ {
    quotient := new(Quotient)
    args := &Args{i + 10, i}
    divCall := client.Go("Arith.Divide", args, quotient, nil)
    dones = append(dones, divCall.Done)
    log.Print("send", i)

  // Asynchronous read after
  for idx, done := range dones {
    replyCall := <-done // will be equal to divCall
    args := replyCall.Args.(*Args)
    reply := replyCall.Reply.(*Quotient)
    log.Printf("%d / %d = %d, %d %% %d = %d\n", args.A, args.B, reply.Quo,
      args.A, args.B, reply.Rem)
    log.Print("recv", idx)

What can we learn

Finally, make a learning summary.

  1. Asynchronous operation is implemented for different requests on the unified connection, and data consistency needs to be ensured through request and response.
  2. Realization of an object pool by linked list
  3. A simple practice of implementing the hijack method in http package, upgrading to rpc Protocol through http protocol. Hijack the tcp connection of the original http protocol and use it for rpc.
  4. rpc implementation, through gob coding, should not support communication with other languages. We need to realize the encoding and decoding mode by ourselves.
  5. The implementation of rpc does not support the registration and discovery of services, so we need to maintain the service provider ourselves.

Tags: Go codec network encoding Programming

Posted on Sat, 27 Jun 2020 23:01:36 -0400 by lasith