Okhttp source code reading

First look at the usage:

val client = OkHttpClient.Builder().build()
val request = Request.Builder()
	.url("https://www.baidu.com")
	.build()
val call = client.newCall(request)
call.enqueue(object : okhttp3.Callback {
    override fun onFailure(call: okhttp3.Call, e: IOException) {

    }

    override fun onResponse(call: okhttp3.Call, response: okhttp3.Response) {
        Log.e("TAG", "onResponse: ${response.body?.string()}")
    }
})

First, a client is created through the builder mode. Then a call is created through the newCall method, and finally a network request is made through the call.

A parameter passed in the newCall method is Request, which is spelled by itself. This method is to create a standby network Request through Request.

override fun newCall(request: Request): Call {
  return RealCall.newRealCall(this, request, forWebSocket = false)
}

// RealCall 
companion object {
    fun newRealCall(
      client: OkHttpClient,  originalRequest: Request,   forWebSocket: Boolean
    ): RealCall {
        //Create a RealCall 
        //client, request, webSocket are generally not used						
      return RealCall(client, originalRequest, forWebSocket).apply {
          //Transmitter
        transmitter = Transmitter(client, this)
      }
    }
  }

In the newRealCall method, a RealCall object is returned. Therefore, RealCall is the implementation class of Call.

enqueue method

call.enqueue(object : okhttp3.Callback {
    override fun onFailure(call: okhttp3.Call, e: IOException) {
    }
    override fun onResponse(call: okhttp3.Call, response: okhttp3.Response) {
        Log.e("TAG", "onResponse: ${response.body?.string()}")
    }
})

//RealCall 
  override fun enqueue(responseCallback: Callback) {
    synchronized(this) {
      check(!executed) { "Already Executed" }
      executed = true
    }
    transmitter.callStart()
    client.dispatcher.enqueue(AsyncCall(responseCallback))
  }

In enqueue, the callback is passed to dispath, and an asynchronous Call is created. Let's take a look at the dispatcher, enqueue and AsyncCall respectively

  • dispatcher

    val dispatcher: Dispatcher = builder.dispatcher
    
    //Policy when asynchronous requests are executed. Each dispatcher uses one [ExecutorService] to run the call internally.
    //If you provide your own actuator, it should be able to run [configured maximum] [maxRequests] calls at the same time.
    class Dispatcher constructor() {
        
      // The maximum number of requests, after which we will wait
      @get:Synchronized var maxRequests = 64
      //...... 
        
      //The maximum number of requests from the host to prevent too much pressure on the server
      @get:Synchronized var maxRequestsPerHost = 5
        
      @get:Synchronized
      @get:JvmName("executorService") val executorService: ExecutorService
        get() {
          if (executorServiceOrNull == null) {
          executorServiceOrNull = ThreadPoolExecutor(0, Int.MAX_VALUE, 60, TimeUnit.SECONDS,
                SynchronousQueue(), threadFactory("OkHttp Dispatcher", false))
        }
          return executorServiceOrNull!!
        }
    }
    

    Dispatcher is mainly used to manage threads. Each new request requires a separate thread, so that different requests will not be blocked.

    His internal implementation uses ExecutorService,

  • enqueue of dispatcher

    internal fun enqueue(call: AsyncCall) {
      synchronized(this) {
        readyAsyncCalls.add(call)
    	//.....
      }
      promoteAndExecute()
    }
    

    Call readyAsyncCalls, which is a standby queue ready for execution.

    private fun promoteAndExecute(): Boolean {
      assert(!Thread.holdsLock(this))
    
      val executableCalls = mutableListOf<AsyncCall>()
      val isRunning: Boolean
      synchronized(this) {
        val i = readyAsyncCalls.iterator()
         // Traverse standby queue
        while (i.hasNext()) {
          val asyncCall = i.next()
    	  //Determine the maximum number of connections and the number of host s, which do not meet the direct break	
          if (runningAsyncCalls.size >= this.maxRequests) break // Max capacity.
          if (asyncCall.callsPerHost().get() >= this.maxRequestsPerHost) continue // Host max capacity.
    	 // Remove from standby	
          i.remove()
          asyncCall.callsPerHost().incrementAndGet()
           //Add asynCall to list and run queue
          executableCalls.add(asyncCall)
          runningAsyncCalls.add(asyncCall)
        }
        isRunning = runningCallsCount() > 0
      }
     
      for (i in 0 until executableCalls.size) {
        val asyncCall = executableCalls[i]
        // Call executeOn of AsyncCall to start execution
        asyncCall.executeOn(executorService)
      }
      return isRunning
    }
    

    The enqueue of dispatcher is mainly to judge the maximum number of connections and host, then add asynccall to a list and run queue, and finally traverse the list to call the executeOn method of asynccall,

    Let's take a look at AsyncCall

  • AsyncCall

    internal inner class AsyncCall( private val responseCallback: Callback) : Runnable {
    	
        fun executeOn(executorService: ExecutorService) {
          assert(!Thread.holdsLock(client.dispatcher))
          var success = false
          try {
            //implement  
            executorService.execute(this)
            success = true
          } catch (e: RejectedExecutionException) {
            val ioException = InterruptedIOException("executor rejected")
            ioException.initCause(e)
            transmitter.noMoreExchanges(ioException)
            responseCallback.onFailure(this@RealCall, ioException)
          } finally {
            if (!success) {
              client.dispatcher.finished(this) // This call is no longer running!
            }
          }
        }
    
        override fun run() {
          threadName("OkHttp ${redactedUrl()}") {
            var signalledCallback = false
            transmitter.timeoutEnter()
            try {
              //  
              val response = getResponseWithInterceptorChain()
              signalledCallback = true
              //
              responseCallback.onResponse(this@RealCall, response)
            } catch (e: IOException) {
              if (signalledCallback) {
                // Do not signal the callback twice!
                Platform.get().log(INFO, "Callback failure for ${toLoggableString()}", e)
              } else {
                //
                responseCallback.onFailure(this@RealCall, e)
              }
            } finally {
              client.dispatcher.finished(this)
            }
          }
        }
      }
    }
    

    Pay attention to the executeOn method, which is called in the enqueue of dispatcher. After executorService is executed here, the corresponding run method will also be executed.

    In the run method, we use getResponseWithInterceptorChain() to make the request and get the response response and finally call callback, which is the callback that we sent in when requested.

    There are some important methods in run, let's talk about them later.

    Up to now, the general logic of using enqueue to request has been very clear. The most critical code is:

    client.dispatcher.enqueue(AsyncCall(responseCallback))
    

    The general process is as follows:

    1, create a RealCall and then call enqueue.

    2, call the enqueue of dispatcher in enqueue.

    3. The run method of AsyncCall is triggered in enqueue method of dispatcher,

    4. Request and respond in the run method

execute method

val response = call.execute()
override fun execute(): Response {
  synchronized(this) {
    check(!executed) { "Already Executed" }
    executed = true
  }
  transmitter.timeoutEnter()
  transmitter.callStart()
  try {
    client.dispatcher.executed(this)
     // Very direct, called directly.
    return getResponseWithInterceptorChain()
  } finally {
    client.dispatcher.finished(this)
  }
}

No thread cutting is required in execute, so it is called directly.

Configuration items in OkhttpClient

  • dispatcher

    Used to schedule threads, optimize performance, etc., such as maxRequest = 64, etc

  • proxy

    agent

  • protocols

    Version of supported http protocol

    enum class Protocol(private val protocol: String) {
    
      HTTP_1_0("http/1.0"),
    
      HTTP_1_1("http/1.1"),
        
      HTTP_2("h2"),
      //......
     }
    
  • connectionSpecs

    ConnectionSpecs is used to specify the socket connection during http transmission. For https connection, ConnectionSpecs is a type of Password Suite that describes the TLS version supported by the client to the server when building TLS connection,

    When connecting through https, you need to attach the TLS version, cipher Suite (acceptable symmetric, asymmetric and hash algorithms) and other data supported by the client to the server. These are in the ConnectionSpecs class.

    // List of supported password Suites
    private val RESTRICTED_CIPHER_SUITES = arrayOf(
        // TLSv1.3.
        CipherSuite.TLS_AES_128_GCM_SHA256,
        CipherSuite.TLS_AES_256_GCM_SHA384,
        CipherSuite.TLS_CHACHA20_POLY1305_SHA256,
    
        // TLSv1.0, TLSv1.1, TLSv1.2.
        CipherSuite.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
        CipherSuite.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
        CipherSuite.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
        CipherSuite.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
        CipherSuite.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,
        CipherSuite.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256)
    
    //List of supported password Suites
    private val APPROVED_CIPHER_SUITES = arrayOf(
        // TLSv1.3.
        CipherSuite.TLS_AES_128_GCM_SHA256,
        CipherSuite.TLS_AES_256_GCM_SHA384,
        CipherSuite.TLS_CHACHA20_POLY1305_SHA256,
    
        // TLSv1.0, TLSv1.1, TLSv1.2.
        CipherSuite.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
        CipherSuite.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
        CipherSuite.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
        CipherSuite.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
        CipherSuite.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,
        CipherSuite.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,
    
        // Note that the following cipher suites are all on HTTP/2's bad cipher suites list. We'll
        // continue to include them until better suites are commonly available.
        CipherSuite.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,
        CipherSuite.TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,
        CipherSuite.TLS_RSA_WITH_AES_128_GCM_SHA256,
        CipherSuite.TLS_RSA_WITH_AES_256_GCM_SHA384,
        CipherSuite.TLS_RSA_WITH_AES_128_CBC_SHA,
        CipherSuite.TLS_RSA_WITH_AES_256_CBC_SHA,
        CipherSuite.TLS_RSA_WITH_3DES_EDE_CBC_SHA)
        
         /** Secure TLS connections require the closest client and closest server */
        @JvmField
        val RESTRICTED_TLS = Builder(true)
            .cipherSuites(*RESTRICTED_CIPHER_SUITES)
            .tlsVersions(TlsVersion.TLS_1_3, TlsVersion.TLS_1_2)
            .supportsTlsExtensions(true)
            .build()
    
        /**
         *Modern TLS configuration, which is applicable to most clients and servers that can connect to, is the default configuration of OkHttp
         */
        @JvmField
        val MODERN_TLS = Builder(true)
            .cipherSuites(*APPROVED_CIPHER_SUITES)
            .tlsVersions(TlsVersion.TLS_1_3, TlsVersion.TLS_1_2)
            .supportsTlsExtensions(true)
            .build()
    
        /**
         *Backward compatible configuration compared to MODERN_TLS. There are fewer TLS versions supported. Only TLS is supported_ 1_ 0 version
         */
        @JvmField
        val COMPATIBLE_TLS = Builder(true)
            .cipherSuites(*APPROVED_CIPHER_SUITES)
            .tlsVersions(TlsVersion.TLS_1_3, TlsVersion.TLS_1_2, TlsVersion.TLS_1_1, TlsVersion.TLS_1_0)
            .supportsTlsExtensions(true)
            .build()
    
    
        /** An unencrypted, unauthenticated connection is an HTTP connection*/
        @JvmField
        val CLEARTEXT = Builder(false).build()
    

    XXXX above_ TLS configures whether you want to use http or https.

    If you want to use https, what version of tls do you use

  • interceptors´╝înetworkInterceptors

    Interceptor

  • eventListenerFactory

    Used for statistics.

  • cookieJar

    The memory of the cookie. okhttp does not store cookies by default. You need to store them yourself.

  • cache

    cache

  • socketFactory

    http itself has no port to create a TCP port.

  • certificateChainCleaner

    The certificates taken from the server sometimes get many certificates. Certificatechain cleaner is used for sorting. After sorting, it is a chain or a sequence. The last certificate is the local root certificate.

  • hostnameVerifier

    It is used to verify the host name of https to verify whether the host of the other party is the host you need to access

  • certificatePinner

    Certificate holder, used to verify self signed certificate!

  • connectionPool

    Connection pool

  • followRedirects

    Do I need to redirect when I encounter 301

  • followSslRedirects

    When you access http, but redirect to https, or access https, but redirect to http, do you want to redirect.

  • retryOnConnectionFailure

    Need to reconnect when the request fails

  • connectTimeout

    When the time of tcp connection exceeds this event, an error will be reported

  • readTimeout

    Waiting time for download response

  • writeTimeout

    Wait time for write requests

  • pingInterval

    For webSocket, it is a two-way interactive channel, which requires a long connection. pingInterval indicates how many times to ping.

getResponseWithInterceptorChain

In the enqueue and execute methods, the last call is the getResponseWithInterceptorChain method. In this method, you will make a network request and get the response. Let's take a look at the source code

fun getResponseWithInterceptorChain(): Response {
  // Get and create all interceptors
  val interceptors = mutableListOf<Interceptor>()
  interceptors += client.interceptors
  interceptors += RetryAndFollowUpInterceptor(client)
  interceptors += BridgeInterceptor(client.cookieJar)
  interceptors += CacheInterceptor(client.cache)
  interceptors += ConnectInterceptor
  if (!forWebSocket) {
    interceptors += client.networkInterceptors
  }
  interceptors += CallServerInterceptor(forWebSocket)
 
  //
  val chain = RealInterceptorChain(interceptors, transmitter, null, 0, originalRequest, this,
      client.connectTimeoutMillis, client.readTimeoutMillis, client.writeTimeoutMillis)

  //.....
    val response = chain.proceed(originalRequest)
  //......
}

First, we get the interceptors we added, and then we add many built-in interceptors.

Then we create a chain, which is a chain, and the blocker is placed in the chain.

When a request is made, it will be executed from the beginning of the chain and returned after the request is completed.

The role of chain: for example, I am the owner of a restaurant. I received an order for fast food, and then I finished the fast food and gave it to the rider in the restaurant. Then the rider sent the fast food to the customer's home. The customer received the money and gave it to the rider. The rider would take part of the money as road food and give the rest to the boss.

This is a chain. The boss is the starting end, who is responsible for making fast food and giving it to the rider. The middle node is the rider, who is responsible for sending fast food to the user. The end point is the customer, who will pay after receiving fast food.

Each node in the chain will do some processing for the request, and pass it to the next node until the end, and return to the beginning.

When a network request is made, a chain will be created. The node in the chain is the interceptor. It will process the request more or less, and then hand it to the next node for processing.

override fun proceed(request: Request): Response {
  return proceed(request, transmitter, exchange)
}

@Throws(IOException::class)
fun proceed(request: Request, transmitter: Transmitter, exchange: Exchange?): Response {
  if (index >= interceptors.size) throw AssertionError()

  calls++

  //.....

  // Call the next interceptor in the chain.
  val next = RealInterceptorChain(interceptors, transmitter, exchange,
      index + 1, request, call, connectTimeout, readTimeout, writeTimeout)
  val interceptor = interceptors[index]

  @Suppress("USELESS_ELVIS")
  val response = interceptor.intercept(next) ?: throw NullPointerException(
      "interceptor $interceptor returned null")

 //......
  return response
}

In process, get the next interceptor and call its intercept method. In intercept, call process again to get the next interceptor.... ; a loop will be formed here until the last node returns the response, and the final data will return to the place where the call begins.

It should be noted that the current intercept is not completed when the next interceptor is called.

Let's take a look at some built-in interceptors

 // Build a full stack of interceptors.
    val interceptors = mutableListOf<Interceptor>()
    interceptors += client.interceptors
    interceptors += RetryAndFollowUpInterceptor(client)
    interceptors += BridgeInterceptor(client.cookieJar)
    interceptors += CacheInterceptor(client.cache)
    interceptors += ConnectInterceptor
    if (!forWebSocket) {
      interceptors += client.networkInterceptors
    }
    interceptors += CallServerInterceptor(forWebSocket)

  • RetryAndFollowUpInterceptor

    Try again and follow the redirection if necessary. It may throw an IOException

  • BridgeInterceptor(client.cookieJar)

    Connect application code and network code. First, it builds network requests based on user requests. Then it continues to call the network. Finally, it builds a user response from the network response.

    override fun intercept(chain: Interceptor.Chain): Response {
      val userRequest = chain.request()
      val requestBuilder = userRequest.newBuilder()
    
      val body = userRequest.body
      if (body != null) {
        val contentType = body.contentType()
        if (contentType != null) {
          requestBuilder.header("Content-Type", contentType.toString())
        }
    
        val contentLength = body.contentLength()
        if (contentLength != -1L) {
          requestBuilder.header("Content-Length", contentLength.toString())
          requestBuilder.removeHeader("Transfer-Encoding")
        } else {
          requestBuilder.header("Transfer-Encoding", "chunked")
          requestBuilder.removeHeader("Content-Length")
        }
      }
    
      if (userRequest.header("Host") == null) {
        requestBuilder.header("Host", userRequest.url.toHostHeader())
      }
    
      if (userRequest.header("Connection") == null) {
        requestBuilder.header("Connection", "Keep-Alive")
      }
    
    
      var transparentGzip = false
      if (userRequest.header("Accept-Encoding") == null && userRequest.header("Range") == null) {
        transparentGzip = true
        //Support gzip data type
        requestBuilder.header("Accept-Encoding", "gzip")
      }
    
      val cookies = cookieJar.loadForRequest(userRequest.url)
      if (cookies.isNotEmpty()) {
        requestBuilder.header("Cookie", cookieHeader(cookies))
      }
    
      if (userRequest.header("User-Agent") == null) {
        requestBuilder.header("User-Agent", userAgent)
      }
    	
      // Before proceeding, before requesting  
      val networkResponse = chain.proceed(requestBuilder.build())
      // After proced ure, after request	
      cookieJar.receiveHeaders(userRequest.url, networkResponse.headers)
    
      val responseBuilder = networkResponse.newBuilder()
          .request(userRequest)
      //Understanding gzip	
      if (transparentGzip &&
          "gzip".equals(networkResponse.header("Content-Encoding"), ignoreCase = true) &&
          networkResponse.promisesBody()) {
        val responseBody = networkResponse.body
        if (responseBody != null) {
          val gzipSource = GzipSource(responseBody.source())
          val strippedHeaders = networkResponse.headers.newBuilder()
              .removeAll("Content-Encoding")
              .removeAll("Content-Length")
              .build()
          responseBuilder.headers(strippedHeaders)
          val contentType = networkResponse.header("Content-Type")
          responseBuilder.body(RealResponseBody(contentType, -1L, gzipSource.buffer()))
        }
      }
    
      return responseBuilder.build()
    }
    

    Before proceeding, set general settings. These settings are in the header, such as contentType, length, host and so on.

    After the process, if the server sends gzip data, decompress it, add the data to the body, and return the response

  • CacheInterceptor

    Do some cache processing, such as judge whether there is a cache before the request, write the cache after the request is successful, etc

  • ConnectInterceptor

    Handle http, https, tcp connection problems, interact with the network, and return response

    override fun intercept(chain: Interceptor.Chain): Response {
      val realChain = chain as RealInterceptorChain
      val request = realChain.request()
      val transmitter = realChain.transmitter()
    
      // We need the network to satisfy this request. Possibly for validating a conditional GET.
      val doExtensiveHealthChecks = request.method != "GET"
      val exchange = transmitter.newExchange(chain, doExtensiveHealthChecks)
    
      return realChain.proceed(request, transmitter, exchange)
    }
    
  • networkInterceptors

    NetWorkIntercept is put at the back, and this interceptor is used for network related operations.

If you have any questions, please also point out

Tags: network encoding OkHttp socket

Posted on Fri, 26 Jun 2020 03:10:57 -0400 by saish