Analysis of thread isolation

preface

With the popularity of microservices, single applications are divided into independent microprocesses. A simple request may require multiple microservices to process together, which in fact increases the probability of error. Therefore, how to ensure that when a single microservice fails, the negative impact on the whole system is minimized? This requires the thread isolation we will introduce today .

Thread model

Before introducing thread isolation, let's first understand the mainstream container, the thread model of the framework, because microservices are independent processes, and the calls between them are actually network io. The processing containers of network IO, such as tomcat, the communication framework, such as netty, and the microservices framework, such as dubbo, are all very good to help us deal with the underlying network IO flow, so that we can pay more attention to the industry Business handling;

Netty

Netty is a high-performance communication framework based on java nio, which uses the master-slave multithreading model Netty thread model of netty series A picture of is as follows:

The main thread is responsible for authentication and connection, and the slave thread is responsible for reading and writing the connection after success. The code is roughly as follows:

EventLoopGroup bossGroup = new NioEventLoopGroup(1);
EventLoopGroup workerGroup = new NioEventLoopGroup();

ServerBootstrap b = new ServerBootstrap();
b.group(bossGroup, workerGroup);

The main thread is a single thread, and the slave thread is a thread pool with a default number of CPUs * 2. You can do a simple test in our business handler:

    public void channelRead(ChannelHandlerContext ctx, Object msg) {
        System.out.println("thread name=" + Thread.currentThread().getName() + " server receive msg=" + msg);
    }

The server prints the current thread when reading data:

thread name=nioEventLoopGroup-3-1 server receive msg="..."

It can be found that the thread used here is actually the same as the io thread;

Dubbo

In fact, the underlying communication framework of Dubbo uses Netty, but Dubbo does not directly use the io thread of Netty to process business. It can simply output the current thread name on the producer side:

thread name=DubboServerHandler-192.168.1.115:20880-thread-2,...

It can be found that the use of business logic is not nioEventLoopGroup thread. This is because Dubbo has its own thread model. You can see the model diagram provided on the official website:

The Dispatcher scheduler can configure the message processing thread:

  • All all messages are sent to the thread pool, including request, response, connection event, disconnection event, heartbeat, etc.
  • direct: all messages are not distributed to the thread pool, and all messages are directly executed on IO threads.
  • message: only request response messages are sent to the thread pool. Other connection disconnection events, heartbeat and other messages are directly executed on the IO thread.
  • execution only requests messages are sent to the thread pool, excluding response, response and other disconnection events, heartbeat and other messages, which are directly executed on the IO thread.
  • connection on the IO thread, put the disconnection events into the queue, execute one by one orderly, and send other messages to the thread pool.

Dubbo uses FixedThreadPool by default, and the number of threads is 200 by default;

Tomcat

Tomcat can configure four kinds of thread models: BIO, NIO, APR, AIO; Tomcat8 starts to configure NIO by default, which is similar to Netty's thread model, and can be understood as the Reactor mode, but it's just introduced here; among them, the maxThreads parameter is used to configure the number of workers specialized in IO, which is 200 by default; the current thread name can be output in the business Controller:

ThreadName=http-nio-8888-exec-1...

It can be found that the thread processing business is Tomcat's io thread;

Why thread isolation

From the thread model introduced above, we can know that the io threads used in business processing, such as Tomcat and netty, will cause some problems. For example, the current service process needs to synchronously call three other microservices, but due to the problem of one service, the thread is blocked, and then the more blocked, the more occupied all io threads, the final service cannot Accept the data until it breaks;
Dubbo itself isolates the IO thread from the business thread. If there are problems, the IO thread will not be affected. However, if there are the same problems, the business thread will be full;
The purpose of thread isolation is to control a service in a small range if there is a problem, so as not to affect the overall situation;

How to do thread isolation

The principle of thread isolation is also very simple. Each request is allocated with a separate thread pool. Each request does not affect each other. Of course, some mature frameworks such as hystrix (not updated) and Sentinel can also be used;

Thread pool isolation

Spring boot + Tomcat do a simple isolation test. In order to facilitate the simulation configuration of MaxThreads=5, isolation Controller is provided as follows:

@RequestMapping("/h1")
String home() throws Exception {
    System.out.println("h1-->ThreadName=" + Thread.currentThread().getName());
    Thread.sleep(200000);
    return "h1";
}
    
@RequestMapping("/h3")
String home3() {
    System.out.println("h3-->ThreadName=" + Thread.currentThread().getName());
    return "h3";
}

Request 5 times / h1, request again / h3, observe the log:

h1-->ThreadName=http-nio-8888-exec-1
h1-->ThreadName=http-nio-8888-exec-2
h1-->ThreadName=http-nio-8888-exec-3
h1-->ThreadName=http-nio-8888-exec-4
h1-->ThreadName=http-nio-8888-exec-5

It can be found that the h1 request is full of 5 threads, and Tomcat cannot accept the request when h3 is requested. To modify the h1 request, use the thread pool to process:

ExecutorService executorService = Executors.newFixedThreadPool(2);
List<Future<String>> list = new CopyOnWriteArrayList<Future<String>>();
@RequestMapping("/h2")
String home2() throws Exception {
    Future<String> result = executorService.submit(new Callable<String>() {
        @Override
        public String call() throws Exception {
            System.out.println("h2-->ThreadName=" + Thread.currentThread().getName());
            Thread.sleep(200000);
            return "h2";
        }
    });
    list.add(result);
    //handle at a lower grade
    if (list.size() >= 3) {
        return "h2-fallback";
    }
    String resultStr = result.get();
    list.remove(result);
    return resultStr;
}

For the above pseudo code, use thread pool to execute asynchronously, and do degradation processing beyond the limit, so that when h3 is requested again, it will not be affected; of course, the above code is relatively simple, we can use a mature isolation framework;

Hystrix

Hystrix provides two isolation strategies: bulk head pattern and semaphore isolation. The most recommended and commonly used one is thread pool isolation. The thread pool isolation of hystrix creates different thread pools for different resources, and different service calls occur in different thread pools. When the online process pool is queued, timed out and other blocking situations, it can quickly fail, and it can provide a fallback mechanism. You can see a simple example:

public class HelloCommand extends HystrixCommand<String> {

    public HelloCommand(String name) {
        super(Setter.withGroupKey(HystrixCommandGroupKey.Factory.asKey("ThreadPoolTestGroup"))
                .andCommandKey(HystrixCommandKey.Factory.asKey("testCommandKey"))
                .andThreadPoolKey(HystrixThreadPoolKey.Factory.asKey(name))
                .andCommandPropertiesDefaults(
                        HystrixCommandProperties.Setter().withExecutionTimeoutInMilliseconds(20000))
                .andThreadPoolPropertiesDefaults(HystrixThreadPoolProperties.Setter().withMaxQueueSize(5) // Configure queue size
                        .withCoreSize(2) // Configure the number of threads in the thread pool
        ));
    }

    @Override
    protected String run() throws InterruptedException {
        StringBuffer sb = new StringBuffer("Thread name=" + Thread.currentThread().getName() + ",");
        Thread.sleep(2000);
        return sb.append(System.currentTimeMillis()).toString();
    }

    @Override
    protected String getFallback() {
        return "Thread name=" + Thread.currentThread().getName() + ",fallback order";
    }

    public static void main(String[] args) throws InterruptedException, ExecutionException {
        List<Future<String>> list = new ArrayList<>();
        System.out.println("Thread name=" + Thread.currentThread().getName());
        for (int i = 0; i < 8; i++) {
            Future<String> future = new HelloCommand("hystrix-order").queue();
            list.add(future);
        }
        for (Future<String> future : list) {
            System.out.println(future.get());
        }
        Thread.sleep(1000000);
    }
}

As configured above, the number of threads processing this business is 2, and the maximum number of threads that can be put into the queue when they are full is specified. The result of running this program is as follows:

Thread name=main
Thread name=hystrix-hystrix-order-1,1589776137342
Thread name=hystrix-hystrix-order-2,1589776137342
Thread name=hystrix-hystrix-order-1,1589776139343
Thread name=hystrix-hystrix-order-2,1589776139343
Thread name=hystrix-hystrix-order-1,1589776141343
Thread name=hystrix-hystrix-order-2,1589776141343
Thread name=hystrix-hystrix-order-2,1589776143343
Thread name=main,fallback order

The main thread execution can be understood as the io thread. The business execution uses the hystrix thread. The number of threads 2 + queue 5 can handle 7 concurrent requests at the same time, and the excess part can be directly fallback;

Semaphore isolation

The advantage of thread pool isolation is that it has a high degree of isolation. It can process a resource's thread pool without affecting other resources, but the cost is that the cost of thread context switching is large, especially for low latency calls;
In the above introduction to the thread model, we found that Tomcat provides 200 io threads by default, Dubbo provides 200 business threads by default, and the number of threads is already very large. If each command uses a thread pool, the number of threads will be very large, which has a great impact on the system. A lighter isolation method is semaphore isolation, which only limits the combination of calling a resource The cost of creating thread pool is relatively small because of the number of sends rather than the explicit way. Both hystrix and Sentinel provide semaphore isolation. Hystrix has stopped updating, while Sentinel simply does not provide thread isolation, or thread isolation is unnecessary, and can be replaced by lighter semaphore isolation;

summary

Starting from the thread model, this paper talks about IO threads, why IO threads and business threads should be separated, and how to realize them. Finally, it briefly introduces the lighter signal isolation, why is it lighter? In fact, the business is still in io thread processing, but it will limit the concurrent number of a resource, and there is no redundant thread; of course, it is not line Program isolation has no value. In fact, it depends on the actual situation and the thread model of the container and framework you use

Tags: Programming Netty Tomcat Dubbo network

Posted on Mon, 18 May 2020 02:17:58 -0400 by psytae