Ultimate performance of distributed storage Redis

1, Redis installation and configuration

1.1 download address

1.2 viewing redis version information

redis-server -v

redis-cli + info

1.3 modify the redis configuration file redis.conf

# Background operation
daemonize yes

protected-mode no

#bind 127.0.0.1

2, Development and evolution of Redis

2.1 why is redis single threaded?

Redis has many versions of 3.X, 4.X and 6.X. different versions have different architectures. There is no limit to the version. Ask whether the single thread is not rigorous.

Version 3.x, the earliest version, redis is single threaded
Version 4.x, strictly speaking, is not a single thread, but the thread responsible for processing client requests is a single thread, but it began to add a bit of multithreading (asynchronous deletion)
After the latest version of 6.0.x, we bid farewell to the impression of single thread and use a new multi thread to solve the problem

2.2 Redis is a single thread

It mainly means that the reading and writing of redis's network IO and key value pairs are completed by one thread. Redis processes the client's requests, including acquisition (socket reading), parsing, execution, content return (socket writing), etc., by a sequential serial main thread, which is the so-called "single thread". This is also the main process for redis to provide external key value storage services.

However, other redis functions, such as persistence, asynchronous deletion, cluster data synchronization, etc., are actually executed by additional threads. Redis worker threads are single threaded, but for the whole redis, they are multi-threaded.

2.3 file event handler

Redis has developed its own network event handler based on Reactor mode: this handler is called file event handler:

The file event processor uses the I/O multiplexing program to listen to multiple sockets at the same time, and associates different event processors for the sockets according to the tasks currently performed by the sockets.
When the monitored socket is ready to perform operations such as accept, read, write and close, file events corresponding to the operation will be generated. At this time, the file event processor will call the event processor associated with the socket to handle these events.

Although the file event processor runs in a single thread mode, by using the I/O multiplexing program to listen to multiple sockets, the file event processor not only realizes the high-performance network communication model, but also can be well connected with other modules in Redis server that also operate in a single thread mode, which maintains the simplicity of the single thread design in Redis.

Composition of file event processor

Figure IMAGE_CONSTRUCT_OF_FILE_EVENT_HANDLER shows the four components of the file event handler: socket, I/O multiplexer, file event dispatcher, and event handler.

2.4 some problems

1. The main reason for redis3. X's fast performance in the single thread Era

Memory based operation: all data of Redis exists in memory, so all operations are memory level, so its performance is relatively high;
Simple data structure: Redis's data structure is specially designed, and most of the time complexity of finding and operating these simple data structures is O(1), so the performance is relatively high;
Multiplexing and non blocking I/O: Redis uses the I/O multiplexing function to listen to multiple socket connection clients, so that one thread connection can be used to process multiple requests, reducing the overhead caused by thread switching and avoiding I/O blocking operations
Avoid context switching: because it is a single thread model, unnecessary context switching and multi-threaded competition are avoided, which saves the time and performance consumption caused by multi-threaded switching.

2. The father of redis replied to the single thread before Redis4.0

Using the single thread model makes Redis easier to develop and maintain, because the single thread model is convenient for development and testing
Even if the single thread model is used, the requests of multiple clients can be processed concurrently, mainly using multiplexing and non blocking IO
For Redis system, the main performance bottleneck is memory or network bandwidth rather than CPU

3. Why did Redis4.X introduce multithreading content and functions

A: under normal circumstances, the del instruction can quickly delete data. When the deleted key is a very large object, such as a hash set containing thousands of elements, the del instruction will cause the main process of Redis to get stuck.
This is the most classic fault in the redis3.x single thread era, and the headache of large key deletion

Solution

A multi-threaded module is added in Redis4.0. Of course, multi threading in this version is mainly to solve the problem of low efficiency in deleting data.
Asynchronous delete command

unlink key
flushdb async
flushall async

The deletion work is handed over to the children (child threads) in the background to delete data asynchronously.

3, Introduction to multithreading and IO multiplexing of Redis6

For Redis system, the main performance bottleneck is memory or network bandwidth rather than CPU
**Memory: * * it can be solved with money
Finally, the bottleneck of Redis can be preliminarily determined as network IO

3.1 5 IO models in unix network programming

Blocking IO - blocking IO

NoneBlocking IO - non blocking IO

IO multiplexing - IO multiplexing

At this time, one of the IO models is the classic Reactor design pattern
I/O multiplexing, in short, is a mechanism to ensure that non blocking I/O of Redis can be successfully completed by monitoring file read-write events and then notifying threads to perform relevant operations.
Multichannel refers to multiple socket connections
Reuse refers to reusing a thread. There are three main multiplexing technologies: select, poll and epoll.

epoll is the latest and best multiplexing technology. Using multiplex I/O multiplexing technology can enable a single thread to efficiently process multiple connection requests (minimize the time consumption of network IO), and Redis operates data in memory very fast (the operation in memory will not become the performance bottleneck here). The above two points mainly contribute to Redis's high throughput.

I/O reading and writing are blocked. For example, when there is data in the socket, Redis will copy the data from the kernel space to the user space through calling, and then give it to Redis for calling. The copying process is blocked. When the amount of data is larger, the more time is required for copying, and these operations are completed based on a single thread.

In Redis6.0, the multithreading function is added to improve the I/O reading and writing performance. Its main implementation idea is to split the IO reading and writing tasks of the main thread into a group of independent threads, so that the reading and writing of multiple sockets can be parallelized. The multi-channel I/O multiplexing technology can enable a single thread to efficiently process multiple connection requests (minimize the time consumption of network IO), The most time-consuming socket reading, request parsing and writing are outsourced separately, and the remaining command execution is still executed serially by the main thread and interacts with the data in memory

It can be seen from the above figure that network IO operation becomes multithreaded, and other core parts are still thread safe, which is a good method.

Signal driven IO - signal driven IO

Asynchronous IO - asynchronous IO

3.2 multithreading in redis6

Redis places all data in memory, and the response time of memory is about 100 nanoseconds. For small data packets, redis server can process 2W to 10W QPS, which is also the limit of redis processing. For 80% of companies, single thread redis is enough.
Multithreading in Redis6 is turned off by default, which can be seen from redis.conf

If you need to use multithreading, you need to complete two settings in redis.conf

# Turn on Multithreading
io-threads-do-reads yes
# Set the number of threads
io-threads 4

As for the setting of the number of threads, the official suggestion is that if the CPU has 4 cores, the recommended number of threads is set to 2 or 3. If the CPU has 8 and the recommended number of threads is set to 6, the number of threads must be less than the number of machine cores. The larger the number of threads, the better.

In Redis6.0, I/O multi-threaded reading and writing is introduced, so that more tasks can be handled more efficiently. Redis only programs I/O reading and writing to multi threads, and the command execution is still executed by the main thread in series. Therefore, there will be no thread safety problem when operating redis under multi threads.

4, Use TKMaper to generate CRUD

4.1 create mysql table t_user

create table t_user (
    id int(10) unsigned not null auto_increment,
    username varchar(50) not null default '' comment 'user name',
    password varchar(50) not null default '' comment 'password',
    sex tinyint(4) not null default '0' comment 'Gender 0=Female 1=male',
    deleted tinyint(4) unsigned not null default '0' comment 'Delete flag: 0 is not deleted by default, and 1 is deleted',
    update_time timestamp not null default current_timestamp on update current_timestamp comment 'Update time',
    create_time timestamp not null default current_timestamp comment 'Creation time',
    primary key(id)
) auto_increment=1 default charset=utf8 comment='User table';

4.2 build maven project

4.3 change pom

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.3.10.RELEASE</version>
    </parent>

    <groupId>com.hmx</groupId>
    <artifactId>mybatis_generator</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>8</maven.compiler.source>
        <maven.compiler.target>8</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <java.version>1.8</java.version>
        <hutool.version>5.5.8</hutool.version>
        <druid.version>1.1.18</druid.version>
        <mapper.version>4.1.5</mapper.version>
        <pagehelper.version>5.1.4</pagehelper.version>
        <mysql.version>8.0.25</mysql.version>
        <swagger2.version>3.0.0</swagger2.version>
        <mybatis.spring.version>2.1.3</mybatis.spring.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <!--Mybatis-->
        <!--<dependency>
            <groupId>org.mybatis</groupId>
            <artifactId>mybatis</artifactId>
            <version>3.5.4</version>
        </dependency>-->
        <!--mybatis-spring-->
        <dependency>
            <groupId>org.mybatis.spring.boot</groupId>
            <artifactId>mybatis-spring-boot-starter</artifactId>
            <version>${mybatis.spring.version}</version>
        </dependency>
        <!--Mybatis Generator-->
        <dependency>
            <groupId>org.mybatis.generator</groupId>
            <artifactId>mybatis-generator-core</artifactId>
            <version>1.4.0</version>
            <scope>compile</scope>
            <optional>true</optional>
        </dependency>
        <!--currency Mapper tk Use it alone with the version number-->
        <dependency>
            <groupId>tk.mybatis</groupId>
            <artifactId>mapper</artifactId>
            <version>${mapper.version}</version>
        </dependency>
        <!--persistence-->
        <!--<dependency>
            <groupId>javax.persistence</groupId>
            <artifactId>persistence-api</artifactId>
            <version>1.0.2</version>
        </dependency>-->
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>

        </dependency>
    </dependencies>

    <build>
        <resources>
            <resource>
                <directory>${basedir}/src/main/java</directory>
                <includes>
                    <include>**/*.xml</include>
                </includes>
            </resource>
            <resource>
                <directory>${basedir}/src/main/resources</directory>
            </resource>
        </resources>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
            <plugin>
                <groupId>org.mybatis.generator</groupId>
                <artifactId>mybatis-generator-maven-plugin</artifactId>
                <version>1.3.6</version>
                <configuration>
                    <configurationFile>${basedir}/src/main/resources/generatorConfig.xml</configurationFile>
                    <overwrite>true</overwrite>
                    <verbose>true</verbose>
                </configuration>
                <dependencies>
                    <dependency>
                        <groupId>mysql</groupId>
                        <artifactId>mysql-connector-java</artifactId>
                        <version>${mysql.version}</version>
                    </dependency>
                    <dependency>
                        <groupId>tk.mybatis</groupId>
                        <artifactId>mapper</artifactId>
                        <version>${mapper.version}</version>
                    </dependency>
                </dependencies>
            </plugin>
        </plugins>
    </build>

</project>

4.4 create a new configuration file in SRC \ main \ resources directory

Config.properties (configure database connection information, and some other information

# User table name
package.name=com.hmx.redis

jdbc.driverClass=com.mysql.cj.jdbc.Driver
jdbc.url=jdbc:mysql://47.98.134.37:3307/redis?serverTimezone=Asia/Shanghai
jdbc.user=root
jdbc.password=root

generatorConfig.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE generatorConfiguration
        PUBLIC "-//mybatis.org//DTD MyBatis Generator Configuration 1.0//EN"
        "http://mybatis.org/dtd/mybatis-generator-config_1_0.dtd">

<generatorConfiguration>
    <properties resource="config.properties" />
    <context id="Mysql" targetRuntime="MyBatis3Simple" defaultModelType="flat">
        <property name="beginningDelimiter" value=""/>
        <property name="beginningDelimiter" value=""/>

        <!-- Generated pojo，take implements Serializable-->
        <plugin type="tk.mybatis.mapper.generator.MapperPlugin">
            <property name="mappers" value="tk.mybatis.mapper.common.Mapper"/>
            <property name="caseSensitive" value="true"/>
        </plugin>
        <commentGenerator>
            <!-- Remove automatically generated comments true: Yes: false:no -->
            <!--<property name="suppressAllComments" value="true" />-->
        </commentGenerator>

        <!-- Database link URL,User name and password -->
        <jdbcConnection driverClass="${jdbc.driverClass}"
                        connectionURL="${jdbc.url}"
                        userId="${jdbc.user}"
                        password="${jdbc.password}">
        </jdbcConnection>

        <!--
        default false，hold JDBC DECIMAL and NUMERIC Type resolves to Integer
            true，hold JDBC DECIMAL and NUMERIC Type resolves to java.math.BigDecimal
        -->
        <javaTypeResolver>
            <property name="forceBigDecimals" value="false" />
        </javaTypeResolver>

        <!--
        generate model Model, corresponding package path, and file storage path(targetProject)，targetProject You can specify a specific path,as./src/main/java，
        You can also use“ MAVEN"To automatically generate, so that the generated code will be generated in target/generatord-source Directory
        -->
        <!--<javaModelGenerator targetPackage="com.joey.mybaties.test.pojo" targetProject="MAVEN">-->
        <javaModelGenerator targetPackage="${package.name}.entity" targetProject="./src/main/java">
            <property name="enableSubPackages" value="true"/>
            <!-- The space before and after the value returned from the database is cleaned up  -->
            <property name="trimStrings" value="true" />
        </javaModelGenerator>

        <!--Corresponding mapper.xml file  -->
        <sqlMapGenerator targetPackage="${package.name}.mapper" targetProject="./src/main/resources">
            <property name="enableSubPackages" value="true"/>
        </sqlMapGenerator>

        <!-- Corresponding Mapper Interface class file -->
        <javaClientGenerator type="XMLMAPPER" targetPackage="${package.name}.mapper" targetProject="./src/main/java">
            <property name="enableSubPackages" value="true"/>
        </javaClientGenerator>


        <!-- List all tables to generate code. Here, no code generation is configured Example file -->
        <table tableName="t_user" domainObjectName="User">
            <generatedKey column="id" sqlStatement="JDBC"/>
        </table>
    </context>
</generatorConfiguration>

4.5 reverse engineering one key generation dao layer

Double click the mybatis generator: generate plug-in to generate entity+mapper interface + mapper.xml to implement SQL

5, SRM(SpringBoot+redis+mybatis) case

5.1 create maven project

5.2 change Pom