1, Redis installation and configuration
1.1 download address
https://redis.io/download.html
1.2 viewing redis version information
redis-server -v
redis-cli + info
1.3 modify the redis configuration file redis.conf
# Background operation daemonize yes protected-mode no #bind 127.0.0.1
2, Development and evolution of Redis
2.1 why is redis single threaded?
Redis has many versions of 3.X, 4.X and 6.X. different versions have different architectures. There is no limit to the version. Ask whether the single thread is not rigorous.
- Version 3.x, the earliest version, redis is single threaded
- Version 4.x, strictly speaking, is not a single thread, but the thread responsible for processing client requests is a single thread, but it began to add a bit of multithreading (asynchronous deletion)
- After the latest version of 6.0.x, we bid farewell to the impression of single thread and use a new multi thread to solve the problem
2.2 Redis is a single thread
It mainly means that the reading and writing of redis's network IO and key value pairs are completed by one thread. Redis processes the client's requests, including acquisition (socket reading), parsing, execution, content return (socket writing), etc., by a sequential serial main thread, which is the so-called "single thread". This is also the main process for redis to provide external key value storage services.
However, other redis functions, such as persistence, asynchronous deletion, cluster data synchronization, etc., are actually executed by additional threads. Redis worker threads are single threaded, but for the whole redis, they are multi-threaded.
2.3 file event handler
Redis has developed its own network event handler based on Reactor mode: this handler is called file event handler:
- The file event processor uses the I/O multiplexing program to listen to multiple sockets at the same time, and associates different event processors for the sockets according to the tasks currently performed by the sockets.
- When the monitored socket is ready to perform operations such as accept, read, write and close, file events corresponding to the operation will be generated. At this time, the file event processor will call the event processor associated with the socket to handle these events.
Although the file event processor runs in a single thread mode, by using the I/O multiplexing program to listen to multiple sockets, the file event processor not only realizes the high-performance network communication model, but also can be well connected with other modules in Redis server that also operate in a single thread mode, which maintains the simplicity of the single thread design in Redis.
Composition of file event processor
Figure IMAGE_CONSTRUCT_OF_FILE_EVENT_HANDLER shows the four components of the file event handler: socket, I/O multiplexer, file event dispatcher, and event handler.
2.4 some problems
1. The main reason for redis3. X's fast performance in the single thread Era
- Memory based operation: all data of Redis exists in memory, so all operations are memory level, so its performance is relatively high;
- Simple data structure: Redis's data structure is specially designed, and most of the time complexity of finding and operating these simple data structures is O(1), so the performance is relatively high;
- Multiplexing and non blocking I/O: Redis uses the I/O multiplexing function to listen to multiple socket connection clients, so that one thread connection can be used to process multiple requests, reducing the overhead caused by thread switching and avoiding I/O blocking operations
- Avoid context switching: because it is a single thread model, unnecessary context switching and multi-threaded competition are avoided, which saves the time and performance consumption caused by multi-threaded switching.
2. The father of redis replied to the single thread before Redis4.0
- Using the single thread model makes Redis easier to develop and maintain, because the single thread model is convenient for development and testing
- Even if the single thread model is used, the requests of multiple clients can be processed concurrently, mainly using multiplexing and non blocking IO
- For Redis system, the main performance bottleneck is memory or network bandwidth rather than CPU
3. Why did Redis4.X introduce multithreading content and functions
A: under normal circumstances, the del instruction can quickly delete data. When the deleted key is a very large object, such as a hash set containing thousands of elements, the del instruction will cause the main process of Redis to get stuck.
This is the most classic fault in the redis3.x single thread era, and the headache of large key deletion
Solution
A multi-threaded module is added in Redis4.0. Of course, multi threading in this version is mainly to solve the problem of low efficiency in deleting data.
Asynchronous delete command
unlink key flushdb async flushall async
The deletion work is handed over to the children (child threads) in the background to delete data asynchronously.
3, Introduction to multithreading and IO multiplexing of Redis6
For Redis system, the main performance bottleneck is memory or network bandwidth rather than CPU
**Memory: * * it can be solved with money
Finally, the bottleneck of Redis can be preliminarily determined as network IO
3.1 5 IO models in unix network programming
Blocking IO - blocking IO
NoneBlocking IO - non blocking IO
IO multiplexing - IO multiplexing
At this time, one of the IO models is the classic Reactor design pattern
I/O multiplexing, in short, is a mechanism to ensure that non blocking I/O of Redis can be successfully completed by monitoring file read-write events and then notifying threads to perform relevant operations.
Multichannel refers to multiple socket connections
Reuse refers to reusing a thread. There are three main multiplexing technologies: select, poll and epoll.
epoll is the latest and best multiplexing technology. Using multiplex I/O multiplexing technology can enable a single thread to efficiently process multiple connection requests (minimize the time consumption of network IO), and Redis operates data in memory very fast (the operation in memory will not become the performance bottleneck here). The above two points mainly contribute to Redis's high throughput.
I/O reading and writing are blocked. For example, when there is data in the socket, Redis will copy the data from the kernel space to the user space through calling, and then give it to Redis for calling. The copying process is blocked. When the amount of data is larger, the more time is required for copying, and these operations are completed based on a single thread.
In Redis6.0, the multithreading function is added to improve the I/O reading and writing performance. Its main implementation idea is to split the IO reading and writing tasks of the main thread into a group of independent threads, so that the reading and writing of multiple sockets can be parallelized. The multi-channel I/O multiplexing technology can enable a single thread to efficiently process multiple connection requests (minimize the time consumption of network IO), The most time-consuming socket reading, request parsing and writing are outsourced separately, and the remaining command execution is still executed serially by the main thread and interacts with the data in memory
It can be seen from the above figure that network IO operation becomes multithreaded, and other core parts are still thread safe, which is a good method.
Signal driven IO - signal driven IO
Asynchronous IO - asynchronous IO
3.2 multithreading in redis6
Redis places all data in memory, and the response time of memory is about 100 nanoseconds. For small data packets, redis server can process 2W to 10W QPS, which is also the limit of redis processing. For 80% of companies, single thread redis is enough.
Multithreading in Redis6 is turned off by default, which can be seen from redis.conf
If you need to use multithreading, you need to complete two settings in redis.conf
# Turn on Multithreading io-threads-do-reads yes # Set the number of threads io-threads 4
As for the setting of the number of threads, the official suggestion is that if the CPU has 4 cores, the recommended number of threads is set to 2 or 3. If the CPU has 8 and the recommended number of threads is set to 6, the number of threads must be less than the number of machine cores. The larger the number of threads, the better.
In Redis6.0, I/O multi-threaded reading and writing is introduced, so that more tasks can be handled more efficiently. Redis only programs I/O reading and writing to multi threads, and the command execution is still executed by the main thread in series. Therefore, there will be no thread safety problem when operating redis under multi threads.
4, Use TKMaper to generate CRUD
4.1 create mysql table t_user
create table t_user ( id int(10) unsigned not null auto_increment, username varchar(50) not null default '' comment 'user name', password varchar(50) not null default '' comment 'password', sex tinyint(4) not null default '0' comment 'Gender 0=Female 1=male', deleted tinyint(4) unsigned not null default '0' comment 'Delete flag: 0 is not deleted by default, and 1 is deleted', update_time timestamp not null default current_timestamp on update current_timestamp comment 'Update time', create_time timestamp not null default current_timestamp comment 'Creation time', primary key(id) ) auto_increment=1 default charset=utf8 comment='User table';
4.2 build maven project
4.3 change pom
pom.xml
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>2.3.10.RELEASE</version> </parent> <groupId>com.hmx</groupId> <artifactId>mybatis_generator</artifactId> <version>1.0-SNAPSHOT</version> <properties> <maven.compiler.source>8</maven.compiler.source> <maven.compiler.target>8</maven.compiler.target> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <java.version>1.8</java.version> <hutool.version>5.5.8</hutool.version> <druid.version>1.1.18</druid.version> <mapper.version>4.1.5</mapper.version> <pagehelper.version>5.1.4</pagehelper.version> <mysql.version>8.0.25</mysql.version> <swagger2.version>3.0.0</swagger2.version> <mybatis.spring.version>2.1.3</mybatis.spring.version> </properties> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <!--Mybatis--> <!--<dependency> <groupId>org.mybatis</groupId> <artifactId>mybatis</artifactId> <version>3.5.4</version> </dependency>--> <!--mybatis-spring--> <dependency> <groupId>org.mybatis.spring.boot</groupId> <artifactId>mybatis-spring-boot-starter</artifactId> <version>${mybatis.spring.version}</version> </dependency> <!--Mybatis Generator--> <dependency> <groupId>org.mybatis.generator</groupId> <artifactId>mybatis-generator-core</artifactId> <version>1.4.0</version> <scope>compile</scope> <optional>true</optional> </dependency> <!--currency Mapper tk Use it alone with the version number--> <dependency> <groupId>tk.mybatis</groupId> <artifactId>mapper</artifactId> <version>${mapper.version}</version> </dependency> <!--persistence--> <!--<dependency> <groupId>javax.persistence</groupId> <artifactId>persistence-api</artifactId> <version>1.0.2</version> </dependency>--> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> </dependencies> <build> <resources> <resource> <directory>${basedir}/src/main/java</directory> <includes> <include>**/*.xml</include> </includes> </resource> <resource> <directory>${basedir}/src/main/resources</directory> </resource> </resources> <plugins> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> <plugin> <groupId>org.mybatis.generator</groupId> <artifactId>mybatis-generator-maven-plugin</artifactId> <version>1.3.6</version> <configuration> <configurationFile>${basedir}/src/main/resources/generatorConfig.xml</configurationFile> <overwrite>true</overwrite> <verbose>true</verbose> </configuration> <dependencies> <dependency> <groupId>mysql</groupId> <artifactId>mysql-connector-java</artifactId> <version>${mysql.version}</version> </dependency> <dependency> <groupId>tk.mybatis</groupId> <artifactId>mapper</artifactId> <version>${mapper.version}</version> </dependency> </dependencies> </plugin> </plugins> </build> </project>
4.4 create a new configuration file in SRC \ main \ resources directory
Config.properties (configure database connection information, and some other information
# User table name package.name=com.hmx.redis jdbc.driverClass=com.mysql.cj.jdbc.Driver jdbc.url=jdbc:mysql://47.98.134.37:3307/redis?serverTimezone=Asia/Shanghai jdbc.user=root jdbc.password=root
generatorConfig.xml
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE generatorConfiguration PUBLIC "-//mybatis.org//DTD MyBatis Generator Configuration 1.0//EN" "http://mybatis.org/dtd/mybatis-generator-config_1_0.dtd"> <generatorConfiguration> <properties resource="config.properties" /> <context id="Mysql" targetRuntime="MyBatis3Simple" defaultModelType="flat"> <property name="beginningDelimiter" value=""/> <property name="beginningDelimiter" value=""/> <!-- Generated pojo,take implements Serializable--> <plugin type="tk.mybatis.mapper.generator.MapperPlugin"> <property name="mappers" value="tk.mybatis.mapper.common.Mapper"/> <property name="caseSensitive" value="true"/> </plugin> <commentGenerator> <!-- Remove automatically generated comments true: Yes: false:no --> <!--<property name="suppressAllComments" value="true" />--> </commentGenerator> <!-- Database link URL,User name and password --> <jdbcConnection driverClass="${jdbc.driverClass}" connectionURL="${jdbc.url}" userId="${jdbc.user}" password="${jdbc.password}"> </jdbcConnection> <!-- default false,hold JDBC DECIMAL and NUMERIC Type resolves to Integer true,hold JDBC DECIMAL and NUMERIC Type resolves to java.math.BigDecimal --> <javaTypeResolver> <property name="forceBigDecimals" value="false" /> </javaTypeResolver> <!-- generate model Model, corresponding package path, and file storage path(targetProject),targetProject You can specify a specific path,as./src/main/java, You can also use“ MAVEN"To automatically generate, so that the generated code will be generated in target/generatord-source Directory --> <!--<javaModelGenerator targetPackage="com.joey.mybaties.test.pojo" targetProject="MAVEN">--> <javaModelGenerator targetPackage="${package.name}.entity" targetProject="./src/main/java"> <property name="enableSubPackages" value="true"/> <!-- The space before and after the value returned from the database is cleaned up --> <property name="trimStrings" value="true" /> </javaModelGenerator> <!--Corresponding mapper.xml file --> <sqlMapGenerator targetPackage="${package.name}.mapper" targetProject="./src/main/resources"> <property name="enableSubPackages" value="true"/> </sqlMapGenerator> <!-- Corresponding Mapper Interface class file --> <javaClientGenerator type="XMLMAPPER" targetPackage="${package.name}.mapper" targetProject="./src/main/java"> <property name="enableSubPackages" value="true"/> </javaClientGenerator> <!-- List all tables to generate code. Here, no code generation is configured Example file --> <table tableName="t_user" domainObjectName="User"> <generatedKey column="id" sqlStatement="JDBC"/> </table> </context> </generatorConfiguration>
4.5 reverse engineering one key generation dao layer
Double click the mybatis generator: generate plug-in to generate entity+mapper interface + mapper.xml to implement SQL
5, SRM(SpringBoot+redis+mybatis) case
5.1 create maven project
5.2 change Pom
pom.xml
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <parent> <artifactId>redis</artifactId> <groupId>com.hmx</groupId> <version>1.0-SNAPSHOT</version> </parent> <modelVersion>4.0.0</modelVersion> <artifactId>redis_20211026</artifactId> <properties> <maven.compiler.source>8</maven.compiler.source> <maven.compiler.target>8</maven.compiler.target> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <java.version>1.8</java.version> <hutool.version>5.5.8</hutool.version> <druid.version>1.1.18</druid.version> <mapper.version>4.1.5</mapper.version> <pagehelper.version>5.1.4</pagehelper.version> <mysql.version>8.0.25</mysql.version> <swagger2.version>3.0.0</swagger2.version> <mybatis.spring.version>2.1.3</mybatis.spring.version> <junit.version>4.13.2</junit.version> </properties> <dependencies> <!--swagger--> <dependency> <groupId>io.springfox</groupId> <artifactId>springfox-boot-starter</artifactId> <version>3.0.0</version> </dependency> <!--SpringBoot And redis integration--> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-redis</artifactId> </dependency> <!--SpringCache Spring Cache support for framework--> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-cache</artifactId> </dependency> <!--SpringCache Connection pool dependent package--> <dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-pool2</artifactId> </dependency> <!--jedis--> <dependency> <groupId>redis.clients</groupId> <artifactId>jedis</artifactId> <version>3.6.1</version> </dependency> <!--springboot integrate druid Connection pool--> <dependency> <groupId>com.alibaba</groupId> <artifactId>druid-spring-boot-starter</artifactId> <version>1.1.10</version> </dependency> <!--mybatis and springboot integration--> <dependency> <groupId>org.mybatis.spring.boot</groupId> <artifactId>mybatis-spring-boot-starter</artifactId> <version>${mybatis.spring.version}</version> </dependency> <!--springboot integration RabbitMQ--> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-amqp</artifactId> </dependency> <!--General basic configuration junit/devtools/test/log4j/lombok/hutool--> <!--hutool--> <dependency> <groupId>cn.hutool</groupId> <artifactId>hutool-all</artifactId> <version>5.7.7</version> </dependency> <!--junit--> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <scope>test</scope> <version>${junit.version}</version> </dependency> <!--Hot deployment--> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-devtools</artifactId> <scope>runtime</scope> <optional>true</optional> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <!--currency Mapper tk Use it alone with the version number--> <dependency> <groupId>tk.mybatis</groupId> <artifactId>mapper</artifactId> <version>${mapper.version}</version> </dependency> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> </dependencies> <build> <resources> <resource> <directory>${basedir}/src/main/java</directory> <includes> <include>**/*.xml</include> </includes> </resource> <resource> <directory>${basedir}/src/main/resources</directory> </resource> </resources> <plugins> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> </plugins> </build> </project>
5.3 drag the dao layer generated in the previous part of reverse engineering into the new project
5.4 preparation of configuration files
application.properties
# Project startup port server.port=5555 # Project name spring.application.name=redis0511 # ======================Logging log related configuration====================== # By default, the log form of global root configuration can be commented out logging.level.root=warn # According to the package structure set by the developer, what level of log monitoring is performed on that package logging.level.com.atguigu.redis=info # Developers can customize the log path and log name logging.file.name=E:/mylog/logs/redis1026.log #%d{HH:mm:ss.SSS} - log output time #%thread - the name of the process that outputs the log, which is useful in web applications and asynchronous task processing #%-5level - log level, with 5 strings aligned to the left #%Logger - the name of the log exporter #%msg - log messages #%n - newline for platform #Logging.pattern.console=%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %Logger- %msg%n logging.pattern.console=%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %Logger- %msg%n logging.pattern.file=%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %Logger- %msg%n # =====================druid related configuration======================= spring.datasource.type=com.alibaba.druid.pool.DruidDataSource spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver spring.datasource.url=jdbc:mysql://47.98.134.37:3306/redis?serverTimezone=Asia/Shanghai spring.datasource.username=root spring.datasource.password=root # It is recommended to configure it to true, which will not affect performance and ensure security. When applying for a connection, check whether the connection is valid by running validationQuery if the idle time is greater than timebetween evictionrunsmillis. spring.datasource.druid.test-while-idle=false # ====================redis related configuration====================== # Redis database index (0 by default) spring.redis.database=0 # Redis server address spring.redis.host=yourip # Redis server connection port spring.redis.port=6379 # Redis server connection password (blank by default) spring.redis.password= # The maximum number of connections in the connection pool (a negative value indicates no limit). The default is - 1. Remember to add the unit ms, otherwise the idea will be red spring.redis.lettuce.pool.max-active=9 # The maximum blocking waiting time of the connection pool (a negative value indicates no limit) is - 1 by default. Remember to add the unit ms, otherwise the idea will report red spring.redis.lettuce.pool.max-wait=-1ms # The maximum free connections in the connection pool are 8 by default spring.redis.lettuce.pool.max-idle=8 # The minimum free connections in the connection pool are 0 by default spring.redis.lettuce.pool.min-idle=0 # =========mybatis related configuration=============== mybatis.mapper-locations=classpath:com/hmx/redis/mapper/*.xml mybatis.type-aliases-package=com.hmx.redis.entity # ==========swagger================ spring.swagger2.enabled=true # ================rabbitmq related configuration=============== #spring.rabbitmq.host=127.0.0.1 #spring.rabbitmq.port=5672 #spring.rabbitmq.username=guest #spring.rabbitmq.password=guest #spring.rabbitmq.virtual-host=/ # ================redis bloom filter configuration==================== #redis.bloom.url=192.168.111.147 #redis.bloom.post=6379 #redis.bloom.init-capacity=10000 #redis.bloom.error-rate=0.01
5.5 main startup
MainApplication.java
@SpringBootApplication //Not the MapperScan integrated by spring and mybatis @MapperScan("com.hmx.redis.mapper") //import tk.mybatis.spring.annotation.MapperScan public class MainApplication { public static void main(String[] args) { SpringApplication.run(MainApplication.class); } }