Distributed processing scheme

Distributed processing scheme


1, Introduction to distributed locks

  • Because the traditional lock is based on the internal of Tomcat server, the lock fails after the cluster is built
  • Use distributed locks to handle

2, Distributed lock solution

2.1 construction environment

1. Create SpringBoot project
2. Prepare rush purchase business

@RestController
public class SecondKillController {
    //1. Prepare inventory of goods
    public static Map<String,Integer> itemStock=new HashMap<>();


    //2. Prepare orders for goods
    public static Map<String,Integer> itemOrder=new HashMap<>();

    static {
        itemStock.put("toothbrush",10000);
        itemOrder.put("toothbrush",0);
    }

    @GetMapping("/kill")
    public String kill(String item) throws InterruptedException {
        //1. Inventory reduction
        Integer stock=itemStock.get(item);
        if (stock<=0){
            return "Insufficient inventory!!!";
        }
        Thread.sleep(100);
        itemStock.put("toothbrush",stock-1);
        //2. Create order
        Thread.sleep(100);
        itemOrder.put(item,itemOrder.get(item)+1);

        //3. Return information
        return "Successful rush purchase!!!"+item+": The remaining inventory is - "+itemStock.get(item)+",The number of orders is:"+itemOrder.get(item);
    }

}

3. Download ab stress test
Download address: https://www.apachehaus.com/cgi-bin/download.plx?dli=QZsZEaTtWQ41ERBJjSI5UWZNlVUNlVSZ0SsZERhdFO
Usage:
cmd open to bin directory, command:

ab -n Number of requests -c Path of concurrent access

4. Testing

2.2 Zookeeper implementation distributed principle

2.3 Zookeeper realizes distributed lock

1. Import dependencies (several dependencies need to be excluded)

	<dependencies>
        <dependency>
            <groupId>org.apache.zookeeper</groupId>
            <artifactId>zookeeper</artifactId>
            <version>3.6.0</version>
            <exclusions>
                <exclusion>
                    <artifactId>log4j</artifactId>
                    <groupId>log4j</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>slf4j-api</artifactId>
                    <groupId>org.slf4j</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>slf4j-log4j12</artifactId>
                    <groupId>org.slf4j</groupId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.apache.curator</groupId>
            <artifactId>curator-recipes</artifactId>
            <version>4.0.1</version>
            <exclusions>
                <exclusion>
                    <artifactId>slf4j-api</artifactId>
                    <groupId>org.slf4j</groupId>
                </exclusion>
            </exclusions>
        </dependency>

		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
		</dependency>

		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
		</dependency>
	</dependencies>

2. Write configuration class

@Configuration
public class ZKConfig {

    @Bean
    public CuratorFramework cf(){
        RetryPolicy retrypolicy=new ExponentialBackoffRetry(3000,2);
        CuratorFramework curatorFramework=CuratorFrameworkFactory.builder()
                 .connectString("192.168.31.240:2181,192.168.31.240:2182,192.168.31.240:2183").
                         retryPolicy(retrypolicy).build();
        curatorFramework.start();
        return curatorFramework;
    }
}

3. Add distributed locks to business code

@RestController
public class SecondKillController {
    @Autowired
    private CuratorFramework cf;

    //1. Prepare inventory of goods
    public static Map<String,Integer> itemStock=new HashMap<>();


    //2. Prepare orders for goods
    public static Map<String,Integer> itemOrder=new HashMap<>();

    static {
        itemStock.put("toothbrush",10000);
        itemOrder.put("toothbrush",0);
    }

    @GetMapping("/kill")
    public String kill(String item) throws Exception {
        InterProcessMutex lock=new InterProcessMutex(cf,"/lock");

        //... lock
        //lock.acquire();
        lock.acquire(1, TimeUnit.SECONDS);  //Specifies how long to queue to give up acquiring lock resources

//------------------------------Business code------------------------------------------
        //Release lock
        lock.release();

        //3. Return information
        return "Successful rush purchase!!!"+item+": The remaining inventory is - "+itemStock.get(item)+",The number of orders is:"+itemOrder.get(item);
    }

}

2.4 principle of redis implementing distributed lock

2.5 Redis implements distributed locks

1. Import dependency

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-redis</artifactId>
        </dependency>

Specify redis connection information

spring:
  redis:
    host: 192.168.31.240
    port: 6379

2. Authoring tool classes

@Component
public class RedisLockUtil {
    @Autowired
    private RedisTemplate redisTemplate;

    @GetMapping("/redis/kill")
    public boolean lcok(String key,String value,int second){
        return redisTemplate.opsForValue().setIfAbsent(key,value,second, TimeUnit.SECONDS);
    }

    @GetMapping("/zk/kill")
    public void  unlock(String key){
        redisTemplate.delete(key);
    }

}

3. Modify business logic code

    @GetMapping("/rediskill")
    public String rediskill(String item) throws Exception {

        //... lock
        if (lock.lcok(item,System.currentTimeMillis()+"",1)){
            //1. Inventory reduction
            Integer stock=itemStock.get(item);
            if (stock<=0){
                return "Insufficient inventory!!!";
            }
            Thread.sleep(100);
            itemStock.put("toothbrush",stock-1);
            //2. Create order
            Thread.sleep(100);
            itemOrder.put(item,itemOrder.get(item)+1);

            //Release lock
            lock.unlock(item);

            //3. Return information
            return "Successful rush purchase!!!"+item+": The remaining inventory is - "+itemStock.get(item)+",The number of orders is:"+itemOrder.get(item);
        }else {
            return "No goods were seized";
        }
    }

4. Testing
ab pressure test

3, Introduction to distributed tasks

4, Distributed task solution

4.1 elastic job introduction

Official website: http://shardingsphere.apache.org/elasticjob/
Secondary development products of Dangdang based on Quartz+Zookeeper
1. Based on Zookeeper distributed lock, only one service is guaranteed to perform scheduled tasks
2. The registration center is implemented based on Zookeeper to automatically help us schedule the specified services to perform scheduled tasks.
3. The registration center is realized based on Zookeeper, which automatically detects the health of the service based on the heartbeat.

4.2 elastic job implementation of distributed tasks

1. Create SpringBoot project
2. Import dependency

        <dependency>
            <groupId>com.dangdang</groupId>
            <artifactId>elastic-job-lite-spring</artifactId>
            <version>2.1.5</version>
        </dependency>

3. Configure Zookeeper information

@Configuration
public class ElasticJobConfig {

    //Registration Center
    @Bean
    public CoordinatorRegistryCenter center(){
        CoordinatorRegistryCenter regCenter = new ZookeeperRegistryCenter(
                new ZookeeperConfiguration("192.168.0.142:2181,192.168.0.142:2182,192.168.0.142:2183", "elastic-job-demo"));
        regCenter.init();
        return regCenter;
    }
}

4. Creates the specified scheduled task

@Component
public class MyElasticJob implements SimpleJob {

    @Override
    public void execute(ShardingContext context) {
        switch (context.getShardingItem()) {
            case 0:
                System.out.println("Execute 0 task");
                break;
            case 1:
                System.out.println("Perform 1 task");
                break;
            case 2:
                System.out.println("Perform 2 tasks");
                break;
            // case n: ...
        }
    }
}

5. Configure the execution cycle and start scheduling tasks

@SpringBootApplication
public class JobApplication {

    public static void main(String[] args) {
        SpringApplication.run(JobApplication.class, args);
    }


    //Specify task scheduling information
    @Bean
    public SpringJobScheduler scheduler(MyElasticJob job, CoordinatorRegistryCenter center){
        // Define job core configuration
        JobCoreConfiguration simpleCoreConfig = JobCoreConfiguration.
                newBuilder("demoSimpleJob", "0/10 * * * * ?", 3).shardingItemParameters("0=A,1=B,2=C").build();
        // Define SIMPLE type configuration
        SimpleJobConfiguration simpleJobConfig = new SimpleJobConfiguration(simpleCoreConfig, MyElasticJob.class.getCanonicalName());
        // Define Lite job root configuration
        LiteJobConfiguration simpleJobRootConfig = LiteJobConfiguration.newBuilder(simpleJobConfig).build();

        //Define SpringJobScheduler
        SpringJobScheduler scheduler=new SpringJobScheduler(job,center,simpleJobRootConfig);
        scheduler.init();
        return scheduler;
    }
}

6. Testing

5, Introduction to distributed transaction

5.1 introduction

5.2 basic theory

  • CAP theory, C: consistency, A: availability, P: partition fault tolerance, whichever is the second.

    • Eureka: AP ensures availability and discards consistency
    • Zookeeper: CP, each node must be able to find a Master to provide external services, abandoning availability.
  • Base theory, BA: basic availability, S: intermediate state, E: final consistency

    • Based on CAP theory, it is a trade-off between consistency and availability in CAP theorem.
    • Core idea: we cannot achieve strong consistency, but each application can be weighed in some appropriate ways according to its own business characteristics to finally achieve consistency.
    • BA: there are some problems in the distributed system due to one reason, which allows the loss of the availability of some services and ensures the high availability of my core functions.
    • S: Allow an intermediate state between systems, which will not affect the normal use of the whole system, and allow data synchronization delay
    • E: After a certain period of time, all data copies in the system can finally reach a consistent state without ensuring strong consistency of system data.

6, Distributed transaction solution

  • Soft transaction (Base theory), rigid transaction (ACID).

6.1 PC submission

  • There are two stages of submission at both ends

    • The first stage is the preparation stage. Participants need to start transactions and execute SQL to ensure that the appropriate data already exists in the database. The participant returns to the transaction manager for OK.
    • In the second stage, when the transaction manager receives the notification from all participants, it sends a Commit request to all participants.
  • Problem 1: the performance of execution is very low, generally more than 10 times that of traditional transactions.

  • Question 2: the transaction manager does not have a timeout.

  • Problem 3: the transaction manager has a single point of failure.

6.2 3PC three terminal submission

  • 1. On the basis of the two-stage submission, the three-stage submission introduces the timeout mechanism, and on the basis of the two-stage submission, another step is added. Before submitting the transaction, ask whether the log information of the database has been improved.

6.3 TCC mechanism

  • TCC (Try, Confirm, Cancel) fits in with your business code.
    • Try: try to pre execute specific business code. Order
    • try succeeded, Confirm: execute the Confirm code again
    • try failed, Cancel: execute the Cancel code again.

6.4 MQ distributed transactions

  • RabbitMQ uses the confirm mechanism to ensure that messages are sent to the MQ service when sending messages. Consumers have a manual ack mechanism to ensure that messages are consumed in MQ.

6.5 LCN implementation of distributed transactions

  • Based on three-stage submission and TCC implementation
    1. Create a coordinator project and two services
    2. Coordinator: add dependencies, write configuration files, and add annotations
  • rely on
        <dependency>
            <groupId>com.codingapi.txlcn</groupId>
            <artifactId>txlcn-tm</artifactId>
            <version>5.0.2.RELEASE</version>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
@SpringBootApplication
@EnableTransactionManagerServer
public class TxManagerApplication {

    public static void main(String[] args) {
        SpringApplication.run(TxManagerApplication.class, args);
    }

}
  • application.yml
server:
  port: 8080
spring:
  datasource:
    driver-class-name: com.mysql.cj.jdbc.Driver
    url: jdbc:mysql:///lcn?serverTimezone=UTC
    username: root
    password: 123456
  redis:
    host: 192.168.31.240
    port: 6379
# Coordinated port number
tx-lcn:
  manager:
    port: 8070
  • Start class addition
@EnableTransactionManagerServer

  • Create TX LCN table
DROP TABLE IF EXISTS `t_tx_exception`;
CREATE TABLE `t_tx_exception`  (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `group_id` varchar(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL,
  `unit_id` varchar(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL,
  `mod_id` varchar(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL,
  `transaction_state` tinyint(4) NULL DEFAULT NULL,
  `registrar` tinyint(4) NULL DEFAULT NULL COMMENT '-1 Unknown 0 Manager Notification transaction failed, 1 client Failed to query the transaction status. 2. The transaction initiator failed to close the transaction group',
  `ex_state` tinyint(4) NULL DEFAULT NULL COMMENT '0 Pending 1 processed',
  `create_time` datetime(0) NULL DEFAULT NULL,
  PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 967 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_general_ci ROW_FORMAT = Dynamic;

SET FOREIGN_KEY_CHECKS = 1;

3. Services: add dependencies, write configuration files, and add annotations

  • rely on
 <dependency>
            <groupId>com.codingapi.txlcn</groupId>
            <artifactId>txlcn-tc</artifactId>
            <version>5.0.1.RELEASE</version>
        </dependency>

        <dependency>
            <groupId>com.codingapi.txlcn</groupId>
            <artifactId>txlcn-txmsg-netty</artifactId>
            <version>5.0.1.RELEASE</version>
        </dependency>

  • application.yml
server:
  port: 8081
spring:
  datasource:
    driver-class-name: com.mysql.cj.jdbc.Driver
    url: jdbc:mysql:///lcn?serverTimezone=UTC
    username: root
    password: 123456
# Coordinated port number
tx-lcn:
  manager:
    port: 8070
  • Add annotation to startup class
@EnableDistributedTransaction

  • Start the transaction and add the transaction in the server layer
@Service
public class ServiceA {
    
    @Autowired
    private ValueDao valueDao; //Local db operation
    
    @Autowired
    private ServiceB serviceB;//Remote B module service
    
    @LcnTransaction //Distributed transaction annotation
    @Transactional //Local transaction annotation
    public String execute(String value) throws BusinessException {
        // step1. call remote service B
        String result = serviceB.rpc(value);  // (1)
        // step2. local store operate. DTX commit if save success, rollback if not.
        valueDao.save(value);  // (2)
        valueDao.saveBackup(value);  // (3)
        return result + " > " + "ok-A";
    }
}

4. Testing

Tags: Java Web Development Redis Spring Boot Spring Cloud

Posted on Mon, 06 Sep 2021 18:40:49 -0400 by tom_b