Distributed coordination Zookeeper (Distributed Lock & Leader election)

Distributed coordination zookeeper (Distributed Lock & leader election)

In the case of microservices, we usually use cluster deployment to relieve node pressure. If multiple users rob a commodity at the same time, if we don't handle it at the back end, there will be a problem. However, traditional synchronized can't solve the cross process problem. Then we must introduce a third-party perspective to help us solve this problem, Some features of zk can help us to realize the problem of distributed locking.

  • [node uniqueness] : because the node name on zk is unique, we can create nodes with the same name on multiple clients at the same time. On zk, the successfully created node is the node that obtains the lock. If the node is not successfully created, we can listen to the node deletion event through zk's watcher mechanism. Once the node is deleted, other nodes can create files.
    • Problem: there may be a crowd startling effect, which is a burden for zkserver communication.
  • [characteristics of ordered node]: all clients create a temporary ordered node, and they are all under a container node (within a certain period of time, if there is no child node under the container node, the container node will be deleted) , then the smallest node is the client that obtains the lock, and those who do not obtain the lock are equivalent to queuing. All they have to do is listen to the deletion event of their next node, which solves the problem of group shock effect.

curator has provided some such implementations. Let's write a demo and analyze its source implementation.

  curator implements distributed locking

pom

<dependency> <groupId>org.apache.curator</groupId> <artifactId>curator-framework</artifactId> <version>5.2.0</version> </dependency> <dependency> <groupId>org.apache.curator</groupId> <artifactId>curator-recipes</artifactId> <version>5.2.0</version> </dependency>

Injection curator

@Configuration public class CuratorConfig { @Bean public CuratorFramework curatorFramework(){ CuratorFramework curatorFramework=CuratorFrameworkFactory .builder() .connectString("192.168.221.128:2181") .sessionTimeoutMs(15000) .connectionTimeoutMs(20000) .retryPolicy(new ExponentialBackoffRetry(1000,10)) .build(); curatorFramework.start(); return curatorFramework; } }

Create multiple threads to simulate the client to deduct inventory, and use the ordered node api provided by curator to realize the integrity of locking function.

 @GetMapping("{goodsNo}")
    public String  purchase(@PathVariable("goodsNo")Integer goodsNo) throws Exception {
        QueryWrapper<GoodsStock> queryWrapper=new QueryWrapper<>();
        queryWrapper.eq("goods_no",goodsNo);
        //Distributed lock based on temporary ordered nodes.
        InterProcessMutex lock=new InterProcessMutex(curatorFramework,"/Locks");
        try {
            //Preempt distributed lock resources (blocked)
            lock.acquire();
            GoodsStock goodsStock = goodsStockService.getOne(queryWrapper);
            Thread.sleep(new Random().nextInt(1000));
            if (goodsStock == null) {
                return "The specified item does not exist";
            }
            if (goodsStock.getStock() < 1) {
                return "Insufficient inventory";
            }
            goodsStock.setStock(goodsStock.getStock() - 1);
            boolean res = goodsStockService.updateById(goodsStock);
            if (res) {
                return "Rush to buy books:" + goodsNo + "success";
            }
        }finally {
            lock.release(); //Release lock
        }
        return "Rush purchase failed";
    }

Source code analysis of distributed lock implemented by cursor

The process is:

Acquire lock: judge whether you only have the smallest node. If yes, you will acquire the lock. If not, you will acquire the lock circularly. Listen to the previous node at one time and wait with wait. When a listening event is received, you will wake up the thread with notify.

Delete lock: because synchronize is a re-entry lock, first delete the re-entry times of the lock, then delete the nodes stored in the map, and then delete the nodes in zk.

[acquire lock]:

private boolean internalLock(long time, TimeUnit unit) throws Exception
{
  
    
    Thread currentThread = Thread.currentThread();
    //Judge whether the current thread has acquired the lock
    LockData lockData = threadData.get(currentThread);
    
    // The current thread has acquired a lock (the reentrant nature of the lock)
    if ( lockData != null )
    {
        // re-entering
        lockData.lockCount.incrementAndGet();
        return true;
    }

    // Attempt to acquire lock
    String lockPath = internals.attemptLock(time, unit, getLockNodeBytes());
    if ( lockPath != null )
    {
        //Build a lock data
        LockData newLockData = new LockData(currentThread, lockPath);
        //Save in map in
        threadData.put(currentThread, newLockData);
        return true;
    }

    return false;
}
String attemptLock(long time, TimeUnit unit, byte[] lockNodeBytes) throws Exception
{
    final long      startMillis = System.currentTimeMillis();
    final Long      millisToWait = (unit != null) ? unit.toMillis(time) : null;
    final byte[]    localLockNodeBytes = (revocable.get() != null) ? new byte[0] : lockNodeBytes;
    int             retryCount = 0;

    String          ourPath = null;
    boolean         hasTheLock = false;
    boolean         isDone = false;
    while ( !isDone )
    {
        isDone = true;

        try
        {
            //Create a temporary ordered node and return the name of the current node
            ourPath = driver.createsTheLock(client, path, localLockNodeBytes);
            //Obtain the lock through cyclic operation
            hasTheLock = internalLockLoop(startMillis, millisToWait, ourPath);
        }
        catch ( KeeperException.NoNodeException e )
        {
            if ( client.getZookeeperClient().getRetryPolicy().allowRetry(retryCount++, System.currentTimeMillis() - startMillis, RetryLoop.getDefaultRetrySleeper()) )
            {
                isDone = false;
            }
            else
            {
                throw e;
            }
        }
    }

    //Returns if the lock has been obtained
    if ( hasTheLock )
    {
        return ourPath;
    }

    return null;
}
private boolean internalLockLoop(long startMillis, Long millisToWait, String ourPath) throws Exception
{
    boolean     haveTheLock = false;
    boolean     doDelete = false;
    try
    {
        if ( revocable.get() != null )
        {
            client.getData().usingWatcher(revocableWatcher).forPath(ourPath);
        }

       //As long as the lock is not obtained and the connection is started, it will cycle and open all the time
        while ( (client.getState() == CuratorFrameworkState.STARTED) && !haveTheLock )
        {
            //Get sorted list
            List<String>        children = getSortedChildren();
            //Gets the serial number of the created node
            String      sequenceNodeName = ourPath.substring(basePath.length() + 1); 
            //Compare whether the serial number is the smallest. If it is the smallest, return. Otherwise, return a node 1 smaller than yourself and wrap it into PredicateResults
            PredicateResults    predicateResults = driver.getsTheLock(client, children, sequenceNodeName, maxLeases);
            // Get lock modification haveTheLock by true Then exit the loop
            if ( predicateResults.getsTheLock() )
            {
                haveTheLock = true;
            }
            else
            {
                //The previous node of the current node
                String  previousSequencePath = basePath + "/" + predicateResults.getPathToWatch();

                synchronized(this)
                {
                    try
                    {
                    //One time listening for the previous node  
                    client.getData().usingWatcher(watcher).forPath(previousSequencePath);
                        if ( millisToWait != null )
                        {
                            millisToWait -= (System.currentTimeMillis() - startMillis);
                            startMillis = System.currentTimeMillis();
                            if ( millisToWait <= 0 )
                            {
                                doDelete = true;    // timed out - delete our node
                                break;
                            }

                            wait(millisToWait);
                        }
                        else
                        {
                            wait();
                        }
                    }
                    catch ( KeeperException.NoNodeException e )
                    {
                        // it has been deleted (i.e. lock released). Try to acquire again
                    }
                }
            }
        }
    }
    catch ( Exception e )
    {
        ThreadUtils.checkInterrupted(e);
        doDelete = true;
        throw e;
    }
    finally
    {
        if ( doDelete )
        {
            deleteOurPath(ourPath);
        }
    }
    return haveTheLock;
}

[delete lock]:

@Override
public void release() throws Exception
{
    /*
        Note on concurrency: a given lockData instance
        can be only acted on by a single thread so locking isn't necessary
     */

    Thread currentThread = Thread.currentThread();
    LockData lockData = threadData.get(currentThread);
    // Unable to get lock information from the mapping table, indicating that no lock is currently held
    if ( lockData == null )
    {
        throw new IllegalMonitorStateException("You do not own the lock: " + basePath);
    }
    // The lock is reentrant, with an initial value of 1 and atoms-1 The lock is not released until 0
    int newLockCount = lockData.lockCount.decrementAndGet();
    if ( newLockCount > 0 )
    {
        return;
    }
    if ( newLockCount < 0 )
    {
        throw new IllegalMonitorStateException("Lock count has gone negative for lock: " + basePath);
    }
    try
    {
        // lockData != null && newLockCount == 0,Release lock resource
        internals.releaseLock(lockData.lockPath);
    }
    finally
    {
        // Finally, remove the lock information of the current thread from the mapping table
        threadData.remove(currentThread);
    }
}
final void releaseLock(String lockPath) throws Exception
{
    //Remove subscription event
    client.removeWatchers();
    revocable.set(null);
    // Deleting a temporary sequential node will only trigger the next sequential node to obtain the lock. Theoretically, there is no competition, only queuing, non preemption, fair lock, first come, first served
    deleteOurPath(lockPath);
}

   Implement leader election with curator

In fact, the bottom layer still uses zk's ordered node feature. Whoever is young is the leader. We use quartz to write a demo here. When a quartz hangs, other quartz will execute immediately. In fact, we use water to listen to other nodes. Once the leader releases the lock (he is the smallest), he will become the leader

Use a factory bean to maintain a state (whether it is a leader) and judge the state when spring starts. If it is a leader, execute scheduled tasks.

//SchedulerFactoryBean It's a project bean,Through this handle quartz Information transferred to Spring In, you can trigger scheduled tasks
public class ZkSchedulerFactoryBean extends SchedulerFactoryBean {

    private LeaderLatch leaderLatch;

    //namespace
    private final String LEADER_PATH="/leader";


    public ZkSchedulerFactoryBean() throws Exception {
        //stay Spring Do not automatically start scheduled tasks when starting
        this.setAutoStartup(false);
        leaderLatch=new LeaderLatch(getClient(),LEADER_PATH);
        //When leader When things change,Will callback LeaderLatchListener
        leaderLatch.addListener(new GlenLeaderLatchListener(this));
        //Start listening
        leaderLatch.start();
    }

    //connect zk
    private CuratorFramework getClient(){
        CuratorFramework curatorFramework= CuratorFrameworkFactory
                .builder()
                .connectString("192.168.43.3:2181")
                .sessionTimeoutMs(15000)
                .connectionTimeoutMs(20000)
                .retryPolicy(new ExponentialBackoffRetry(1000,10))
                .build();
        curatorFramework.start();
        return curatorFramework;
    }

    //Create an instance of scheduled scheduling
    @Override
    protected void startScheduler(Scheduler scheduler, int startupDelay) throws SchedulerException {
        //If it is in the startup state, when the current node grabs leader When, this state will be set to true,Then the scheduled task starts
        if(this.isAutoStartup()) {
            super.startScheduler(scheduler, startupDelay);
        }
    }

    //Release resources
    @Override
    public void destroy() throws SchedulerException {
        CloseableUtils.closeQuietly(leaderLatch);
        super.destroy();
    }
}

zk's listening class. If it is a leader, modify the status in the above factory class and start the scheduled task

public class GlenLeaderLatchListener implements LeaderLatchListener {
    //Method for controlling start and stop of timed task
    private SchedulerFactoryBean schedulerFactoryBean;

    GlenLeaderLatchListener(SchedulerFactoryBean schedulerFactoryBean) {
        this.schedulerFactoryBean = schedulerFactoryBean;
    }

    //When he is leader Set his state to the startup state when he is
    @Override
    public void isLeader() {
        System.out.println(Thread.currentThread().getName()+"Become leader");
        schedulerFactoryBean.setAutoStartup(true);
        schedulerFactoryBean.start();
    }

    //If he's not leader,Stop performing scheduled tasks
    @Override
    public void notLeader() {
        System.out.println(Thread.currentThread().getName()+"seize leader Failed, do not execute task");
        schedulerFactoryBean.setAutoStartup(false);
        schedulerFactoryBean.stop();
    }
}

A class that performs scheduled tasks

public class QuartzJob extends QuartzJobBean {

    @Override
    protected void executeInternal(JobExecutionContext jobExecutionContext) {
        System.out.println("Start scheduled task");
        SimpleDateFormat sdf=new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
        System.out.println("Current system time:"+sdf.format(new Date()));
    }
}

Give these things to Spring

@Configuration
public class QuartzConfiguration {


    @Bean
    public ZkSchedulerFactoryBean schedulerFactoryBean(JobDetail jobDetail,Trigger trigger) throws Exception {
        ZkSchedulerFactoryBean zkSchedulerFactoryBean=new ZkSchedulerFactoryBean();
        zkSchedulerFactoryBean.setJobDetails(jobDetail);
        zkSchedulerFactoryBean.setTriggers(trigger);
        return zkSchedulerFactoryBean;
    }

    //A class that performs scheduled tasks
    @Bean
    public JobDetail jobDetail(){
        return JobBuilder.newJob(QuartzJob.class).storeDurably().build();
    }

    //Class that triggers scheduled tasks, 1 s Execute once, always
    @Bean
    public Trigger trigger(JobDetail jobDetail){
        SimpleScheduleBuilder simpleScheduleBuilder=
                SimpleScheduleBuilder.simpleSchedule().withIntervalInSeconds(1).repeatForever();
        return TriggerBuilder.newTrigger().forJob(jobDetail).withSchedule(simpleScheduleBuilder).build();
    }
}

We started two SpingBoot applications

 

  It is found that there is a leader node on zk, and a client has preempted the leader

At this time, stop one of the nodes, and the other node will immediately monitor the change of the leader, and then start execution

  Because he is already the smallest node

 

Tags: Distribution

Posted on Wed, 24 Nov 2021 14:03:54 -0500 by ghornet