Apache ShenYu source code reading series - data synchronization based on Http long polling

Apache ShenYu Is an asynchronous, high-performance, cross language, responsive API gateway.

In ShenYu gateway, data synchronization refers to how to synchronize the updated data to the gateway after the updated data is sent in the background management system. Apache ShenYu gateway currently supports ZooKeeper, WebSocket, Http long polling, Nacos, Etcd and Consul for data synchronization. The main content of this paper is the source code analysis of data synchronization based on Http long polling.

This paper analyzes the source code based on shenyu-2.4.0. Please refer to the introduction of the official website Data synchronization principle .

1. Http long polling

The relevant descriptions on the official website are directly quoted here:

The data synchronization mechanism of Zookeeper and WebSocket is relatively simple, while HTTP long polling is relatively complex. Apache ShenYu draws on the design idea of Apollo and Nacos, extracts its essence, and realizes the Http long polling data synchronization function. Note that this is not the traditional ajax long polling!

Http long polling mechanism is as shown above. Apache ShenYu gateway actively requests Shenyu admin configuration service, and the read timeout is 90s, which means that the gateway layer will wait up to 90s for requesting configuration service, so that Shenyu admin configuration service can respond to change data in time, so as to realize quasi real-time push.

The Http long polling mechanism is initiated by the gateway to request Shenyu admin, so we will start from the gateway side for this source code analysis.

2. Gateway data synchronization

2.1 loading configuration

The Http long polling data synchronization configuration is loaded through the starter mechanism of spring boot. It will be loaded when we introduce relevant dependencies and have the following configurations in the configuration file.

Introducing dependencies in pom files:

        <!--shenyu data sync start use http-->
        <dependency>
        	<groupId>org.apache.shenyu</groupId>
        	<artifactId>shenyu-spring-boot-starter-sync-data-http</artifactId>
        	<version>${project.version}</version>
        </dependency>

Add a configuration in the application.yml configuration file:

shenyu:
    sync:
       http:
          url : http://localhost:9095

When the gateway is started, the configuration class HttpSyncDataConfiguration will execute and load the corresponding Bean.

/**
 * Http sync data configuration for spring boot.
 */
@Configuration
@ConditionalOnClass(HttpSyncDataService.class)
@ConditionalOnProperty(prefix = "shenyu.sync.http", name = "url")
@Slf4j
public class HttpSyncDataConfiguration {

    /**
     * Http sync data service.
     * Create HttpSyncDataService 
     * @param httpConfig         http Configuration of
     * @param pluginSubscriber   Plug in data subscription
     * @param metaSubscribers    Metadata subscription
     * @param authSubscribers    Authentication data subscription
     * @return the sync data service
     */
    @Bean
    public SyncDataService httpSyncDataService(final ObjectProvider<HttpConfig> httpConfig, final ObjectProvider<PluginDataSubscriber> pluginSubscriber,
                                           final ObjectProvider<List<MetaDataSubscriber>> metaSubscribers, final ObjectProvider<List<AuthDataSubscriber>> authSubscribers) {
        log.info("you use http long pull sync shenyu data");
        return new HttpSyncDataService(Objects.requireNonNull(httpConfig.getIfAvailable()), Objects.requireNonNull(pluginSubscriber.getIfAvailable()),
                metaSubscribers.getIfAvailable(Collections::emptyList), authSubscribers.getIfAvailable(Collections::emptyList));
    }

    /**
     * Http config http config.
     * Read http configuration
     * @return the http config
     */
    @Bean
    @ConfigurationProperties(prefix = "shenyu.sync.http")
    public HttpConfig httpConfig() {
        return new HttpConfig();
    }
}

HttpSyncDataConfiguration is a configuration class for HTTP long polling data synchronization. It is responsible for creating HttpSyncDataService (responsible for the specific implementation of HTTP data synchronization) and HttpConfig (admin attribute configuration). Its notes are as follows:

  • @Configuration: indicates that this is a configuration class;
  • @ConditionalOnClass(HttpSyncDataService.class): condition annotation, indicating that the class HttpSyncDataService is required;
  • @ConditionalOnProperty(prefix = "shenyu.sync.http", name = "url"): the condition annotation should be configured with the property shenyu.sync.http.url.

2.2 attribute initialization

  • HttpSyncDataService

Complete property initialization in the constructor of HttpSyncDataService.

public class HttpSyncDataService implements SyncDataService, AutoCloseable {

    // Omitted attribute field

    public HttpSyncDataService(final HttpConfig httpConfig, final PluginDataSubscriber pluginDataSubscriber, final List<MetaDataSubscriber> metaDataSubscribers, final List<AuthDataSubscriber> authDataSubscribers) {
        // 1. Create a data processor
        this.factory = new DataRefreshFactory(pluginDataSubscriber, metaDataSubscribers, authDataSubscribers);
        // 2. Get admin attribute configuration
        this.httpConfig = httpConfig;
        // The url of Shenyu admin. Multiple URLs are separated by commas (,)
        this.serverList = Lists.newArrayList(Splitter.on(",").split(httpConfig.getUrl()));
        // 3. Create an httpClient to initiate a request to admin
        this.httpClient = createRestTemplate();
        // 4. Start the long polling task
        this.start();
    }

    //......
}

Other functions and related fields are omitted from the above code, and the initialization of attributes is completed in the constructor, mainly including:

  • Create a data processor for subsequent caching of various types of data (plug-ins, selectors, rules, metadata and authentication data);

  • Obtain the admin attribute configuration, mainly to obtain the url of admin. Admin may be a cluster, and multiple are separated by commas (,);

  • Create httpClient using RestTemplate, which is used to send a request to admin;

        private RestTemplate createRestTemplate() {
            OkHttp3ClientHttpRequestFactory factory = new OkHttp3ClientHttpRequestFactory();
    
            // The connection establishment timeout is 10s
            factory.setConnectTimeout((int) this.connectionTimeout.toMillis());
    
            // The gateway actively requests the configuration service of Shenyu admin, and the read timeout is 90s
            factory.setReadTimeout((int) HttpConstants.CLIENT_POLLING_READ_TIMEOUT);
            return new RestTemplate(factory);
        }
    
  • Start long polling task.

2.3 start long polling

  • HttpSyncDataService#start()

In the start() method, two things are done. One is to obtain the full amount of data, that is, request the admin side to obtain all the data to be synchronized, and then cache the obtained data into the gateway memory. The other is to enable multithreading to perform long polling tasks.

private void start() {
        // It is initialized only once and implemented through atomic classes. 
        RUNNING = new AtomicBoolean(false);
        // It could be initialized multiple times, so you need to control that.
        if (RUNNING.compareAndSet(false, true)) {
            // fetch all group configs.
            // Initial startup to obtain full data
            this.fetchGroupConfig(ConfigGroupEnum.values());

            // A background service, a thread
            int threadSize = serverList.size();
            // Custom thread pool
            this.executor = new ThreadPoolExecutor(threadSize, threadSize, 60L, TimeUnit.SECONDS,
                    new LinkedBlockingQueue<>(),
                    ShenyuThreadFactory.create("http-long-polling", true));
            // start long polling, each server creates a thread to listen for changes.
            
            // Start long polling, an admin service, and create a thread for data synchronization
            this.serverList.forEach(server -> this.executor.execute(new HttpLongPollingTask(server)));
        } else {
            log.info("shenyu http long polling was started, executor=[{}]", executor);
        }
    }

2.3.1 obtaining full data
  • HttpSyncDataService#fetchGroupConfig()

ShenYu grouped all data to be synchronized. There are five data types: plug-in, selector, rule, metadata and authentication data.

public enum ConfigGroupEnum {
    APP_AUTH, // Authentication data
    PLUGIN, //plug-in unit
    RULE, // rule
    SELECTOR, // selector
    META_DATA; // metadata
}

Admin may be a cluster. Here, requests are sent to each admin in a circular manner. If one of them is executed successfully, the operation of obtaining the full amount of data from admin and caching it to the gateway will be executed successfully. If an exception occurs, a request is made to the next admin.

private void fetchGroupConfig(final ConfigGroupEnum... groups) throws ShenyuException {
    // Admin may be a cluster, where requests are sent to each admin in a circular manner
        for (int index = 0; index < this.serverList.size(); index++) {
            String server = serverList.get(index);
            try {
                // Really implement
                this.doFetchGroupConfig(server, groups);
                // If one is successful, it will be successful. You can exit the cycle
                break;
            } catch (ShenyuException e) {
                // An exception occurred, trying to execute the next
                // The last one also fails to execute and throws an exception
                // no available server, throw exception.
                if (index >= serverList.size() - 1) {
                    throw e;
                }
                log.warn("fetch config fail, try another one: {}", serverList.get(index + 1));
            }
        }
    }
  • HttpSyncDataService#doFetchGroupConfig()

In this method, first assemble the request parameters, then initiate the request through httpClient, obtain the data from admin, and finally update the obtained data to the gateway memory.

// Send a request to the admin background management system to obtain all synchronous data
private void doFetchGroupConfig(final String server, final ConfigGroupEnum... groups) {
    // 1. Spell request parameters, all grouping enumeration types
    StringBuilder params = new StringBuilder();
    for (ConfigGroupEnum groupKey : groups) {
        params.append("groupKeys").append("=").append(groupKey.name()).append("&");
    }

    // Interface provided by admin side / configs/fetch
    String url = server + "/configs/fetch?" + StringUtils.removeEnd(params.toString(), "&");
    log.info("request configs: [{}]", url);
    String json = null;
    try {
        // 2. Initiate a request to obtain change data
        json = this.httpClient.getForObject(url, String.class);
    } catch (RestClientException e) {
        String message = String.format("fetch config fail from server[%s], %s", url, e.getMessage());
        log.warn(message);
        throw new ShenyuException(message, e);
    }
    // update local cache
    // 3. Update data in gateway memory
    boolean updated = this.updateCacheWithJson(json);
    // If the update is successful, this method is completed
    if (updated) {
        log.info("get latest configs: [{}]", json);
        return;
    }
    // not updated. it is likely that the current config server has not been updated yet. wait a moment.
    log.info("The config of the server[{}] has not been updated or is out of date. Wait for 30s to listen for changes again.", server);
    // The server doesn't update the data, just wait for 30s
    ThreadUtils.sleep(TimeUnit.SECONDS, 30);
}

From the code, you can see that the interface for obtaining full data provided by the admin side is / configs/fetch. I won't go further here, but I'll analyze it later.

If the result data returned by admin is obtained and successfully updated, the execution of this method ends. If the update is not successful, the server may not update the data and wait for 30s.

It needs to be explained in advance. When the gateway judges whether the update is successful, it has the operation of comparing data, which will be mentioned soon.

  • HttpSyncDataService#updateCacheWithJson

Update data in gateway memory. Use GSON for deserialization, get the real data from the attribute data, and then give it to DataRefreshFactory for update.

    private boolean updateCacheWithJson(final String json) {
        // Deserialization using GSON
        JsonObject jsonObject = GSON.fromJson(json, JsonObject.class);
        JsonObject data = jsonObject.getAsJsonObject("data");
        // if the config cache will be updated?
        return factory.executor(data);
    }
  • DataRefreshFactory#executor()

Update data according to different data types and return update results. parallelStream() is used for parallel update, and the specific update logic is handed over to the dataRefresh.refresh() method. In the update result, one data type has been updated, which indicates that the operation has been updated.

    public boolean executor(final JsonObject data) {
        //Parallel update data
        List<Boolean> result = ENUM_MAP.values().parallelStream()
                .map(dataRefresh -> dataRefresh.refresh(data))
                .collect(Collectors.toList());
        //An update indicates that an update operation has occurred this time
        return result.stream().anyMatch(Boolean.TRUE::equals);
    }
  • AbstractDataRefresh#refresh()

The data update logic adopts the template method design pattern, the general operation is completed in the abstract method, and the different implementation logic is completed by the subclass. There are some differences in the specific update logic of the five data types, but there are also general update logic. The class diagram relationship is as follows:

In the general refresh() method, it is responsible for data type conversion, judging whether it needs to be updated, and the actual data refresh operation.

    @Override
    public Boolean refresh(final JsonObject data) {
        boolean updated = false;
        // Data type conversion
        JsonObject jsonObject = convert(data);
        if (null != jsonObject) {
            // Get data type
            ConfigData<T> result = fromJson(jsonObject);
            // Update required
            if (this.updateCacheIfNeed(result)) {
                updated = true;
                // Real update logic, data refresh operation
                refresh(result.getData());
            }
        }
        return updated;
    }
  • AbstractDataRefresh#updateCacheIfNeed()

The process of data conversion is to convert according to different data types. We won't track further to see whether the data needs to be updated. The method name is updateCacheIfNeed(), which is implemented through method overloading.

// result is data
protected abstract boolean updateCacheIfNeed(ConfigData<T> result);

// newVal is the latest value obtained
// What data type is groupEnum
protected boolean updateCacheIfNeed(final ConfigData<T> newVal, final ConfigGroupEnum groupEnum) {
        // If it is the first time, it is directly put into the cache and returns true, indicating that the update has been made this time
        if (GROUP_CACHE.putIfAbsent(groupEnum, newVal) == null) {
            return true;
        }
        ResultHolder holder = new ResultHolder(false);
        GROUP_CACHE.merge(groupEnum, newVal, (oldVal, value) -> {
            // The md5 value is the same and does not need to be updated
            if (StringUtils.equals(oldVal.getMd5(), newVal.getMd5())) {
                log.info("Get the same config, the [{}] config cache will not be updated, md5:{}", groupEnum, oldVal.getMd5());
                return oldVal;
            }

            // The modification time of the current cached data is longer than that of the new data, so it does not need to be updated
            // must compare the last update time
            if (oldVal.getLastModifyTime() >= newVal.getLastModifyTime()) {
                log.info("Last update time earlier than the current configuration, the [{}] config cache will not be updated", groupEnum);
                return oldVal;
            }
            log.info("update {} config: {}", groupEnum, newVal);
            holder.result = true;
            return newVal;
        });
        return holder.result;
    }

As can be seen from the above source code, there are two situations that do not need to be updated:

  • The md5 values of the two data are the same and do not need to be updated;
  • The modification time of the currently cached data is greater than that of the new data, so it does not need to be updated.

Other situations require updating data.

After the analysis, the logic analysis of starting the start() method for the first time and obtaining the full amount of data is completed, followed by the operation of long polling. For convenience, I paste the start() method again:

    private void start() {
        // It could be initialized multiple times, so you need to control that.
        if (RUNNING.compareAndSet(false, true)) {
            // fetch all group configs.
            // Initial startup to obtain full data
            this.fetchGroupConfig(ConfigGroupEnum.values());

            // A background service, a thread
            int threadSize = serverList.size();
            // Custom thread pool
            this.executor = new ThreadPoolExecutor(threadSize, threadSize, 60L, TimeUnit.SECONDS,
                    new LinkedBlockingQueue<>(),
                    ShenyuThreadFactory.create("http-long-polling", true));
            // start long polling, each server creates a thread to listen for changes.
            // Start long polling, an admin service, and create a thread for data synchronization
            this.serverList.forEach(server -> this.executor.execute(new HttpLongPollingTask(server)));
        } else {
            log.info("shenyu http long polling was started, executor=[{}]", executor);
        }
    }
2.3.2 execute long polling task
  • HttpLongPollingTask#run()

The long polling task is HttpLongPollingTask, which implements the Runnable interface, and the task logic is in the run() method. Through the while() loop, the task is executed continuously, that is, long polling. There are three retry logics in each polling. One polling task fails. Wait for 5s to continue. All three times fail. Wait for 5 minutes to try again.

Start long polling, an admin service, and create a thread for data synchronization.

class HttpLongPollingTask implements Runnable {

        private String server;

        // Retry 3 times by default
        private final int retryTimes = 3;

        HttpLongPollingTask(final String server) {
            this.server = server;
        }

        @Override
        public void run() {
            // Always polling
            while (RUNNING.get()) {
                for (int time = 1; time <= retryTimes; time++) {
                    try {
                        doLongPolling(server);
                    } catch (Exception e) {
                        // print warnning log.
                        if (time < retryTimes) {
                            log.warn("Long polling failed, tried {} times, {} times left, will be suspended for a while! {}",
                                    time, retryTimes - time, e.getMessage());
                            // Long polling failed. Wait for 5s to continue
                            ThreadUtils.sleep(TimeUnit.SECONDS, 5);
                            continue;
                        }
                        // print error, then suspended for a while.
                        log.error("Long polling failed, try again after 5 minutes!", e);
                        // All three times failed. Try again in 5 minutes
                        ThreadUtils.sleep(TimeUnit.MINUTES, 5);
                    }
                }
            }
            log.warn("Stop http long polling.");
        }
    }
  • HttpSyncDataService#doLongPolling()

Core logic for executing long polling tasks:

  • Assemble request parameters according to data type: md5 and lastModifyTime;
  • Assembling a request header and a request body;
  • Send a request to admin to judge whether the group data has changed;
  • Get the data according to the changed group.
private void doLongPolling(final String server) {
        // Assembly request parameters: md5 and lastModifyTime
        MultiValueMap<String, String> params = new LinkedMultiValueMap<>(8);
        for (ConfigGroupEnum group : ConfigGroupEnum.values()) {
            ConfigData<?> cacheConfig = factory.cacheConfigData(group);
            if (cacheConfig != null) {
                String value = String.join(",", cacheConfig.getMd5(), String.valueOf(cacheConfig.getLastModifyTime()));
                params.put(group.name(), Lists.newArrayList(value));
            }
        }
        // Assemble request header and request body
        HttpHeaders headers = new HttpHeaders();
        headers.setContentType(MediaType.APPLICATION_FORM_URLENCODED);
        HttpEntity httpEntity = new HttpEntity(params, headers);
        String listenerUrl = server + "/configs/listener";
        log.debug("request listener configs: [{}]", listenerUrl);
        JsonArray groupJson = null;
        //Send a request to admin to judge whether the group data has changed
        //This is just to judge whether a group has changed
        try {
            String json = this.httpClient.postForEntity(listenerUrl, httpEntity, String.class).getBody();
            log.debug("listener result: [{}]", json);
            groupJson = GSON.fromJson(json, JsonObject.class).getAsJsonArray("data");
        } catch (RestClientException e) {
            String message = String.format("listener configs fail, server:[%s], %s", server, e.getMessage());
            throw new ShenyuException(message, e);
        }
        // Get the data according to the changed group
        /**
         * The official website explains this:
         * After receiving the response information, the gateway only knows which Group has changed its configuration, and needs to request the configuration data of the Group again.
         * There may be a question here: why not write out the changed data directly?
         * We also discussed this problem in depth during development, because http long polling mechanism can only ensure quasi real-time. If it is not processed in time at the gateway layer,
         * Or the administrator frequently updates the configuration, which is likely to miss the push of a configuration change. For security reasons, we only inform a Group that the information has changed.
         *
         * Personal understanding:
         * If the change data is written out directly, when the administrator frequently updates the configuration, it is updated for the first time, the client is removed from the blocking queue and the response information is returned to the gateway.
         * If the second update is made at this time, the current client is not in the blocking queue, so this change will be missed.
         * The same is true if the gateway layer does not handle it in time.
         * This is a long polling, a gateway and a synchronization thread. There may be a time-consuming process.
         * If the admin has data changes and the current gateway client is not in the blocking queue, the data will not be found.
         */
        if (groupJson != null) {
            // fetch group configuration async.
            ConfigGroupEnum[] changedGroups = GSON.fromJson(groupJson, ConfigGroupEnum[].class);
            if (ArrayUtils.isNotEmpty(changedGroups)) {
                log.info("Group config changed: {}", Arrays.toString(changedGroups));
                // Take the initiative to obtain the changed data from admin, and take the full amount of data according to different groups
                this.doFetchGroupConfig(server, changedGroups);
            }
        }
    }

One thing that needs to be explained here is: why not get the changed data directly in the long polling task? Instead, first judge which group data has changed, and then request admin again to obtain the changed data?

The official website explains this:

After receiving the response information, the gateway only knows which Group has changed its configuration, and needs to request the configuration data of the Group again.
There may be a question here: why not write out the changed data directly?
We also discussed this problem in depth during development, because http long polling mechanism can only ensure quasi real-time. If it is not processed in time at the gateway layer,
Or the administrator frequently updates the configuration, which is likely to miss the push of a configuration change. For security reasons, we only inform a Group that the information has changed.

Personal understanding is:

If the change data is written out directly, when the administrator frequently updates the configuration, it is updated for the first time, removes the client from the blocking queue and returns the response information to the gateway. If the second update is made at this time, the current client is not in the blocking queue, so this change will be missed. The same is true if the gateway layer does not handle it in time. This is a long polling, a gateway and a synchronization thread. There may be a time-consuming process. If the admin has data changes and the current gateway client is not in the blocking queue, the data will not be found.

We haven't analyzed the processing logic of the admin side yet. Let's talk about it first. On the admin side, the gateway client will be placed in the blocking queue. If there is a data change, the gateway client will leave the queue and send the change data. Therefore, if the gateway client is not blocking the queue when there are data changes, the currently changed data cannot be obtained.

When you know which group data has changed, take the initiative to obtain the changed data from admin. Take the full amount of data according to different groups. The calling method is doFetchGroupConfig(), which has been analyzed earlier.

After analysis, the data synchronization operation at the gateway end is completed. The long polling task is to constantly send a request to the admin to see if the data has changed. If there is a change in the packet data, then take the initiative to send a request to the admin to obtain the changed data, and then update the data in the gateway memory.

Gateway end long polling task flow:

3. admin data synchronization

From the previous analysis, we can see that the gateway side mainly calls two interfaces of admin:

  • /configs/listener: judge whether the group data has changed;
  • /Configurations / fetch: get change group data.

If we analyze these two interfaces directly, some places may be difficult to understand, so we'd better start from the admin startup process to analyze the data synchronization process.

3.1 loading configuration

If the following configuration is made in the configuration file application.yml, it means that data synchronization is carried out through http long polling.

shenyu:
  sync:
      http:
        enabled: true

When the program starts, the configuration loading of data synchronization class is realized through springboot conditional assembly. In this process, an HttpLongPollingDataChangedListener will be created to handle the implementation logic related to long polling.

/**
 * Data synchronization configuration class
 * Realized by spring boot conditional assembly
 * The type Data sync configuration.
 */
@Configuration
public class DataSyncConfiguration {

    /**
     * http Long polling
     * http long polling.
     */
    @Configuration
    @ConditionalOnProperty(name = "shenyu.sync.http.enabled", havingValue = "true")
    @EnableConfigurationProperties(HttpSyncProperties.class)
    static class HttpLongPollingListener {

        @Bean
        @ConditionalOnMissingBean(HttpLongPollingDataChangedListener.class)
        public HttpLongPollingDataChangedListener httpLongPollingDataChangedListener(final HttpSyncProperties httpSyncProperties) {
            return new HttpLongPollingDataChangedListener(httpSyncProperties);
        }
    }
}

3.2 instantiation of data change listener

  • HttpLongPollingDataChangedListener

The data change listener completes the instantiation and initialization operations through the constructor. A blocking queue will be created in the constructor to store the client; Create a thread pool to execute delayed tasks and periodic tasks; Save attribute information related to long polling.

    public HttpLongPollingDataChangedListener(final HttpSyncProperties httpSyncProperties) {
        // The default client (here is the gateway) is 1024
        this.clients = new ArrayBlockingQueue<>(1024);
        // Create thread pool
        // ScheduledThreadPoolExecutor can execute delayed tasks, periodic tasks and ordinary tasks
        this.scheduler = new ScheduledThreadPoolExecutor(1,
                ShenyuThreadFactory.create("long-polling", true));
        // Attribute information for long polling
        this.httpSyncProperties = httpSyncProperties;
    }

In addition, its class diagram relationship is as follows:

The InitializingBean interface is implemented, so the afterInitialize() method is executed during the initialization of the bean. Execute periodic tasks through the thread pool: update the data in memory (CACHE) and execute it every 5 minutes, and start executing after 5 minutes. Refreshing the local cache is to read data from the database to the local cache (here is memory), which is completed through refreshLocalCache().

    /**
     * It is called in the afterpropertieset () method in the InitializingBean interface, that is, it is executed during the initialization of the bean
     */
    @Override
    protected void afterInitialize() {
        long syncInterval = httpSyncProperties.getRefreshInterval().toMillis();
        // Periodically check the data for changes and update the cache

        // Execution cycle task: update the data in memory (CACHE), execute it every 5 minutes, and start executing after 5 minutes
        // Prevent admin from generating data after starting for a period of time; Then, when the gateway connects for the first time, it does not get the full amount of data
        scheduler.scheduleWithFixedDelay(() -> {
            log.info("http sync strategy refresh config start.");
            try {
                // Read data from the database to the local cache (here is memory)
                this.refreshLocalCache();
                log.info("http sync strategy refresh config success.");
            } catch (Exception e) {
                log.error("http sync strategy refresh config error!", e);
            }
        }, syncInterval, syncInterval, TimeUnit.MILLISECONDS);
        log.info("http sync strategy refresh interval: {}ms", syncInterval);
    }
  • refreshLocalCache()

Update the five data types respectively.

    // Read data from the database to the local cache (here is memory)
    private void refreshLocalCache() {
        //Update authentication data
        this.updateAppAuthCache();
        //Update plug-in data
        this.updatePluginCache();
        //Update rule data
        this.updateRuleCache();
        //Update selector data
        this.updateSelectorCache();
        //Update Metadata 
        this.updateMetaDataCache();
    }

The logic of the five update methods is similar. Call the service method to obtain the data, and then put it into the memory CACHE. Take the update rule data method updateRuleCache() as an example, pass in the rule enumeration type, and call ruleService.listAll() to obtain all rule data from the database.

    /**
     * Update rule cache.
     */
    protected void updateRuleCache() {
        this.updateCache(ConfigGroupEnum.RULE, ruleService.listAll());
    }
  • updateCache()

Update the data in memory with the data in the database.

// Map of cached data
protected static final ConcurrentMap<String, ConfigDataCache> CACHE = new ConcurrentHashMap<>();

/**
     * if md5 is not the same as the original, then update lcoal cache.
     * Update data in cache
     * @param group ConfigGroupEnum
     * @param <T> the type of class
     * @param data the new config data
     */
    protected <T> void updateCache(final ConfigGroupEnum group, final List<T> data) {
        //Data serialization
        String json = GsonUtils.getInstance().toJson(data);
        //Incoming md5 value and modification time
        ConfigDataCache newVal = new ConfigDataCache(group.name(), json, Md5Utils.md5(json), System.currentTimeMillis());
        //Update grouping data
        ConfigDataCache oldVal = CACHE.put(newVal.getGroup(), newVal);
        log.info("update config cache[{}], old: {}, updated: {}", group, oldVal, newVal);
    }

The initialization process is to start periodic tasks, regularly obtain data from the database and update memory data.

Next, start to analyze the two interfaces:

  • /configs/listener: judge whether the group data has changed;
  • /Configurations / fetch: get change group data.

3.3 data change polling interface

  • /configs/listener: judge whether the group data has changed;

The interface class is ConfigController, which takes effect only when data synchronization is performed using http long polling. The interface method listener() has no other logic and directly calls doLongPolling() method.

   
/**
 * This Controller only when HttpLongPollingDataChangedListener exist, will take effect.
 */
@ConditionalOnBean(HttpLongPollingDataChangedListener.class)
@RestController
@RequestMapping("/configs")
@Slf4j
public class ConfigController {

    @Resource
    private HttpLongPollingDataChangedListener longPollingListener;
    
    // Omit other logic

    /**
     * Listener.
     * Monitor data changes and execute long polling
     * @param request  the request
     * @param response the response
     */
    @PostMapping(value = "/listener")
    public void listener(final HttpServletRequest request, final HttpServletResponse response) {
        longPollingListener.doLongPolling(request, response);
    }

}
  • HttpLongPollingDataChangedListener#doLongPolling()

Execute long polling task: if there is any data change, it will immediately respond to the client (here is the gateway). Otherwise, the client will be blocked until there is a data change or timeout.

/**
     * Execute long polling: if there is any data change, it will immediately respond to the client (here is the gateway).
     * Otherwise, the client will be blocked until there is a data change or timeout.
     * @param request
     * @param response
     */
    public void doLongPolling(final HttpServletRequest request, final HttpServletResponse response) {
        // compare group md5
        // Compare md5, judge whether the gateway data is consistent with the admin data, and get the changed data group
        List<ConfigGroupEnum> changedGroup = compareChangedGroup(request);
        String clientIp = getRemoteIp(request);
        // response immediately.
        // If there is any changed data, respond to the gateway immediately
        if (CollectionUtils.isNotEmpty(changedGroup)) {
            this.generateResponse(response, changedGroup);
            log.info("send response with the changed group, ip={}, group={}", clientIp, changedGroup);
            return;
        }

         // If there is no change, put the client (here is the gateway) into the blocking queue
        // listen for configuration changed.
        final AsyncContext asyncContext = request.startAsync();
        // AsyncContext.settimeout() does not timeout properly, so you have to control it yourself
        asyncContext.setTimeout(0L);
        // block client's thread.
        scheduler.execute(new LongPollingClient(asyncContext, clientIp, HttpConstants.SERVER_MAX_HOLD_TIMEOUT));
    }

  • HttpLongPollingDataChangedListener#compareChangedGroup()

Judge whether the group data has changed. The judgment logic is to compare the md5 value and lastModifyTime of gateway end and admin end.

  • If the md5 value is different, it needs to be updated;
  • If the lastModifyTime on the admin side is greater than the lastModifyTime on the gateway side, it needs to be updated.
 /**
     * Judge whether the group data has changed
     * @param request
     * @return
     */
    private List<ConfigGroupEnum> compareChangedGroup(final HttpServletRequest request) {
        List<ConfigGroupEnum> changedGroup = new ArrayList<>(ConfigGroupEnum.values().length);
        for (ConfigGroupEnum group : ConfigGroupEnum.values()) {
            // md5 value and lastModifyTime of gateway side data
            String[] params = StringUtils.split(request.getParameter(group.name()), ',');
            if (params == null || params.length != 2) {
                throw new ShenyuException("group param invalid:" + request.getParameter(group.name()));
            }
            String clientMd5 = params[0];
            long clientModifyTime = NumberUtils.toLong(params[1]);
            ConfigDataCache serverCache = CACHE.get(group.name());
            // do check. Judge whether the group data has changed
            if (this.checkCacheDelayAndUpdate(serverCache, clientMd5, clientModifyTime)) {
                changedGroup.add(group);
            }
        }
        return changedGroup;
    }
  • LongPollingClient

If the data is not changed, put the client (here is the gateway) into the blocking queue. The blocking time is 60 seconds, that is, it is removed after 60 seconds and responds to the client.

class LongPollingClient implements Runnable {
      // Other logic is omitted
    
        @Override
        public void run() {
            try {
                // Remove after 60 seconds and respond to the client
                this.asyncTimeoutFuture = scheduler.schedule(() -> {
                    clients.remove(LongPollingClient.this);
                    List<ConfigGroupEnum> changedGroups = compareChangedGroup((HttpServletRequest) asyncContext.getRequest());
                    sendResponse(changedGroups);
                }, timeoutTime, TimeUnit.MILLISECONDS);

                // Add to blocking queue
                clients.add(this);

            } catch (Exception ex) {
                log.error("add long polling client error", ex);
            }
        }

        /**
         * Send response.
         *
         * @param changedGroups the changed groups
         */
        void sendResponse(final List<ConfigGroupEnum> changedGroups) {
            // cancel scheduler
            if (null != asyncTimeoutFuture) {
                asyncTimeoutFuture.cancel(false);
            }
            // Groups responding to changes
            generateResponse((HttpServletResponse) asyncContext.getResponse(), changedGroups);
            asyncContext.complete();
        }
    }

3.4 interface for obtaining change data

  • /configs/fetch: obtain change data;

Obtain packet data and return results according to the parameters passed in by the gateway. The main implementation method is longPollingListener.fetchConfig().

@ConditionalOnBean(HttpLongPollingDataChangedListener.class)
@RestController
@RequestMapping("/configs")
@Slf4j
public class ConfigController {

    @Resource
    private HttpLongPollingDataChangedListener longPollingListener;

    /**
     * Fetch configs shenyu result.
     * Full access to packet data
     * @param groupKeys the group keys
     * @return the shenyu result
     */
    @GetMapping("/fetch")
    public ShenyuAdminResult fetchConfigs(@NotNull final String[] groupKeys) {
        Map<String, ConfigData<?>> result = Maps.newHashMap();
        for (String groupKey : groupKeys) {
            ConfigData<?> data = longPollingListener.fetchConfig(ConfigGroupEnum.valueOf(groupKey));
            result.put(groupKey, data);
        }
        return ShenyuAdminResult.success(ShenyuResultMessage.SUCCESS, result);
    }
    
  // Other interfaces are omitted

}
  • AbstractDataChangedListener#fetchConfig()

Data acquisition is directly obtained from CACHE, and then matched and encapsulated according to different packet types.

    /**
     * fetch configuration from cache.
     * Obtain the full amount of data under the group
     * @param groupKey the group key
     * @return the configuration data
     */
    public ConfigData<?> fetchConfig(final ConfigGroupEnum groupKey) {
        // Get data directly from CACHE
        ConfigDataCache config = CACHE.get(groupKey.name()); 
        switch (groupKey) {
            case APP_AUTH: // Authentication data
                List<AppAuthData> appAuthList = GsonUtils.getGson().fromJson(config.getJson(), new TypeToken<List<AppAuthData>>() {
                }.getType());
                return new ConfigData<>(config.getMd5(), config.getLastModifyTime(), appAuthList);
            case PLUGIN: // Plug in data
                List<PluginData> pluginList = GsonUtils.getGson().fromJson(config.getJson(), new TypeToken<List<PluginData>>() {
                }.getType());
                return new ConfigData<>(config.getMd5(), config.getLastModifyTime(), pluginList);
            case RULE:   // Rule data
                List<RuleData> ruleList = GsonUtils.getGson().fromJson(config.getJson(), new TypeToken<List<RuleData>>() {
                }.getType());
                return new ConfigData<>(config.getMd5(), config.getLastModifyTime(), ruleList);
            case SELECTOR:  // Selector data
                List<SelectorData> selectorList = GsonUtils.getGson().fromJson(config.getJson(), new TypeToken<List<SelectorData>>() {
                }.getType());
                return new ConfigData<>(config.getMd5(), config.getLastModifyTime(), selectorList);
            case META_DATA: // metadata
                List<MetaData> metaList = GsonUtils.getGson().fromJson(config.getJson(), new TypeToken<List<MetaData>>() {
                }.getType());
                return new ConfigData<>(config.getMd5(), config.getLastModifyTime(), metaList);
            default:  // Other types, throw exception
                throw new IllegalStateException("Unexpected groupKey: " + groupKey);
        }

3.5 data change

In the previous article on the source code analysis of websocket data synchronization and zookeeper data synchronization, we know that the design structure of admin data synchronization is as follows:

Various data change listeners are subclasses of DataChangedListener.

After modifying the data on the admin side, send an event notification through the event processing mechanism of Spring. The sending logic is as follows:

/**
 * Event forwarders, which forward the changed events to each ConfigEventListener.
 * Data change event distributor: when the data on the admin side changes, the changed data is synchronized to the ShenYu gateway
 * Data changes depend on Spring's event listening mechanism: applicationeventpublisher -- > applicationevent -- > applicationlistener
 *
 */
@Component
public class DataChangedEventDispatcher implements ApplicationListener<DataChangedEvent>, InitializingBean {

   //Other logic is omitted

    /**
     * This method is called when there are data changes
     * @param event
     */
    @Override
    @SuppressWarnings("unchecked")
    public void onApplicationEvent(final DataChangedEvent event) {
        // Traverse the data change listener (generally, it is better to use a data synchronization method)
        for (DataChangedListener listener : listeners) {
            // What kind of data has changed
            switch (event.getGroupKey()) {
                case APP_AUTH: // Authentication information
                    listener.onAppAuthChanged((List<AppAuthData>) event.getSource(), event.getEventType());
                    break;
                case PLUGIN:  // Plug in information
                    listener.onPluginChanged((List<PluginData>) event.getSource(), event.getEventType());
                    break;
                case RULE:    // Rule information
                    listener.onRuleChanged((List<RuleData>) event.getSource(), event.getEventType());
                    break;
                case SELECTOR:   // Selector information
                    listener.onSelectorChanged((List<SelectorData>) event.getSource(), event.getEventType());
                    break;
                case META_DATA:  // metadata
                    listener.onMetaDataChanged((List<MetaData>) event.getSource(), event.getEventType());
                    break;
                default:  // Other types, throw exception
                    throw new IllegalStateException("Unexpected value: " + event.getGroupKey());
            }
        }
    }
}

Assuming that the plug-in information is modified and the data is synchronized through http long polling, the actual call of listener.onPluginChanged() is org.apache.shenyu.admin.listener.AbstractDataChangedListener#onPluginChanged:

    /**
     * In the operation of admin, some plug-ins have been updated
     * @param changed   the changed
     * @param eventType the event type
     */
    @Override
    public void onPluginChanged(final List<PluginData> changed, final DataEventTypeEnum eventType) {
        if (CollectionUtils.isEmpty(changed)) {
            return;
        }
        // Update memory CACHE
        this.updatePluginCache();
        // Perform change tasks
        this.afterPluginChanged(changed, eventType);
    }

There are two processing operations: one is to update the memory CACHE, which has been analyzed earlier; The other is to execute change tasks in the process pool.

  • HttpLongPollingDataChangedListener#afterPluginChanged()
    @Override
    protected void afterPluginChanged(final List<PluginData> changed, final DataEventTypeEnum eventType) {
        // Execute in thread pool
        scheduler.execute(new DataChangeTask(ConfigGroupEnum.PLUGIN));
    }
  • DataChangeTask

Data change task: remove the clients in the blocking queue in turn, and send a response to notify the gateway that a group of data has changed.

class DataChangeTask implements Runnable {
		//Other logic is omitted 
        @Override
        public void run() {
            // If the number of clients in the blocking queue exceeds the given value of 100, it will be executed in batches
            if (clients.size() > httpSyncProperties.getNotifyBatchSize()) {
                List<LongPollingClient> targetClients = new ArrayList<>(clients.size());
                clients.drainTo(targetClients);
                List<List<LongPollingClient>> partitionClients = Lists.partition(targetClients, httpSyncProperties.getNotifyBatchSize());
               // Batch execution
                partitionClients.forEach(item -> scheduler.execute(() -> doRun(item)));
            } else {
                // Perform tasks
                doRun(clients);
            }
        }

        private void doRun(final Collection<LongPollingClient> clients) {
            // Notify all clients of data changes
            for (Iterator<LongPollingClient> iter = clients.iterator(); iter.hasNext();) {
                LongPollingClient client = iter.next();
                iter.remove();
                // Send response
                client.sendResponse(Collections.singletonList(groupKey));
                log.info("send response with the changed group,ip={}, group={}, changeTime={}", client.ip, groupKey, changeTime);
            }
        }
    }

At this point, the admin side data synchronization logic is analyzed. In the long polling data synchronization based on http, it mainly has three functions:

  • Provide data change monitoring interface;
  • Provide interface for obtaining change data;
  • When there is data change, remove the client in the blocking queue and respond to the result.

Finally, three diagrams are used to describe the process of long polling task at the admin end:

  • /Configure / listener data change listening interface:

  • /Configure / fetch get change data interface:

  • Update data and synchronize data in admin background management system:

4. Summary

This paper mainly analyzes the source code of http long polling data synchronization in ShenYu gateway. The main knowledge points involved are as follows:

  • http long polling is actively requested by the gateway and constantly requested by the admin;
  • The granularity of change data is group (authentication information, plug-ins, selectors, rules, metadata);
  • http long polling results only get the change group, and you need to initiate a request again to obtain group data;
  • Whether the data is updated is determined by the md5 value and the modification time lastModifyTime.

Tags: Java Apache http

Posted on Wed, 03 Nov 2021 01:22:50 -0400 by headmine