Apache ShenYu Is an asynchronous, high-performance, cross language, responsive API gateway.
In ShenYu gateway, data synchronization refers to how to synchronize the updated data to the gateway after the updated data is sent in the background management system. Apache ShenYu gateway currently supports ZooKeeper, WebSocket, Http long polling, Nacos, Etcd and Consul for data synchronization. The main content of this paper is the source code analysis of data synchronization based on Http long polling.
This paper analyzes the source code based on shenyu-2.4.0. Please refer to the introduction of the official website Data synchronization principle .
1. Http long polling
The relevant descriptions on the official website are directly quoted here:
The data synchronization mechanism of Zookeeper and WebSocket is relatively simple, while HTTP long polling is relatively complex. Apache ShenYu draws on the design idea of Apollo and Nacos, extracts its essence, and realizes the Http long polling data synchronization function. Note that this is not the traditional ajax long polling!
Http long polling mechanism is as shown above. Apache ShenYu gateway actively requests Shenyu admin configuration service, and the read timeout is 90s, which means that the gateway layer will wait up to 90s for requesting configuration service, so that Shenyu admin configuration service can respond to change data in time, so as to realize quasi real-time push.
The Http long polling mechanism is initiated by the gateway to request Shenyu admin, so we will start from the gateway side for this source code analysis.
2. Gateway data synchronization
2.1 loading configuration
The Http long polling data synchronization configuration is loaded through the starter mechanism of spring boot. It will be loaded when we introduce relevant dependencies and have the following configurations in the configuration file.
Introducing dependencies in pom files:
<!--shenyu data sync start use http-->
<dependency>
<groupId>org.apache.shenyu</groupId>
<artifactId>shenyu-spring-boot-starter-sync-data-http</artifactId>
<version>${project.version}</version>
</dependency>
Add a configuration in the application.yml configuration file:
shenyu:
sync:
http:
url : http://localhost:9095
When the gateway is started, the configuration class HttpSyncDataConfiguration will execute and load the corresponding Bean.
/**
* Http sync data configuration for spring boot.
*/
@Configuration
@ConditionalOnClass(HttpSyncDataService.class)
@ConditionalOnProperty(prefix = "shenyu.sync.http", name = "url")
@Slf4j
public class HttpSyncDataConfiguration {
/**
* Http sync data service.
* Create HttpSyncDataService
* @param httpConfig http Configuration of
* @param pluginSubscriber Plug in data subscription
* @param metaSubscribers Metadata subscription
* @param authSubscribers Authentication data subscription
* @return the sync data service
*/
@Bean
public SyncDataService httpSyncDataService(final ObjectProvider<HttpConfig> httpConfig, final ObjectProvider<PluginDataSubscriber> pluginSubscriber,
final ObjectProvider<List<MetaDataSubscriber>> metaSubscribers, final ObjectProvider<List<AuthDataSubscriber>> authSubscribers) {
log.info("you use http long pull sync shenyu data");
return new HttpSyncDataService(Objects.requireNonNull(httpConfig.getIfAvailable()), Objects.requireNonNull(pluginSubscriber.getIfAvailable()),
metaSubscribers.getIfAvailable(Collections::emptyList), authSubscribers.getIfAvailable(Collections::emptyList));
}
/**
* Http config http config.
* Read http configuration
* @return the http config
*/
@Bean
@ConfigurationProperties(prefix = "shenyu.sync.http")
public HttpConfig httpConfig() {
return new HttpConfig();
}
}
HttpSyncDataConfiguration is a configuration class for HTTP long polling data synchronization. It is responsible for creating HttpSyncDataService (responsible for the specific implementation of HTTP data synchronization) and HttpConfig (admin attribute configuration). Its notes are as follows:
- @Configuration: indicates that this is a configuration class;
- @ConditionalOnClass(HttpSyncDataService.class): condition annotation, indicating that the class HttpSyncDataService is required;
- @ConditionalOnProperty(prefix = "shenyu.sync.http", name = "url"): the condition annotation should be configured with the property shenyu.sync.http.url.
2.2 attribute initialization
- HttpSyncDataService
Complete property initialization in the constructor of HttpSyncDataService.
public class HttpSyncDataService implements SyncDataService, AutoCloseable {
// Omitted attribute field
public HttpSyncDataService(final HttpConfig httpConfig, final PluginDataSubscriber pluginDataSubscriber, final List<MetaDataSubscriber> metaDataSubscribers, final List<AuthDataSubscriber> authDataSubscribers) {
// 1. Create a data processor
this.factory = new DataRefreshFactory(pluginDataSubscriber, metaDataSubscribers, authDataSubscribers);
// 2. Get admin attribute configuration
this.httpConfig = httpConfig;
// The url of Shenyu admin. Multiple URLs are separated by commas (,)
this.serverList = Lists.newArrayList(Splitter.on(",").split(httpConfig.getUrl()));
// 3. Create an httpClient to initiate a request to admin
this.httpClient = createRestTemplate();
// 4. Start the long polling task
this.start();
}
//......
}
Other functions and related fields are omitted from the above code, and the initialization of attributes is completed in the constructor, mainly including:
-
Create a data processor for subsequent caching of various types of data (plug-ins, selectors, rules, metadata and authentication data);
-
Obtain the admin attribute configuration, mainly to obtain the url of admin. Admin may be a cluster, and multiple are separated by commas (,);
-
Create httpClient using RestTemplate, which is used to send a request to admin;
private RestTemplate createRestTemplate() { OkHttp3ClientHttpRequestFactory factory = new OkHttp3ClientHttpRequestFactory(); // The connection establishment timeout is 10s factory.setConnectTimeout((int) this.connectionTimeout.toMillis()); // The gateway actively requests the configuration service of Shenyu admin, and the read timeout is 90s factory.setReadTimeout((int) HttpConstants.CLIENT_POLLING_READ_TIMEOUT); return new RestTemplate(factory); } -
Start long polling task.
2.3 start long polling
- HttpSyncDataService#start()
In the start() method, two things are done. One is to obtain the full amount of data, that is, request the admin side to obtain all the data to be synchronized, and then cache the obtained data into the gateway memory. The other is to enable multithreading to perform long polling tasks.
private void start() {
// It is initialized only once and implemented through atomic classes.
RUNNING = new AtomicBoolean(false);
// It could be initialized multiple times, so you need to control that.
if (RUNNING.compareAndSet(false, true)) {
// fetch all group configs.
// Initial startup to obtain full data
this.fetchGroupConfig(ConfigGroupEnum.values());
// A background service, a thread
int threadSize = serverList.size();
// Custom thread pool
this.executor = new ThreadPoolExecutor(threadSize, threadSize, 60L, TimeUnit.SECONDS,
new LinkedBlockingQueue<>(),
ShenyuThreadFactory.create("http-long-polling", true));
// start long polling, each server creates a thread to listen for changes.
// Start long polling, an admin service, and create a thread for data synchronization
this.serverList.forEach(server -> this.executor.execute(new HttpLongPollingTask(server)));
} else {
log.info("shenyu http long polling was started, executor=[{}]", executor);
}
}
2.3.1 obtaining full data
- HttpSyncDataService#fetchGroupConfig()
ShenYu grouped all data to be synchronized. There are five data types: plug-in, selector, rule, metadata and authentication data.
public enum ConfigGroupEnum {
APP_AUTH, // Authentication data
PLUGIN, //plug-in unit
RULE, // rule
SELECTOR, // selector
META_DATA; // metadata
}
Admin may be a cluster. Here, requests are sent to each admin in a circular manner. If one of them is executed successfully, the operation of obtaining the full amount of data from admin and caching it to the gateway will be executed successfully. If an exception occurs, a request is made to the next admin.
private void fetchGroupConfig(final ConfigGroupEnum... groups) throws ShenyuException {
// Admin may be a cluster, where requests are sent to each admin in a circular manner
for (int index = 0; index < this.serverList.size(); index++) {
String server = serverList.get(index);
try {
// Really implement
this.doFetchGroupConfig(server, groups);
// If one is successful, it will be successful. You can exit the cycle
break;
} catch (ShenyuException e) {
// An exception occurred, trying to execute the next
// The last one also fails to execute and throws an exception
// no available server, throw exception.
if (index >= serverList.size() - 1) {
throw e;
}
log.warn("fetch config fail, try another one: {}", serverList.get(index + 1));
}
}
}
- HttpSyncDataService#doFetchGroupConfig()
In this method, first assemble the request parameters, then initiate the request through httpClient, obtain the data from admin, and finally update the obtained data to the gateway memory.
// Send a request to the admin background management system to obtain all synchronous data
private void doFetchGroupConfig(final String server, final ConfigGroupEnum... groups) {
// 1. Spell request parameters, all grouping enumeration types
StringBuilder params = new StringBuilder();
for (ConfigGroupEnum groupKey : groups) {
params.append("groupKeys").append("=").append(groupKey.name()).append("&");
}
// Interface provided by admin side / configs/fetch
String url = server + "/configs/fetch?" + StringUtils.removeEnd(params.toString(), "&");
log.info("request configs: [{}]", url);
String json = null;
try {
// 2. Initiate a request to obtain change data
json = this.httpClient.getForObject(url, String.class);
} catch (RestClientException e) {
String message = String.format("fetch config fail from server[%s], %s", url, e.getMessage());
log.warn(message);
throw new ShenyuException(message, e);
}
// update local cache
// 3. Update data in gateway memory
boolean updated = this.updateCacheWithJson(json);
// If the update is successful, this method is completed
if (updated) {
log.info("get latest configs: [{}]", json);
return;
}
// not updated. it is likely that the current config server has not been updated yet. wait a moment.
log.info("The config of the server[{}] has not been updated or is out of date. Wait for 30s to listen for changes again.", server);
// The server doesn't update the data, just wait for 30s
ThreadUtils.sleep(TimeUnit.SECONDS, 30);
}
From the code, you can see that the interface for obtaining full data provided by the admin side is / configs/fetch. I won't go further here, but I'll analyze it later.
If the result data returned by admin is obtained and successfully updated, the execution of this method ends. If the update is not successful, the server may not update the data and wait for 30s.
It needs to be explained in advance. When the gateway judges whether the update is successful, it has the operation of comparing data, which will be mentioned soon.
- HttpSyncDataService#updateCacheWithJson
Update data in gateway memory. Use GSON for deserialization, get the real data from the attribute data, and then give it to DataRefreshFactory for update.
private boolean updateCacheWithJson(final String json) {
// Deserialization using GSON
JsonObject jsonObject = GSON.fromJson(json, JsonObject.class);
JsonObject data = jsonObject.getAsJsonObject("data");
// if the config cache will be updated?
return factory.executor(data);
}
- DataRefreshFactory#executor()
Update data according to different data types and return update results. parallelStream() is used for parallel update, and the specific update logic is handed over to the dataRefresh.refresh() method. In the update result, one data type has been updated, which indicates that the operation has been updated.
public boolean executor(final JsonObject data) {
//Parallel update data
List<Boolean> result = ENUM_MAP.values().parallelStream()
.map(dataRefresh -> dataRefresh.refresh(data))
.collect(Collectors.toList());
//An update indicates that an update operation has occurred this time
return result.stream().anyMatch(Boolean.TRUE::equals);
}
- AbstractDataRefresh#refresh()
The data update logic adopts the template method design pattern, the general operation is completed in the abstract method, and the different implementation logic is completed by the subclass. There are some differences in the specific update logic of the five data types, but there are also general update logic. The class diagram relationship is as follows:
In the general refresh() method, it is responsible for data type conversion, judging whether it needs to be updated, and the actual data refresh operation.
@Override
public Boolean refresh(final JsonObject data) {
boolean updated = false;
// Data type conversion
JsonObject jsonObject = convert(data);
if (null != jsonObject) {
// Get data type
ConfigData<T> result = fromJson(jsonObject);
// Update required
if (this.updateCacheIfNeed(result)) {
updated = true;
// Real update logic, data refresh operation
refresh(result.getData());
}
}
return updated;
}
- AbstractDataRefresh#updateCacheIfNeed()
The process of data conversion is to convert according to different data types. We won't track further to see whether the data needs to be updated. The method name is updateCacheIfNeed(), which is implemented through method overloading.
// result is data
protected abstract boolean updateCacheIfNeed(ConfigData<T> result);
// newVal is the latest value obtained
// What data type is groupEnum
protected boolean updateCacheIfNeed(final ConfigData<T> newVal, final ConfigGroupEnum groupEnum) {
// If it is the first time, it is directly put into the cache and returns true, indicating that the update has been made this time
if (GROUP_CACHE.putIfAbsent(groupEnum, newVal) == null) {
return true;
}
ResultHolder holder = new ResultHolder(false);
GROUP_CACHE.merge(groupEnum, newVal, (oldVal, value) -> {
// The md5 value is the same and does not need to be updated
if (StringUtils.equals(oldVal.getMd5(), newVal.getMd5())) {
log.info("Get the same config, the [{}] config cache will not be updated, md5:{}", groupEnum, oldVal.getMd5());
return oldVal;
}
// The modification time of the current cached data is longer than that of the new data, so it does not need to be updated
// must compare the last update time
if (oldVal.getLastModifyTime() >= newVal.getLastModifyTime()) {
log.info("Last update time earlier than the current configuration, the [{}] config cache will not be updated", groupEnum);
return oldVal;
}
log.info("update {} config: {}", groupEnum, newVal);
holder.result = true;
return newVal;
});
return holder.result;
}
As can be seen from the above source code, there are two situations that do not need to be updated:
- The md5 values of the two data are the same and do not need to be updated;
- The modification time of the currently cached data is greater than that of the new data, so it does not need to be updated.
Other situations require updating data.
After the analysis, the logic analysis of starting the start() method for the first time and obtaining the full amount of data is completed, followed by the operation of long polling. For convenience, I paste the start() method again:
private void start() {
// It could be initialized multiple times, so you need to control that.
if (RUNNING.compareAndSet(false, true)) {
// fetch all group configs.
// Initial startup to obtain full data
this.fetchGroupConfig(ConfigGroupEnum.values());
// A background service, a thread
int threadSize = serverList.size();
// Custom thread pool
this.executor = new ThreadPoolExecutor(threadSize, threadSize, 60L, TimeUnit.SECONDS,
new LinkedBlockingQueue<>(),
ShenyuThreadFactory.create("http-long-polling", true));
// start long polling, each server creates a thread to listen for changes.
// Start long polling, an admin service, and create a thread for data synchronization
this.serverList.forEach(server -> this.executor.execute(new HttpLongPollingTask(server)));
} else {
log.info("shenyu http long polling was started, executor=[{}]", executor);
}
}
2.3.2 execute long polling task
- HttpLongPollingTask#run()
The long polling task is HttpLongPollingTask, which implements the Runnable interface, and the task logic is in the run() method. Through the while() loop, the task is executed continuously, that is, long polling. There are three retry logics in each polling. One polling task fails. Wait for 5s to continue. All three times fail. Wait for 5 minutes to try again.
Start long polling, an admin service, and create a thread for data synchronization.
class HttpLongPollingTask implements Runnable {
private String server;
// Retry 3 times by default
private final int retryTimes = 3;
HttpLongPollingTask(final String server) {
this.server = server;
}
@Override
public void run() {
// Always polling
while (RUNNING.get()) {
for (int time = 1; time <= retryTimes; time++) {
try {
doLongPolling(server);
} catch (Exception e) {
// print warnning log.
if (time < retryTimes) {
log.warn("Long polling failed, tried {} times, {} times left, will be suspended for a while! {}",
time, retryTimes - time, e.getMessage());
// Long polling failed. Wait for 5s to continue
ThreadUtils.sleep(TimeUnit.SECONDS, 5);
continue;
}
// print error, then suspended for a while.
log.error("Long polling failed, try again after 5 minutes!", e);
// All three times failed. Try again in 5 minutes
ThreadUtils.sleep(TimeUnit.MINUTES, 5);
}
}
}
log.warn("Stop http long polling.");
}
}
- HttpSyncDataService#doLongPolling()
Core logic for executing long polling tasks:
- Assemble request parameters according to data type: md5 and lastModifyTime;
- Assembling a request header and a request body;
- Send a request to admin to judge whether the group data has changed;
- Get the data according to the changed group.
private void doLongPolling(final String server) {
// Assembly request parameters: md5 and lastModifyTime
MultiValueMap<String, String> params = new LinkedMultiValueMap<>(8);
for (ConfigGroupEnum group : ConfigGroupEnum.values()) {
ConfigData<?> cacheConfig = factory.cacheConfigData(group);
if (cacheConfig != null) {
String value = String.join(",", cacheConfig.getMd5(), String.valueOf(cacheConfig.getLastModifyTime()));
params.put(group.name(), Lists.newArrayList(value));
}
}
// Assemble request header and request body
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_FORM_URLENCODED);
HttpEntity httpEntity = new HttpEntity(params, headers);
String listenerUrl = server + "/configs/listener";
log.debug("request listener configs: [{}]", listenerUrl);
JsonArray groupJson = null;
//Send a request to admin to judge whether the group data has changed
//This is just to judge whether a group has changed
try {
String json = this.httpClient.postForEntity(listenerUrl, httpEntity, String.class).getBody();
log.debug("listener result: [{}]", json);
groupJson = GSON.fromJson(json, JsonObject.class).getAsJsonArray("data");
} catch (RestClientException e) {
String message = String.format("listener configs fail, server:[%s], %s", server, e.getMessage());
throw new ShenyuException(message, e);
}
// Get the data according to the changed group
/**
* The official website explains this:
* After receiving the response information, the gateway only knows which Group has changed its configuration, and needs to request the configuration data of the Group again.
* There may be a question here: why not write out the changed data directly?
* We also discussed this problem in depth during development, because http long polling mechanism can only ensure quasi real-time. If it is not processed in time at the gateway layer,
* Or the administrator frequently updates the configuration, which is likely to miss the push of a configuration change. For security reasons, we only inform a Group that the information has changed.
*
* Personal understanding:
* If the change data is written out directly, when the administrator frequently updates the configuration, it is updated for the first time, the client is removed from the blocking queue and the response information is returned to the gateway.
* If the second update is made at this time, the current client is not in the blocking queue, so this change will be missed.
* The same is true if the gateway layer does not handle it in time.
* This is a long polling, a gateway and a synchronization thread. There may be a time-consuming process.
* If the admin has data changes and the current gateway client is not in the blocking queue, the data will not be found.
*/
if (groupJson != null) {
// fetch group configuration async.
ConfigGroupEnum[] changedGroups = GSON.fromJson(groupJson, ConfigGroupEnum[].class);
if (ArrayUtils.isNotEmpty(changedGroups)) {
log.info("Group config changed: {}", Arrays.toString(changedGroups));
// Take the initiative to obtain the changed data from admin, and take the full amount of data according to different groups
this.doFetchGroupConfig(server, changedGroups);
}
}
}
One thing that needs to be explained here is: why not get the changed data directly in the long polling task? Instead, first judge which group data has changed, and then request admin again to obtain the changed data?
The official website explains this:
After receiving the response information, the gateway only knows which Group has changed its configuration, and needs to request the configuration data of the Group again.
There may be a question here: why not write out the changed data directly?
We also discussed this problem in depth during development, because http long polling mechanism can only ensure quasi real-time. If it is not processed in time at the gateway layer,
Or the administrator frequently updates the configuration, which is likely to miss the push of a configuration change. For security reasons, we only inform a Group that the information has changed.
Personal understanding is:
If the change data is written out directly, when the administrator frequently updates the configuration, it is updated for the first time, removes the client from the blocking queue and returns the response information to the gateway. If the second update is made at this time, the current client is not in the blocking queue, so this change will be missed. The same is true if the gateway layer does not handle it in time. This is a long polling, a gateway and a synchronization thread. There may be a time-consuming process. If the admin has data changes and the current gateway client is not in the blocking queue, the data will not be found.
We haven't analyzed the processing logic of the admin side yet. Let's talk about it first. On the admin side, the gateway client will be placed in the blocking queue. If there is a data change, the gateway client will leave the queue and send the change data. Therefore, if the gateway client is not blocking the queue when there are data changes, the currently changed data cannot be obtained.
When you know which group data has changed, take the initiative to obtain the changed data from admin. Take the full amount of data according to different groups. The calling method is doFetchGroupConfig(), which has been analyzed earlier.
After analysis, the data synchronization operation at the gateway end is completed. The long polling task is to constantly send a request to the admin to see if the data has changed. If there is a change in the packet data, then take the initiative to send a request to the admin to obtain the changed data, and then update the data in the gateway memory.
Gateway end long polling task flow:
3. admin data synchronization
From the previous analysis, we can see that the gateway side mainly calls two interfaces of admin:
- /configs/listener: judge whether the group data has changed;
- /Configurations / fetch: get change group data.
If we analyze these two interfaces directly, some places may be difficult to understand, so we'd better start from the admin startup process to analyze the data synchronization process.
3.1 loading configuration
If the following configuration is made in the configuration file application.yml, it means that data synchronization is carried out through http long polling.
shenyu:
sync:
http:
enabled: true
When the program starts, the configuration loading of data synchronization class is realized through springboot conditional assembly. In this process, an HttpLongPollingDataChangedListener will be created to handle the implementation logic related to long polling.
/**
* Data synchronization configuration class
* Realized by spring boot conditional assembly
* The type Data sync configuration.
*/
@Configuration
public class DataSyncConfiguration {
/**
* http Long polling
* http long polling.
*/
@Configuration
@ConditionalOnProperty(name = "shenyu.sync.http.enabled", havingValue = "true")
@EnableConfigurationProperties(HttpSyncProperties.class)
static class HttpLongPollingListener {
@Bean
@ConditionalOnMissingBean(HttpLongPollingDataChangedListener.class)
public HttpLongPollingDataChangedListener httpLongPollingDataChangedListener(final HttpSyncProperties httpSyncProperties) {
return new HttpLongPollingDataChangedListener(httpSyncProperties);
}
}
}
3.2 instantiation of data change listener
- HttpLongPollingDataChangedListener
The data change listener completes the instantiation and initialization operations through the constructor. A blocking queue will be created in the constructor to store the client; Create a thread pool to execute delayed tasks and periodic tasks; Save attribute information related to long polling.
public HttpLongPollingDataChangedListener(final HttpSyncProperties httpSyncProperties) {
// The default client (here is the gateway) is 1024
this.clients = new ArrayBlockingQueue<>(1024);
// Create thread pool
// ScheduledThreadPoolExecutor can execute delayed tasks, periodic tasks and ordinary tasks
this.scheduler = new ScheduledThreadPoolExecutor(1,
ShenyuThreadFactory.create("long-polling", true));
// Attribute information for long polling
this.httpSyncProperties = httpSyncProperties;
}
In addition, its class diagram relationship is as follows:
The InitializingBean interface is implemented, so the afterInitialize() method is executed during the initialization of the bean. Execute periodic tasks through the thread pool: update the data in memory (CACHE) and execute it every 5 minutes, and start executing after 5 minutes. Refreshing the local cache is to read data from the database to the local cache (here is memory), which is completed through refreshLocalCache().
/**
* It is called in the afterpropertieset () method in the InitializingBean interface, that is, it is executed during the initialization of the bean
*/
@Override
protected void afterInitialize() {
long syncInterval = httpSyncProperties.getRefreshInterval().toMillis();
// Periodically check the data for changes and update the cache
// Execution cycle task: update the data in memory (CACHE), execute it every 5 minutes, and start executing after 5 minutes
// Prevent admin from generating data after starting for a period of time; Then, when the gateway connects for the first time, it does not get the full amount of data
scheduler.scheduleWithFixedDelay(() -> {
log.info("http sync strategy refresh config start.");
try {
// Read data from the database to the local cache (here is memory)
this.refreshLocalCache();
log.info("http sync strategy refresh config success.");
} catch (Exception e) {
log.error("http sync strategy refresh config error!", e);
}
}, syncInterval, syncInterval, TimeUnit.MILLISECONDS);
log.info("http sync strategy refresh interval: {}ms", syncInterval);
}
- refreshLocalCache()
Update the five data types respectively.
// Read data from the database to the local cache (here is memory)
private void refreshLocalCache() {
//Update authentication data
this.updateAppAuthCache();
//Update plug-in data
this.updatePluginCache();
//Update rule data
this.updateRuleCache();
//Update selector data
this.updateSelectorCache();
//Update Metadata
this.updateMetaDataCache();
}
The logic of the five update methods is similar. Call the service method to obtain the data, and then put it into the memory CACHE. Take the update rule data method updateRuleCache() as an example, pass in the rule enumeration type, and call ruleService.listAll() to obtain all rule data from the database.
/**
* Update rule cache.
*/
protected void updateRuleCache() {
this.updateCache(ConfigGroupEnum.RULE, ruleService.listAll());
}
- updateCache()
Update the data in memory with the data in the database.
// Map of cached data
protected static final ConcurrentMap<String, ConfigDataCache> CACHE = new ConcurrentHashMap<>();
/**
* if md5 is not the same as the original, then update lcoal cache.
* Update data in cache
* @param group ConfigGroupEnum
* @param <T> the type of class
* @param data the new config data
*/
protected <T> void updateCache(final ConfigGroupEnum group, final List<T> data) {
//Data serialization
String json = GsonUtils.getInstance().toJson(data);
//Incoming md5 value and modification time
ConfigDataCache newVal = new ConfigDataCache(group.name(), json, Md5Utils.md5(json), System.currentTimeMillis());
//Update grouping data
ConfigDataCache oldVal = CACHE.put(newVal.getGroup(), newVal);
log.info("update config cache[{}], old: {}, updated: {}", group, oldVal, newVal);
}
The initialization process is to start periodic tasks, regularly obtain data from the database and update memory data.
Next, start to analyze the two interfaces:
- /configs/listener: judge whether the group data has changed;
- /Configurations / fetch: get change group data.
3.3 data change polling interface
- /configs/listener: judge whether the group data has changed;
The interface class is ConfigController, which takes effect only when data synchronization is performed using http long polling. The interface method listener() has no other logic and directly calls doLongPolling() method.
/**
* This Controller only when HttpLongPollingDataChangedListener exist, will take effect.
*/
@ConditionalOnBean(HttpLongPollingDataChangedListener.class)
@RestController
@RequestMapping("/configs")
@Slf4j
public class ConfigController {
@Resource
private HttpLongPollingDataChangedListener longPollingListener;
// Omit other logic
/**
* Listener.
* Monitor data changes and execute long polling
* @param request the request
* @param response the response
*/
@PostMapping(value = "/listener")
public void listener(final HttpServletRequest request, final HttpServletResponse response) {
longPollingListener.doLongPolling(request, response);
}
}
- HttpLongPollingDataChangedListener#doLongPolling()
Execute long polling task: if there is any data change, it will immediately respond to the client (here is the gateway). Otherwise, the client will be blocked until there is a data change or timeout.
/**
* Execute long polling: if there is any data change, it will immediately respond to the client (here is the gateway).
* Otherwise, the client will be blocked until there is a data change or timeout.
* @param request
* @param response
*/
public void doLongPolling(final HttpServletRequest request, final HttpServletResponse response) {
// compare group md5
// Compare md5, judge whether the gateway data is consistent with the admin data, and get the changed data group
List<ConfigGroupEnum> changedGroup = compareChangedGroup(request);
String clientIp = getRemoteIp(request);
// response immediately.
// If there is any changed data, respond to the gateway immediately
if (CollectionUtils.isNotEmpty(changedGroup)) {
this.generateResponse(response, changedGroup);
log.info("send response with the changed group, ip={}, group={}", clientIp, changedGroup);
return;
}
// If there is no change, put the client (here is the gateway) into the blocking queue
// listen for configuration changed.
final AsyncContext asyncContext = request.startAsync();
// AsyncContext.settimeout() does not timeout properly, so you have to control it yourself
asyncContext.setTimeout(0L);
// block client's thread.
scheduler.execute(new LongPollingClient(asyncContext, clientIp, HttpConstants.SERVER_MAX_HOLD_TIMEOUT));
}
- HttpLongPollingDataChangedListener#compareChangedGroup()
Judge whether the group data has changed. The judgment logic is to compare the md5 value and lastModifyTime of gateway end and admin end.
- If the md5 value is different, it needs to be updated;
- If the lastModifyTime on the admin side is greater than the lastModifyTime on the gateway side, it needs to be updated.
/**
* Judge whether the group data has changed
* @param request
* @return
*/
private List<ConfigGroupEnum> compareChangedGroup(final HttpServletRequest request) {
List<ConfigGroupEnum> changedGroup = new ArrayList<>(ConfigGroupEnum.values().length);
for (ConfigGroupEnum group : ConfigGroupEnum.values()) {
// md5 value and lastModifyTime of gateway side data
String[] params = StringUtils.split(request.getParameter(group.name()), ',');
if (params == null || params.length != 2) {
throw new ShenyuException("group param invalid:" + request.getParameter(group.name()));
}
String clientMd5 = params[0];
long clientModifyTime = NumberUtils.toLong(params[1]);
ConfigDataCache serverCache = CACHE.get(group.name());
// do check. Judge whether the group data has changed
if (this.checkCacheDelayAndUpdate(serverCache, clientMd5, clientModifyTime)) {
changedGroup.add(group);
}
}
return changedGroup;
}
- LongPollingClient
If the data is not changed, put the client (here is the gateway) into the blocking queue. The blocking time is 60 seconds, that is, it is removed after 60 seconds and responds to the client.
class LongPollingClient implements Runnable {
// Other logic is omitted
@Override
public void run() {
try {
// Remove after 60 seconds and respond to the client
this.asyncTimeoutFuture = scheduler.schedule(() -> {
clients.remove(LongPollingClient.this);
List<ConfigGroupEnum> changedGroups = compareChangedGroup((HttpServletRequest) asyncContext.getRequest());
sendResponse(changedGroups);
}, timeoutTime, TimeUnit.MILLISECONDS);
// Add to blocking queue
clients.add(this);
} catch (Exception ex) {
log.error("add long polling client error", ex);
}
}
/**
* Send response.
*
* @param changedGroups the changed groups
*/
void sendResponse(final List<ConfigGroupEnum> changedGroups) {
// cancel scheduler
if (null != asyncTimeoutFuture) {
asyncTimeoutFuture.cancel(false);
}
// Groups responding to changes
generateResponse((HttpServletResponse) asyncContext.getResponse(), changedGroups);
asyncContext.complete();
}
}
3.4 interface for obtaining change data
- /configs/fetch: obtain change data;
Obtain packet data and return results according to the parameters passed in by the gateway. The main implementation method is longPollingListener.fetchConfig().
@ConditionalOnBean(HttpLongPollingDataChangedListener.class)
@RestController
@RequestMapping("/configs")
@Slf4j
public class ConfigController {
@Resource
private HttpLongPollingDataChangedListener longPollingListener;
/**
* Fetch configs shenyu result.
* Full access to packet data
* @param groupKeys the group keys
* @return the shenyu result
*/
@GetMapping("/fetch")
public ShenyuAdminResult fetchConfigs(@NotNull final String[] groupKeys) {
Map<String, ConfigData<?>> result = Maps.newHashMap();
for (String groupKey : groupKeys) {
ConfigData<?> data = longPollingListener.fetchConfig(ConfigGroupEnum.valueOf(groupKey));
result.put(groupKey, data);
}
return ShenyuAdminResult.success(ShenyuResultMessage.SUCCESS, result);
}
// Other interfaces are omitted
}
- AbstractDataChangedListener#fetchConfig()
Data acquisition is directly obtained from CACHE, and then matched and encapsulated according to different packet types.
/**
* fetch configuration from cache.
* Obtain the full amount of data under the group
* @param groupKey the group key
* @return the configuration data
*/
public ConfigData<?> fetchConfig(final ConfigGroupEnum groupKey) {
// Get data directly from CACHE
ConfigDataCache config = CACHE.get(groupKey.name());
switch (groupKey) {
case APP_AUTH: // Authentication data
List<AppAuthData> appAuthList = GsonUtils.getGson().fromJson(config.getJson(), new TypeToken<List<AppAuthData>>() {
}.getType());
return new ConfigData<>(config.getMd5(), config.getLastModifyTime(), appAuthList);
case PLUGIN: // Plug in data
List<PluginData> pluginList = GsonUtils.getGson().fromJson(config.getJson(), new TypeToken<List<PluginData>>() {
}.getType());
return new ConfigData<>(config.getMd5(), config.getLastModifyTime(), pluginList);
case RULE: // Rule data
List<RuleData> ruleList = GsonUtils.getGson().fromJson(config.getJson(), new TypeToken<List<RuleData>>() {
}.getType());
return new ConfigData<>(config.getMd5(), config.getLastModifyTime(), ruleList);
case SELECTOR: // Selector data
List<SelectorData> selectorList = GsonUtils.getGson().fromJson(config.getJson(), new TypeToken<List<SelectorData>>() {
}.getType());
return new ConfigData<>(config.getMd5(), config.getLastModifyTime(), selectorList);
case META_DATA: // metadata
List<MetaData> metaList = GsonUtils.getGson().fromJson(config.getJson(), new TypeToken<List<MetaData>>() {
}.getType());
return new ConfigData<>(config.getMd5(), config.getLastModifyTime(), metaList);
default: // Other types, throw exception
throw new IllegalStateException("Unexpected groupKey: " + groupKey);
}
3.5 data change
In the previous article on the source code analysis of websocket data synchronization and zookeeper data synchronization, we know that the design structure of admin data synchronization is as follows:
Various data change listeners are subclasses of DataChangedListener.
After modifying the data on the admin side, send an event notification through the event processing mechanism of Spring. The sending logic is as follows:
/**
* Event forwarders, which forward the changed events to each ConfigEventListener.
* Data change event distributor: when the data on the admin side changes, the changed data is synchronized to the ShenYu gateway
* Data changes depend on Spring's event listening mechanism: applicationeventpublisher -- > applicationevent -- > applicationlistener
*
*/
@Component
public class DataChangedEventDispatcher implements ApplicationListener<DataChangedEvent>, InitializingBean {
//Other logic is omitted
/**
* This method is called when there are data changes
* @param event
*/
@Override
@SuppressWarnings("unchecked")
public void onApplicationEvent(final DataChangedEvent event) {
// Traverse the data change listener (generally, it is better to use a data synchronization method)
for (DataChangedListener listener : listeners) {
// What kind of data has changed
switch (event.getGroupKey()) {
case APP_AUTH: // Authentication information
listener.onAppAuthChanged((List<AppAuthData>) event.getSource(), event.getEventType());
break;
case PLUGIN: // Plug in information
listener.onPluginChanged((List<PluginData>) event.getSource(), event.getEventType());
break;
case RULE: // Rule information
listener.onRuleChanged((List<RuleData>) event.getSource(), event.getEventType());
break;
case SELECTOR: // Selector information
listener.onSelectorChanged((List<SelectorData>) event.getSource(), event.getEventType());
break;
case META_DATA: // metadata
listener.onMetaDataChanged((List<MetaData>) event.getSource(), event.getEventType());
break;
default: // Other types, throw exception
throw new IllegalStateException("Unexpected value: " + event.getGroupKey());
}
}
}
}
Assuming that the plug-in information is modified and the data is synchronized through http long polling, the actual call of listener.onPluginChanged() is org.apache.shenyu.admin.listener.AbstractDataChangedListener#onPluginChanged:
/**
* In the operation of admin, some plug-ins have been updated
* @param changed the changed
* @param eventType the event type
*/
@Override
public void onPluginChanged(final List<PluginData> changed, final DataEventTypeEnum eventType) {
if (CollectionUtils.isEmpty(changed)) {
return;
}
// Update memory CACHE
this.updatePluginCache();
// Perform change tasks
this.afterPluginChanged(changed, eventType);
}
There are two processing operations: one is to update the memory CACHE, which has been analyzed earlier; The other is to execute change tasks in the process pool.
- HttpLongPollingDataChangedListener#afterPluginChanged()
@Override
protected void afterPluginChanged(final List<PluginData> changed, final DataEventTypeEnum eventType) {
// Execute in thread pool
scheduler.execute(new DataChangeTask(ConfigGroupEnum.PLUGIN));
}
- DataChangeTask
Data change task: remove the clients in the blocking queue in turn, and send a response to notify the gateway that a group of data has changed.
class DataChangeTask implements Runnable {
//Other logic is omitted
@Override
public void run() {
// If the number of clients in the blocking queue exceeds the given value of 100, it will be executed in batches
if (clients.size() > httpSyncProperties.getNotifyBatchSize()) {
List<LongPollingClient> targetClients = new ArrayList<>(clients.size());
clients.drainTo(targetClients);
List<List<LongPollingClient>> partitionClients = Lists.partition(targetClients, httpSyncProperties.getNotifyBatchSize());
// Batch execution
partitionClients.forEach(item -> scheduler.execute(() -> doRun(item)));
} else {
// Perform tasks
doRun(clients);
}
}
private void doRun(final Collection<LongPollingClient> clients) {
// Notify all clients of data changes
for (Iterator<LongPollingClient> iter = clients.iterator(); iter.hasNext();) {
LongPollingClient client = iter.next();
iter.remove();
// Send response
client.sendResponse(Collections.singletonList(groupKey));
log.info("send response with the changed group,ip={}, group={}, changeTime={}", client.ip, groupKey, changeTime);
}
}
}
At this point, the admin side data synchronization logic is analyzed. In the long polling data synchronization based on http, it mainly has three functions:
- Provide data change monitoring interface;
- Provide interface for obtaining change data;
- When there is data change, remove the client in the blocking queue and respond to the result.
Finally, three diagrams are used to describe the process of long polling task at the admin end:
- /Configure / listener data change listening interface:
- /Configure / fetch get change data interface:
- Update data and synchronize data in admin background management system:
4. Summary
This paper mainly analyzes the source code of http long polling data synchronization in ShenYu gateway. The main knowledge points involved are as follows:
- http long polling is actively requested by the gateway and constantly requested by the admin;
- The granularity of change data is group (authentication information, plug-ins, selectors, rules, metadata);
- http long polling results only get the change group, and you need to initiate a request again to obtain group data;
- Whether the data is updated is determined by the md5 value and the modification time lastModifyTime.