CAP principle
CAP principle, also known as CAP theorem, refers to that in a distributed system, Consistency, Availability and Partition tolerance can only achieve two points at most, and it is impossible to give consideration to the three.
BASE
BASE is short for basic available, Soft state and Eventually consistent.
BASE is the result of the trade-off between consistency and availability in CAP. It comes from the conclusion of distributed practice of large-scale Internet systems. It is gradually evolved based on the CAP theorem. Its core idea is that even if Strong consistency cannot be achieved, each application can adopt appropriate ways to achieve the final consistency of the system according to its own business characteristics (Eventual consistency). Next, we will focus on the three elements in BASE in detail. Basic availability: it refers to that the distributed system is allowed to lose some availability in case of unpredictable failure.
Comparison of common registries
ZookeeperEurekaConsulNacosEtcdData consistencyCPAPCPAP/CPCPhealth examinationKeep AliveClientBeatTCP/HTTP/grpc/CmdTCP/HTTP/MySql/ClientBeatClientBeatLoad balancing strategyRibbonFabioWeight / metadata/SelectorAvalanche protectionnothingyesnothingyesAuto logoff instance√×√access protocolTCPHTTPHTTP/DNSHTTP/DNSHTTPListening Support√√√√√Multi data center×√√√Cross registry synchronization××√√Spring Cloud integration√√√√Dubbo integration√××√K8s integration××√√√Deployment difficulty41324development language JavaJavaGoJavaGofunctionDistributed data collaborationService discovery based on HTTP protocolMulti mechanism service discovery and storageMulti mechanism service discovery, KV storage, configuration center, large and comprehensive functionsDistributed data collaborationTimelinessSecond orderIt depends on the specific configuration. By default, the service instance information is updated in 30s, and the failed nodes are eliminated in 90s. Under this configuration, the latest configuration can be obtained in 2 minutesSee the specific configurationUnder normal conditions, it is seconds, and abnormal conditions depend on the specific configuration. The default is 15sA basic registry requires the following four basic functions:
- Register service instance information
- heartbeat mechanism
- Reject failed service instance information
- Querying service instance information
zookeeper
zk itself is not intended to be a registry, but its general tree storage structure and znode machine can indirectly complete the necessary functions of service discovery. For example, we have two services a and b
/ ├ a ┆ ├ a1 ┆ └ a2 └ b └ b1
In this way, the instance information under service a can be obtained by querying node a.
In zk, temporary nodes such as a1, a2 and b1 can be created to store service instance information. When the service instance is closed or the communication is abnormal, zookeeper can automatically delete these temporary nodes, so as to realize the culling mechanism.
zk, once the service is suspended, zk senses and notifies the timeliness of other services, notifies the timeliness of other services after the service is registered in zk, and whether there will be short-term problems in the availability after the leader is suspended in exchange for consistency
**Registration mechanism: * * the client actively creates a temporary node
**Heartbeat mechanism: * * because the temporary node is created, it depends on zk's own session mechanism
**Culling mechanism: * * after the session fails, the temporary nodes are automatically culled
**Query mechanism: * * use zk protocol to query nodes
Eureka
Compared with zookeeper, Eureka is specifically used as a registry. It provides all the functions required by the registry. It provides SDK and HTTP interfaces to access Eureka Server
Some of its API s are as follows. See more Eureka REST operations
OperationHTTP actionDescriptionRegister new application instancePOST /eureka/v2/apps/appIDInput: JSON/XML payload HTTP Code: 204 on successDe-register application instanceDELETE /eureka/v2/apps/appID/instanceIDHTTP Code: 200 on successSend application instance heartbeatPUT /eureka/v2/apps/appID/instanceIDHTTP Code: * 200 on success * 404 if instanceID doesn't existQuery for all instancesGET /eureka/v2/appsHTTP Code: 200 on success Output: JSON/XMLQuery for all appID instancesGET /eureka/v2/apps/appIDHTTP Code: 200 on success Output: JSON/XMLQuery for a specific appID/instanceIDGET /eureka/v2/apps/appID/instanceIDHTTP Code: 200 on success Output: JSON/XMLIf you don't want to use the HTTP interface, you can also directly use the HTTP interface provided by Eureka Java SDK
Eureka focuses more on AP. Through the self-protection mechanism, Eureka can retain most of the node information in case of network abnormalities to prevent avalanche
If the Eureka server detects that more registered clients than expected have terminated their connections in an inappropriate manner and are waiting for eviction, they will enter self-protection mode. This is to ensure that catastrophic network events do not clear the Eureka registry data and propagate it down to all clients.
The working mechanism of the self-protection mechanism is: if more than 85% of the client nodes do not have a normal heartbeat within 15 minutes, Eureka considers that there is a network failure between the client and the registry, and Eureka Server automatically enters the self-protection mechanism. At this time, the following situations will occur:
- Eureka Server no longer removes services that should expire because they have not received a heartbeat for a long time from the registration list.
- Eureka Server can still accept the registration and query requests of new services, but it will not be synchronized to other nodes to ensure that the current node is still available.
- When the network is stable, the new registration information of the current Eureka Server will be synchronized to other nodes.
Specific visible Server Self Preservation Mode
When Eureka enters the self-protection mechanism, the service instance cannot be eliminated. The Client may query the suspended instance information when querying.
Eureka is in peer-to-peer mode. It may die before synchronizing the data. At this time, you can continue to pull the registry from other machines, but what you see is not the latest data, but ensures availability, strong consistency and final consistency
**Registration mechanism: * * the client actively creates node information (using SDK or HTTP interface)
**Heartbeat mechanism: * * the client actively maintains reporting (using SDK or HTTP interface, reporting once every 30s by default)
**Culling mechanism: * * after the client does not receive three heartbeats, the server actively deletes them
**Query mechanism: * * the client actively queries node information (using SDK or HTTP interface)
Consul
For the comparison between Consul and other registries, because Consul itself has a document, it will not be described here Consul VS Other
Consul itself provides Go SDK and HTTP interfaces, including API s for service registration, health check, service query, kv operation and other functions. Although there is no official SDK in other languages, some individuals have encapsulated it. Maybe you can use unofficial or seal the HTTP interface yourself.
Compared with Eureka, Consul provides a variety of heartbeat mechanisms, including:
- Script + Interval
- HTTP + Interval
- TCP + Interval
- Time to Live (TTL)
- Docker + Interval
- gRPC + Interval
- H2ping + Interval
- Alias
**Registration mechanism: * * the client actively creates node information (using SDK or HTTP interface)
**Heartbeat mechanism: * * the server performs heartbeat test on the client according to your heartbeat mechanism (different from Eureka and zk, this is initiated by the server to the client)
**Elimination mechanism: * * after the server fails to successfully detect the heartbeat response of the client, the server actively deletes it
**Query mechanism: * * the client actively queries node information (using SDK or HTTP interface)
Nacos
Nacos supports DNS based and RPC based service discovery. Service provider usage Native SDK,OpenAPI , or one Independent Agent TODO After registering the Service, Service consumers can use it DNS TODO or HTTP&API Find and discover services.
Nacos provides real-time health check of services to prevent sending requests to unhealthy hosts or service instances. Nacos supports health check of transport layer (PING or TCP) and application layer (such as HTTP, MySQL, user defined). For complex cloud environment and network topology environment (such as VPC, edge network, etc.) For health check of services, Nacos provides two health check modes: agent reporting mode and server-side active detection. Nacos also provides a unified health check dashboard to help you manage service availability and traffic according to health status.
**Registration mechanism: * * the client actively creates node information (using SDK or HTTP interface)
**Heartbeat mechanism: * * the client actively maintains reporting (using SDK or HTTP interface, reporting once every 30s by default)
**Culling mechanism: * * after the client does not receive three heartbeats, the server actively deletes them
**Query mechanism: * * the client actively queries node information (using SDK or HTTP interface)
How does Spring Cloud implement service governanceAnalysis on service governance of Spring Cloud Commons
When designing Spring, you usually consider extending and eliminating the template code. This design also exists in Spring clone.
In the Spring Cloud system, Spring Cloud Commons is the most important project, which defines interfaces related to service registration, service discovery, load balancing and some public components. By looking at this project, we can briefly understand the core process of Spring Cloud registration and discovery.
The following project structure is provided in the spring clone commons project (some code files and structures are omitted here)
└── src ├── main │ ├── java │ │ └── org │ │ └── springframework │ │ └── cloud │ │ ├── client │ │ │ ├── DefaultServiceInstance.java │ │ │ ├── ServiceInstance.java Spring Cloud Definition of service instance information │ │ │ ├── discovery Service discovery related │ │ │ │ ├── DiscoveryClient.java │ │ │ │ ├── EnableDiscoveryClient.java │ │ │ │ ├── EnableDiscoveryClientImportSelector.java │ │ │ │ ├── ManagementServerPortUtils.java │ │ │ │ ├── ReactiveDiscoveryClient.java │ │ │ │ ├── composite │ │ │ │ │ ├── CompositeDiscoveryClient.java │ │ │ │ │ ├── CompositeDiscoveryClientAutoConfiguration.java │ │ │ │ │ └── reactive │ │ │ │ │ ├── ReactiveCompositeDiscoveryClient.java │ │ │ │ │ └── ReactiveCompositeDiscoveryClientAutoConfiguration.java │ │ │ │ ├── health Health examination related │ │ │ │ ├── DiscoveryClientHealthIndicator.java │ │ │ │ ├── DiscoveryClientHealthIndicatorProperties.java │ │ │ │ ├── DiscoveryCompositeHealthContributor.java │ │ │ │ ├── DiscoveryHealthIndicator.java │ │ │ │ └── reactive │ │ │ │ ├── ReactiveDiscoveryClientHealthIndicator.java │ │ │ │ ├── ReactiveDiscoveryCompositeHealthContributor.java │ │ │ │ └── ReactiveDiscoveryHealthIndicator.java │ │ │ ├── loadbalancer Here is the logic related to load balancing │ │ │ └── serviceregistry Service registration related │ │ │ ├── AbstractAutoServiceRegistration.java │ │ │ ├── AutoServiceRegistration.java │ │ │ ├── AutoServiceRegistrationAutoConfiguration.java │ │ │ ├── AutoServiceRegistrationConfiguration.java │ │ │ ├── AutoServiceRegistrationProperties.java │ │ │ ├── Registration.java │ │ │ ├── ServiceRegistry.java │ │ │ ├── ServiceRegistryAutoConfiguration.java │ │ ├── commons │ │ ├── httpclient http Factory class, which can be selected in configuration Apache Http still OKHttp │ │ │ ├── ApacheHttpClientFactory.java │ │ │ └── OkHttpClientFactory.java │ │ └── util │ │ ├── IdUtils.java Generate instances through this tool class id │ │ └── InetUtils.java Spring Cloud It is through this tool class that the service items are obtained ip Addressable │ └── resources │ └── META-INF │ ├── additional-spring-configuration-metadata.json │ └── spring.factories └── test ├── java Test related codes
The source code corresponding to each part can be seen in the project structure. In service governance, the first is the service information ServiceInstance, including
- The service name ServiceId is our similar XXX server (spring.application.name)
- Service instance unique identifier InstanceId
- host
- port
- Some extension information metadata, which is mainly used to provide three-party implementations with the following extension information
// In order to shorten the space, some comments have been deleted public interface ServiceInstance { default String getInstanceId() { return null; } String getServiceId(); String getHost(); int getPort(); boolean isSecure(); URI getUri(); Map<String, String> getMetadata(); default String getScheme() { return null; } }
Service registration
Registration is a registration implementation provided by Spring Cloud
public interface Registration extends ServiceInstance { // There is really no code in it }
The actual interface for service registration is ServiceRegistry
public interface ServiceRegistry<R extends Registration> { /** * Registers the registration. A registration typically has information about an * instance, such as its hostname and port. * @param registration registration meta data */ void register(R registration); /** * Deregisters the registration. * @param registration registration meta data */ void deregister(R registration); /** * Closes the ServiceRegistry. This is a lifecycle method. */ void close(); /** * Sets the status of the registration. The status values are determined by the * individual implementations. * @param registration The registration to update. * @param status The status to set. * @see org.springframework.cloud.client.serviceregistry.endpoint.ServiceRegistryEndpoint */ void setStatus(R registration, String status); /** * Gets the status of a particular registration. * @param registration The registration to query. * @param <T> The type of the status. * @return The status of the registration. * @see org.springframework.cloud.client.serviceregistry.endpoint.ServiceRegistryEndpoint */ <T> T getStatus(R registration); }
A simple service registration function can be completed by implementing service registry
Service discovery
There are two service discovery definition interfaces DiscoveryClient and ReactiveDiscoveryClient under discovery
It provides the following functions:
- Get all service names
- Obtain the corresponding service instance list according to the service name
public interface DiscoveryClient extends Ordered { /** * Default order of the discovery client. */ int DEFAULT_ORDER = 0; /** * A human-readable description of the implementation, used in HealthIndicator. * @return The description. */ String description(); /** * Gets all ServiceInstances associated with a particular serviceId. * @param serviceId The serviceId to query. * @return A List of ServiceInstance. */ List<ServiceInstance> getInstances(String serviceId); /** * @return All known service IDs. */ List<String> getServices(); /** * Default implementation for getting order of discovery clients. * @return order */ @Override default int getOrder() { return DEFAULT_ORDER; } }
Service discovery can be completed by implementing DiscoveryClient
Health testing
ReactiveDiscoveryClientHealthIndicator provides health detection
- Get a list of all service names from DiscoveryClient
- Obtain the corresponding service instance list according to the service name list
- Perform health detection on each instance. If the response is successful, it is UP; otherwise, it is DOWN
public class ReactiveDiscoveryClientHealthIndicator implements ReactiveDiscoveryHealthIndicator, Ordered, ApplicationListener<InstanceRegisteredEvent<?>> { private final ReactiveDiscoveryClient discoveryClient; private final DiscoveryClientHealthIndicatorProperties properties; private final Log log = LogFactory.getLog(ReactiveDiscoveryClientHealthIndicator.class); private AtomicBoolean discoveryInitialized = new AtomicBoolean(false); private int order = Ordered.HIGHEST_PRECEDENCE; public ReactiveDiscoveryClientHealthIndicator(ReactiveDiscoveryClient discoveryClient, DiscoveryClientHealthIndicatorProperties properties) { this.discoveryClient = discoveryClient; this.properties = properties; } @Override public void onApplicationEvent(InstanceRegisteredEvent<?> event) { if (this.discoveryInitialized.compareAndSet(false, true)) { this.log.debug("Discovery Client has been initialized"); } } @Override public Mono<Health> health() { if (this.discoveryInitialized.get()) { return doHealthCheck(); } else { return Mono.just( Health.status(new Status(Status.UNKNOWN.getCode(), "Discovery Client not initialized")).build()); } } private Mono<Health> doHealthCheck() { // @formatter:off return Mono.justOrEmpty(this.discoveryClient) .flatMapMany(ReactiveDiscoveryClient::getServices) .collectList() .defaultIfEmpty(emptyList()) .map(services -> { ReactiveDiscoveryClient client = this.discoveryClient; String description = (this.properties.isIncludeDescription()) ? client.description() : ""; return Health.status(new Status("UP", description)) .withDetail("services", services).build(); }) .onErrorResume(exception -> { this.log.error("Error", exception); return Mono.just(Health.down().withException(exception).build()); }); // @formatter:on } @Override public String getName() { return discoveryClient.description(); } @Override public int getOrder() { return this.order; } public void setOrder(int order) { this.order = order; } }
Through the above interface definition and the built-in health detection logic, we can see the simplest logic to be implemented for service governance
- Implement the service registry function
- Implement DiscoveryClient functionality
Spring cloud consult implementation
Implement the service registry function
In spring cloud consult, first customize the implementation of Registration
NewService is some service instance information defined by consult
public class ConsulRegistration implements Registration { private final NewService service; private ConsulDiscoveryProperties properties; public ConsulRegistration(NewService service, ConsulDiscoveryProperties properties) { this.service = service; this.properties = properties; } public NewService getService() { return this.service; } protected ConsulDiscoveryProperties getProperties() { return this.properties; } public String getInstanceId() { return getService().getId(); } public String getServiceId() { return getService().getName(); } @Override public String getHost() { return getService().getAddress(); } @Override public int getPort() { return getService().getPort(); } @Override public boolean isSecure() { return this.properties.getScheme().equalsIgnoreCase("https"); } @Override public URI getUri() { return DefaultServiceInstance.getUri(this); } @Override public Map<String, String> getMetadata() { return getService().getMeta(); } }
NewService
It contains the basic information of the service and some unique functions provided by Consul itself, such as Tags and Check
// General getter, setter and toString methods are deleted public class NewService { @SerializedName("ID") private String id; @SerializedName("Name") private String name; @SerializedName("Tags") private List<String> tags; @SerializedName("Address") private String address; @SerializedName("Meta") private Map<String, String> meta; @SerializedName("Port") private Integer port; @SerializedName("EnableTagOverride") private Boolean enableTagOverride; @SerializedName("Check") private NewService.Check check; @SerializedName("Checks") private List<NewService.Check> checks; public NewService() { } public static class Check { @SerializedName("Script") private String script; @SerializedName("DockerContainerID") private String dockerContainerID; @SerializedName("Shell") private String shell; @SerializedName("Interval") private String interval; @SerializedName("TTL") private String ttl; @SerializedName("HTTP") private String http; @SerializedName("Method") private String method; @SerializedName("Header") private Map<String, List<String>> header; @SerializedName("TCP") private String tcp; @SerializedName("Timeout") private String timeout; @SerializedName("DeregisterCriticalServiceAfter") private String deregisterCriticalServiceAfter; @SerializedName("TLSSkipVerify") private Boolean tlsSkipVerify; @SerializedName("Status") private String status; @SerializedName("GRPC") private String grpc; @SerializedName("GRPCUseTLS") private Boolean grpcUseTLS; public Check() { } } }
ConsulServiceRegistry implements ServiceRegistry
public class ConsulServiceRegistry implements ServiceRegistry<ConsulRegistration> { private static Log log = LogFactory.getLog(ConsulServiceRegistry.class); private final ConsulClient client; private final ConsulDiscoveryProperties properties; private final TtlScheduler ttlScheduler; private final HeartbeatProperties heartbeatProperties; public ConsulServiceRegistry(ConsulClient client, ConsulDiscoveryProperties properties, TtlScheduler ttlScheduler, HeartbeatProperties heartbeatProperties) { this.client = client; this.properties = properties; this.ttlScheduler = ttlScheduler; this.heartbeatProperties = heartbeatProperties; } @Override public void register(ConsulRegistration reg) { log.info("Registering service with consul: " + reg.getService()); try { // The service is also registered through the api interface provided by consumer this.client.agentServiceRegister(reg.getService(), this.properties.getAclToken()); NewService service = reg.getService(); if (this.heartbeatProperties.isEnabled() && this.ttlScheduler != null && service.getCheck() != null && service.getCheck().getTtl() != null) { this.ttlScheduler.add(reg.getInstanceId()); } } catch (ConsulException e) { if (this.properties.isFailFast()) { log.error("Error registering service with consul: " + reg.getService(), e); ReflectionUtils.rethrowRuntimeException(e); } log.warn("Failfast is false. Error registering service with consul: " + reg.getService(), e); } } @Override public void deregister(ConsulRegistration reg) { if (this.ttlScheduler != null) { this.ttlScheduler.remove(reg.getInstanceId()); } if (log.isInfoEnabled()) { log.info("Deregistering service with consul: " + reg.getInstanceId()); } this.client.agentServiceDeregister(reg.getInstanceId(), this.properties.getAclToken()); } @Override public void close() { } @Override public void setStatus(ConsulRegistration registration, String status) { if (status.equalsIgnoreCase(OUT_OF_SERVICE.getCode())) { this.client.agentServiceSetMaintenance(registration.getInstanceId(), true); } else if (status.equalsIgnoreCase(UP.getCode())) { this.client.agentServiceSetMaintenance(registration.getInstanceId(), false); } else { throw new IllegalArgumentException("Unknown status: " + status); } } // Service instance status @Override public Object getStatus(ConsulRegistration registration) { String serviceId = registration.getServiceId(); Response<List<Check>> response = this.client.getHealthChecksForService(serviceId, HealthChecksForServiceRequest.newBuilder().setQueryParams(QueryParams.DEFAULT).build()); List<Check> checks = response.getValue(); for (Check check : checks) { if (check.getServiceId().equals(registration.getInstanceId())) { if (check.getName().equalsIgnoreCase("Service Maintenance Mode")) { return OUT_OF_SERVICE.getCode(); } } } return UP.getCode(); } }
Consumdiscoveryclient implements DiscoveryClient
In the discovery logic, queries are also made through the api interface provided by consumer
public class ConsulDiscoveryClient implements DiscoveryClient { private final ConsulClient client; private final ConsulDiscoveryProperties properties; public ConsulDiscoveryClient(ConsulClient client, ConsulDiscoveryProperties properties) { this.client = client; this.properties = properties; } @Override public String description() { return "Spring Cloud Consul Discovery Client"; } @Override public List<ServiceInstance> getInstances(final String serviceId) { return getInstances(serviceId, new QueryParams(this.properties.getConsistencyMode())); } public List<ServiceInstance> getInstances(final String serviceId, final QueryParams queryParams) { List<ServiceInstance> instances = new ArrayList<>(); addInstancesToList(instances, serviceId, queryParams); return instances; } private void addInstancesToList(List<ServiceInstance> instances, String serviceId, QueryParams queryParams) { HealthServicesRequest.Builder requestBuilder = HealthServicesRequest.newBuilder() .setPassing(this.properties.isQueryPassing()).setQueryParams(queryParams) .setToken(this.properties.getAclToken()); String queryTag = this.properties.getQueryTagForService(serviceId); if (queryTag != null) { requestBuilder.setTag(queryTag); } HealthServicesRequest request = requestBuilder.build(); Response<List<HealthService>> services = this.client.getHealthServices(serviceId, request); for (HealthService service : services.getValue()) { instances.add(new ConsulServiceInstance(service, serviceId)); } } public List<ServiceInstance> getAllInstances() { List<ServiceInstance> instances = new ArrayList<>(); Response<Map<String, List<String>>> services = this.client .getCatalogServices(CatalogServicesRequest.newBuilder().setQueryParams(QueryParams.DEFAULT).build()); for (String serviceId : services.getValue().keySet()) { addInstancesToList(instances, serviceId, QueryParams.DEFAULT); } return instances; } @Override public List<String> getServices() { CatalogServicesRequest request = CatalogServicesRequest.newBuilder().setQueryParams(QueryParams.DEFAULT) .setToken(this.properties.getAclToken()).build(); return new ArrayList<>(this.client.getCatalogServices(request).getValue().keySet()); } @Override public int getOrder() { return this.properties.getOrder(); } }
summary
Briefly, the service governance logic of spring cloud consult is roughly the same. Of course, spring cloud consult has to deal with a lot of details and a lot of code
In the Spring Cloud system, Consul does not provide the function of service request forwarding, but only provides the functions of saving, querying and health detection of service information
reference resources- Consul official introduction https://www.consul.io/docs/intro
- Spring Cloud Consul https://github.com/spring-cloud/spring-cloud-consul
- Spring Cloud Commons https://github.com/spring-cloud/spring-cloud-commons
- Nacos documentation https://nacos.io
- https://github.com/Netflix/eureka