Source code analysis of Zhiyu dubbo

dubbo source code analysis

file

Official documents
gitbook document

dubbo load profile

dubbo-2.6.4.jar spring.handlers The file indicates the class to parse xml
The contents of the document are as follows

http\://dubbo.apache.org/schema/dubbo=com.alibaba.dubbo.config.spring.schema.DubboNamespaceHandler
http\://code.alibabatech.com/schema/dubbo=com.alibaba.dubbo.config.spring.schema.DubboNamespaceHandler

The init method of DubboNamespaceHandler will
Creating multiple dubbobeandefinitionparsers parses xml in its parse method
The parse method places each object configured in the xml in a bean definition

The above code parses the xml configuration information into BeanDefinition and puts it into the spring container

dubbo service provision

dubbo provides services mainly in the following steps
1. Get the configuration information of dubbo in the spring container and verify the configuration information at the same time
2. Verify the configuration information parameters and create the url object according to the configuration information
-------(the main information that the server and the client call each other is stored in the URL)
3. create Invoker object
-------(the Invoker object uses Javassist to wrap the specified interface, and loads the implementation class when it is called at the same time of parameter verification.)
4. Local registration (put the service into a map according to the name and Invoker)
5. Remote registration
5.1 enable netty server
5.2 connect to the registration center zk and write information
5.3 monitoring Registration Center

1. Verify configuration information

The entry for dubbo to provide services is in the ServiceBean class. First, observe the ServiceBean class

public class ServiceBean<T> extends ServiceConfig<T> implements InitializingBean, DisposableBean, ApplicationContextAware, ApplicationListener<ContextRefreshedEvent>, BeanNameAware {}

First, analyze the method afterpropertieset() of the InitializingBean interface
This method is executed when spring calls the initialization method at project startup

Method ModuleConfig.class , RegistryConfig.class ,
MonitorConfig.class ,  ProtocolConfig.class  Obtain all BeanDefinition objects of type and verify the object information at the same time

Next, analyze the method setApplicationContext() in the ApplicationContextAware interface
This method is called when the spring initialization callback is injected into the ApplicationContext object
Method to add servicebean (itself) to spring's listener,
serviceBean is the ApplicationListener interface implementation class, which is also a listener
Some codes are as follows

//This will trigger onApplicationEvent() in the ApplicationListener class when the spring startup is completed 
Method method = applicationContext.getClass()
		.getMethod("addApplicationListener", new Class<?>[]{ApplicationListener.class}); 
method.invoke(applicationContext, new Object[]{this});

Next, analyze the method onApplicationEvent(ContextRefreshedEvent event) in the ApplicationListener class
After spring initialization, the following methods will be called

public void onApplicationEvent(ContextRefreshedEvent event) {
    //If it is a non delayed export & &! You have exported & &! Do not export
    if (isDelay() && !isExported() && !isUnexported()) {
        if (logger.isInfoEnabled()) {
            logger.info("The service ready on spring started. service: " + getInterface());
        }
        //Export service implements service provision in this sentence
        //Next, analyze the export() method
        export();
    }
}

2. Create URL

We usually make the following configuration, which is stored in ServiceConfig with the corresponding

<dubbo:service interface="com.alibaba.dubbo.config.spring.api.DemoService"   ref="demoService" />

Next, let's continue to look at the export() method in the ServiceBean class

//First, a series of verifications are carried out
//The interfaceName variable is in xml dubbo:service  interface properties configured in
//Load the next class to determine whether it exists
    interfaceClass = Class.forName(interfaceName, true, Thread.currentThread()
                        .getContextClassLoader());
    //Export service
    doExportUrls();

Let's move on to the doExportUrls() method

private void doExportUrls() {
    //Get all registry addresses, dubbo can configure multiple registries
    List<URL> registryURLs = loadRegistries(true);
    //Each registry can be configured with multiple protocols
    for (ProtocolConfig protocolConfig : protocols) {
        //Provision of services in accordance with the agreement and the registry
        doExportUrlsFor1Protocol(protocolConfig, registryURLs);
    }
}

Next, continue to study the doexporturlsfor1 protocol (protocol config, registry URLs) method

//Take out the configuration information and put it into a map, and then use the map to create the URL object
URL url = new URL(name, host, port,provider.getContextpath() + path, map);
//The method getParameter(key,defaultValue) in the url is used to get the information in the map

The URL object is created

3. Local registration

The doExportUrlsFor1Protocol() method continues

	if (!"remote".equalsIgnoreCase(scope))
 	{
 	    //Local service registration
		exportLocal(url);
	}

Next, we study the method of exportLocal(url)

 private void exportLocal(URL url) {
        if (!Constants.LOCAL_PROTOCOL.equalsIgnoreCase(url.getProtocol())) {
            URL local = URL.valueOf(url.toFullString())
                    .setProtocol(Constants.LOCAL_PROTOCOL) 
                     //Constants.LOCAL_ The value of protocol is injvm, so InjvmProtocol will be called
                    .setHost(LOCALHOST)
                    .setPort(0);
            ServiceClassHolder.getInstance().pushServiceClass(getServiceClass(ref));
            //Call here Protocol.class  The export method in InjvmProtocol
            Exporter<?> exporter = protocol.export(
                    //This ref is< dubbo:service >Ref attribute in (references the id name of a bean)
                    //An Invoker will be created here
                    proxyFactory.getInvoker(ref, (Class) interfaceClass, local));
            exporters.add(exporter);
            logger.info("Export dubbo service " + interfaceClass.getName() + " to local registry");
        }
    }

Next, study the getInvoker(ref,interfaceClass,local) method of proxyFactory, which will create an Invoker object
Because the annotation on the ProxyFactory interface is @ SPI("javassist"), the JavassistProxyFactory class will be called
The getInvoker method is as follows

public <T> Invoker<T> getInvoker(T proxy, Class<T> type, URL url) {
        // Create a wrapper class. The returned Invoker object executes all methods and calls the invokeMethod method in the wrapper class
        final Wrapper wrapper = Wrapper.getWrapper(proxy.getClass().getName().indexOf('$') < 0 ? proxy.getClass() : type);
        return new AbstractProxyInvoker<T>(proxy, type, url) {
            @Override
            protected Object doInvoke(T proxy, String methodName,
                                      Class<?>[] parameterTypes,
                                      Object[] arguments) throws Throwable {
                return wrapper.invokeMethod(proxy, methodName, parameterTypes, arguments);
            }
        };
    }

How is the wrapper object created? Let's go on to analyze Wrapper.getWrapper(ref)

public static Wrapper getWrapper(Class<?> c) {
    while (ClassGenerator.isDynamicClass(c)) 
        c = c.getSuperclass();

    //If it is an object type, directly return OBJECT_WRAPPER a simple object
    if (c == Object.class)
        return OBJECT_WRAPPER;
    //See if there is any in the cache
    Wrapper ret = WRAPPER_MAP.get(c);
    if (ret == null) {
        //No proxy object of this ref class is created directly in the cache
        ret = makeWrapper(c);
        //Put objects in cache
        WRAPPER_MAP.put(c, ret);
    }
    return ret;
}

Next, we analyze the makeWrapper(ref) method. How does it create Wrapper objects

//There is a lot of code in this class. It mainly creates an object proxy, which will load the implementation class to be called according to the parameters when it is called
//First, create all fields
	   for (Field f : c.getFields()) {
            String fn = f.getName();
            Class<?> ft = f.getType();
            if (Modifier.isStatic(f.getModifiers()) || Modifier.isTransient(f.getModifiers()))
                continue;
            c1.append(" if( $2.equals(\"").append(fn).append("\") ){ w.").append(fn).append("=").append(arg(ft, "$3")).append("; return; }");
            c2.append(" if( $2.equals(\"").append(fn).append("\") ){ return ($w)w.").append(fn).append("; }");
            pts.put(fn, ft);
        }
//Then create all the methods at once
 			for (Method m : methods) {
            if (m.getDeclaringClass() == Object.class) //ignore Object's method.
                continue;

            String mn = m.getName();
            c3.append(" if( \"").append(mn).append("\".equals( $2 ) ");
            int len = m.getParameterTypes().length;
            c3.append(" && ").append(" $3.length == ").append(len);

            boolean override = false;
            for (Method m2 : methods) {
                if (m != m2 && m.getName().equals(m2.getName())) {
                    override = true;
                    break;
                }
            }
            if (override) {
                if (len > 0) {
                    for (int l = 0; l < len; l++) {
                        c3.append(" && ").append(" $3[").append(l).append("].getName().equals(\"")
                                .append(m.getParameterTypes()[l].getName()).append("\")");
                    }
                }
            }

            c3.append(" ) { ");

            if (m.getReturnType() == Void.TYPE)
                c3.append(" w.").append(mn).append('(').append(args(m.getParameterTypes(), "$4")).append(");").append(" return null;");
            else
                c3.append(" return ($w)w.").append(mn).append('(').append(args(m.getParameterTypes(), "$4")).append(");");

            c3.append(" }");

            mns.add(mn);
            if (m.getDeclaringClass() == c)
                dmns.add(mn);
            ms.put(ReflectUtils.getDesc(m), m);
        }
//The start method of get set is is processed by regular matching
			for (Map.Entry<String, Method> entry : ms.entrySet()) {
            String md = entry.getKey();
            Method method = (Method) entry.getValue();
            if ((matcher = ReflectUtils.GETTER_METHOD_DESC_PATTERN.matcher(md)).matches()) {
                String pn = propertyName(matcher.group(1));
                c2.append(" if( $2.equals(\"").append(pn).append("\") ){ return ($w)w.").append(method.getName()).append("(); }");
                pts.put(pn, method.getReturnType());
            } else if ((matcher = ReflectUtils.IS_HAS_CAN_METHOD_DESC_PATTERN.matcher(md)).matches()) {
                String pn = propertyName(matcher.group(1));
                c2.append(" if( $2.equals(\"").append(pn).append("\") ){ return ($w)w.").append(method.getName()).append("(); }");
                pts.put(pn, method.getReturnType());
            } else if ((matcher = ReflectUtils.SETTER_METHOD_DESC_PATTERN.matcher(md)).matches()) {
                Class<?> pt = method.getParameterTypes()[0];
                String pn = propertyName(matcher.group(1));
                c1.append(" if( $2.equals(\"").append(pn).append("\") ){ w.").append(method.getName()).append("(").append(arg(pt, "$3")).append("); return; }");
                pts.put(pn, pt);
            }
        }
 //Generate class file based on information
  			 Class<?> wc = cc.toClass();
            // setup static field.
            wc.getField("pts").set(null, pts);
            wc.getField("pns").set(null, pts.keySet().toArray(new String[0]));
            wc.getField("mns").set(null, mns.toArray(new String[0]));
            wc.getField("dmns").set(null, dmns.toArray(new String[0]));
            int ix = 0;
            for (Method m : ms.values())
                wc.getField("mts" + ix++).set(null, m.getParameterTypes());
            return (Wrapper) wc.newInstance();

At this time, the Invoke object is created, which will wrap the incoming object and load the implementation class when the call is resolved
Because it is a local registration service, it is time to save the created object
Next, we study the exporter method of the InjvmProtocol class in the exportLocal(url) method

 public <T> Exporter<T> export(Invoker<T> invoker) throws RpcException {
        return new InjvmExporter<T>(invoker, invoker.getUrl().getServiceKey(), exporterMap);
    }
	//The construction method of the InjvmExporter is as follows. You can see that the Invoker is put into the exporterMap    
	InjvmExporter(Invoker<T> invoker, String key, Map<String, Exporter<?>> exporterMap) {
	        super(invoker);
	        this.key = key;
	        this.exporterMap = exporterMap;
	        //Store invoker 
	        exporterMap.put(key, this);
	 }

4. Remote registration

Then we study the follow-up code of do export URLs for 1 protocol (protocol config, registry URLs)
The code is as follows

  if (registryURLs != null && !registryURLs.isEmpty()) {
        //Loop through all registry addresses
        for (URL registryURL : registryURLs) {
      //Replace the protocol in the URL with registry for the following protocol.export() calling RegistryProtocol implementation class
                URL monitorUrl = loadMonitor(registryURL);
               //Creating an invoker for a remote service provider
               //Here, the logic is the same as that of the local export to create the invoker. Both call getInvoker() in JavassistProxyFactory
               Invoker<?> invoker = proxyFactory.getInvoker(ref, (Class) interfaceClass, registryURL.addParameterAndEncoded(Constants.EXPORT_KEY, url.toFullString()));
               //Wrap the invoker and serviceConfig objects
               DelegateProviderMetaDataInvoker wrapperInvoker = new DelegateProviderMetaDataInvoker(invoker, this);
			  //Providing services through the invoker object
              Exporter<?> exporter = protocol.export(wrapperInvoker);
              //Add to cache
              exporters.add(exporter);
             }
   }

4.1 enable netty server

Next research protocol.export(wrapperInvoker) method
First of all, how is protocol created

private static final Protocol protocol = ExtensionLoader.getExtensionLoader(Protocol.class).getAdaptiveExtension();

Next, we will study the getadapteextension () method
This method will dynamically create the proxy class of Protocol. The proxy class code is as follows

  public class Protocol$Adpative implements com.alibaba.dubbo.rpc.Protocol {
        public void destroy() {
            throw new UnsupportedOperationException("method public abstract void com.alibaba.dubbo.rpc.Protocol.destroy() of interface com.alibaba.dubbo.rpc.Protocol is not adaptive method!");
        }

        public int getDefaultPort() {
            throw new UnsupportedOperationException("method public abstract int com.alibaba.dubbo.rpc.Protocol.getDefaultPort() of interface com.alibaba.dubbo.rpc.Protocol is not adaptive method!");
        }

        public com.alibaba.dubbo.rpc.Invoker refer(java.lang.Class arg0, com.alibaba.dubbo.common.URL arg1) throws com.alibaba.dubbo.rpc.RpcException {
            if (arg1 == null) throw new IllegalArgumentException("url == null");
            com.alibaba.dubbo.common.URL url = arg1;
            String extName = (url.getProtocol() == null ? "dubbo" : url.getProtocol());
            if (extName == null)
                throw new IllegalStateException("Fail to get extension(com.alibaba.dubbo.rpc.Protocol) name from url(" + url.toString() + ") use keys([protocol])");
            com.alibaba.dubbo.rpc.Protocol extension = (com.alibaba.dubbo.rpc.Protocol) ExtensionLoader.getExtensionLoader(com.alibaba.dubbo.rpc.Protocol.class).getExtension(extName);
            return extension.refer(arg0, arg1);
        }

        public com.alibaba.dubbo.rpc.Exporter export(com.alibaba.dubbo.rpc.Invoker arg0) throws com.alibaba.dubbo.rpc.RpcException {
            if (arg0 == null) throw new IllegalArgumentException("com.alibaba.dubbo.rpc.Invoker argument == null");
            if (arg0.getUrl() == null)
                throw new IllegalArgumentException("com.alibaba.dubbo.rpc.Invoker argument getUrl() == null");
            com.alibaba.dubbo.common.URL url = arg0.getUrl();
            String extName = (url.getProtocol() == null ? "dubbo" : url.getProtocol());
            if (extName == null)
                throw new IllegalStateException("Fail to get extension(com.alibaba.dubbo.rpc.Protocol) name from url(" + url.toString() + ") use keys([protocol])");
            com.alibaba.dubbo.rpc.Protocol extension = (com.alibaba.dubbo.rpc.Protocol) ExtensionLoader.getExtensionLoader(com.alibaba.dubbo.rpc.Protocol.class).getExtension(extName);
            return extension.export(arg0);
        }
    }

According to the above code, the implementation class specified is called according to the protocol parameter in the incoming parameter url
At this time, the parameter of the protocol is registry, so the export() method in the RegistryProtocol class is called
Next, analyze the code of opening Netty service in export()

 public <T> Exporter<T> export(final Invoker<T> originInvoker) throws RpcException {
        //Open netty service
        final ExporterChangeableWrapper<T> exporter = doLocalExport(originInvoker);
    }

Next, the doLocalExport(originInvoker) method is analyzed. The code is as follows

String key = getCacheKey(originInvoker);
        ExporterChangeableWrapper<T> exporter = (ExporterChangeableWrapper<T>) bounds.get(key);
        if (exporter == null) {
            synchronized (bounds) {
                exporter = (ExporterChangeableWrapper<T>) bounds.get(key);
                if (exporter == null) {
                    final Invoker<?> invokerDelegete = new InvokerDelegete<T>(originInvoker, getProviderUrl(originInvoker));
                    exporter = new ExporterChangeableWrapper<T>((Exporter<T>) protocol.export(invokerDelegete), originInvoker);
                    bounds.put(key, exporter);
                }
            }
        }
        return exporter;

The most important part of the above code is protocol.export (invokerdelegeete) method
Next, I will analyze this method
First of all, this protocol.export() which implementation class will be called, where the protocol object is a member variable of this class
The implementation class of this class is generally DubboProtocol. Next, let's see the method expand() of DubboProtocol

 public <T> Exporter<T> export(Invoker<T> invoker) throws RpcException {
        URL url = invoker.getUrl();

        String key = serviceKey(url);
        DubboExporter<T> exporter = new DubboExporter<T>(invoker, key, exporterMap);
        exporterMap.put(key, exporter);

        Boolean isStubSupportEvent = url.getParameter(Constants.STUB_EVENT_KEY, Constants.DEFAULT_STUB_EVENT);
        Boolean isCallbackservice = url.getParameter(Constants.IS_CALLBACK_SERVICE, false);
        if (isStubSupportEvent && !isCallbackservice) {
            String stubServiceMethods = url.getParameter(Constants.STUB_EVENT_METHODS_KEY);
            if (stubServiceMethods == null || stubServiceMethods.length() == 0) {
                if (logger.isWarnEnabled()) {
                    logger.warn(new IllegalStateException("consumer [" + url.getParameter(Constants.INTERFACE_KEY) +
                            "], has set stubproxy support event ,but no stub methods founded."));
                }
            } else {
                stubServiceMethodsMap.put(url.getServiceKey(), stubServiceMethods);
            }
        }
		//This method is mainly used to start the Netty service
        openServer(url);
        optimizeSerialization(url);
        return exporter;
    }

Next, analyze the openServer(url) method
The code is as follows

private void openServer(URL url) {
        // find server.
        String key = url.getAddress();
        //client can export a service which's only for server to invoke
        boolean isServer = url.getParameter(Constants.IS_SERVER_KEY, true);
        if (isServer) {
            //Judge whether the service is opened in the cache
            ExchangeServer server = serverMap.get(key);
            if (server == null) {
                serverMap.put(key, createServer(url));
            } else {
                // server supports reset, use together with override
                server.reset(url);
            }
        }
    }

Next, analyze the createServer(url) method
The main codes are as follows

   try {
      //Here, a requestHandler object is passed in, which is the ExchangeHandlerAdapter() {} declared in DubboProtocol
      //This object will be used to process the request service at the end. First, write down this class
      server = Exchangers.bind(url, requestHandler);
  } catch (RemotingException e) {
       throw new RpcException("Fail to start server(url: " + url + ") " + e.getMessage(), e);
 }

Next, we will analyze Exchangers.bind(url, requestHandler); method
In the end, the bind method of HeaderExchanger class will be called
The code is as follows

public ExchangeServer bind(URL url, ExchangeHandler handler) throws RemotingException {
        //Here, the handler just passed in is wrapped twice
        //So what does the handler look like now
        //new DecodeHandler(new HeaderExchangeHandler(new ExchangeHandlerAdapter()))
        //Note down the handler, because the handler will finally process the request service
        return new HeaderExchangeServer(Transporters.bind(url, new DecodeHandler(new HeaderExchangeHandler(handler))));
}

The next analysis Transporters.bind(url,handler) method, which will be called to implementation class NettyTransporter after spi call
The code is as follows. Let's analyze the Netty3.x version

public Server bind(URL url, ChannelHandler listener) throws RemotingException {
    //After a complex call, I finally saw the code to open Netty
    return new NettyServer(url, listener);
}

Next let's take a look at what NettyServer(url, listener) has done

public NettyServer(URL url, ChannelHandler handler) throws RemotingException {
    //The handler is wrapped here
    //The packaging is as follows 
    //new MultiMessageHandler(new HeartbeatHandler(
    //new AllChannelHandler(new DecodeHandler(
    //new HeaderExchangeHandler(new ExchangeHandlerAdapter())))))
    super(url, ChannelHandlers.wrap(handler, ExecutorUtil.setThreadName(url, SERVER_THREAD_POOL_NAME)));
}

Next, it analyzes what the parent class construction method of new NettyServer(url, listener); does
Skip simple calls and look at the main code directly

protected void doOpen() throws Throwable {
    NettyHelper.setNettyLoggerFactory();
    //Create boss thread pool
    ExecutorService boss = Executors.newCachedThreadPool(new NamedThreadFactory("NettyServerBoss", true));
    //Create worker thread pool
    ExecutorService worker = Executors.newCachedThreadPool(new NamedThreadFactory("NettyServerWorker", true));
    //Create factory
    ChannelFactory channelFactory = new NioServerSocketChannelFactory(boss, worker, getUrl().getPositiveParameter(Constants.IO_THREADS_KEY, Constants.DEFAULT_IO_THREADS));
    //Create core configuration class
    bootstrap = new ServerBootstrap(channelFactory);
    
    //Create a haddler to wrap the NettyServer. The passed handler is stored in the parent class member variable of NettyServer
    final NettyHandler nettyHandler = new NettyHandler(getUrl(), this);
    channels = nettyHandler.getChannels();
  
    bootstrap.setOption("child.tcpNoDelay", true);
    bootstrap.setPipelineFactory(new ChannelPipelineFactory() {
        @Override
        public ChannelPipeline getPipeline() {
            NettyCodecAdapter adapter = new NettyCodecAdapter(getCodec(), getUrl(), NettyServer.this);
            ChannelPipeline pipeline = Channels.pipeline();
            //When timeout occurs
            /*int idleTimeout = getIdleTimeout();
            if (idleTimeout > 10000) {
                pipeline.addLast("timer", new IdleStateHandler(timer, idleTimeout / 1000, 0, 0));
            }*/
            pipeline.addLast("decoder", adapter.getDecoder());
            pipeline.addLast("encoder", adapter.getEncoder());
            pipeline.addLast("handler", nettyHandler);
            return pipeline;
        }
    });
    // bind
    channel = bootstrap.bind(getBindAddress());
}

This enables the NettyServer server. How does the server handle the request
Let's write down the class that processes the request for subsequent analysis
adapter.getDecoder() / / the encoder uses the NettyCodecAdapter class
adapter.getEncoder() / / the decoder uses the NettyCodecAdapter class
nettyHandler / / the handler handling the request is wrapped with the following class
new NettyHandler(new MultiMessageHandler(new HeartbeatHandler(new AllChannelHandler(new DecodeHandler(new HeaderExchangeHandler(new ExchangeHandlerAdapter()))))))

4.2 connect to the registration center zk and write information

Next, we analyze the export(Invoker) method of the RegistryProtocol class
The code of export is as follows

//Get registration center address
URL registryUrl = getRegistryUrl(originInvoker);
//Link the registration center here according to the address of the registration center
final Registry registry = getRegistry(originInvoker);
//Remove some parameters
final URL registeredProviderUrl = getRegisteredProviderUrl(originInvoker);
//to judge to delay publish whether or not
boolean register = registeredProviderUrl.getParameter("register", true);
//Pack it and save it to the invoker
ProviderConsumerRegTable.registerProvider(originInvoker, registryUrl, registeredProviderUrl);

if (register) {
    //Perform registration
    register(registryUrl, registeredProviderUrl);
    ProviderConsumerRegTable.getProviderWrapper(originInvoker).setReg(true);
}

Let's first analyze the getRegistry(originInvoker); method
Then analyze register(registryUrl, registeredProviderUrl); method
In the getRegistry(originInvoker); method, if the registry takes zk as an example, skipping the calling code will finally call the curatorzookeepclient (URL URL URL) method of the curatorzookeepclient class,
The following code is the connection zk, and the code is as follows

 public CuratorZookeeperClient(URL url) {
        super(url);
        try {
            //Connect to zk server
            CuratorFrameworkFactory.Builder builder = CuratorFrameworkFactory.builder()
                    .connectString(url.getBackupAddress())
                    .retryPolicy(new RetryNTimes(1, 1000))
                    .connectionTimeoutMs(5000);
            String authority = url.getAuthority();
            if (authority != null && authority.length() > 0) {
                builder = builder.authorization("digest", authority.getBytes());
            }
            client = builder.build();
            client.getConnectionStateListenable().addListener(new ConnectionStateListener() {
                @Override
                public void stateChanged(CuratorFramework client, ConnectionState state) {
                    if (state == ConnectionState.LOST) {
                        CuratorZookeeperClient.this.stateChanged(StateListener.DISCONNECTED);
                    } else if (state == ConnectionState.CONNECTED) {
                        CuratorZookeeperClient.this.stateChanged(StateListener.CONNECTED);
                    } else if (state == ConnectionState.RECONNECTED) {
                        CuratorZookeeperClient.this.stateChanged(StateListener.RECONNECTED);
                    }
                }
            });
            //connect
            client.start();
        } catch (Exception e) {
            throw new IllegalStateException(e.getMessage(), e);
        }
    }

Next, analyze register(registryUrl, registeredProviderUrl); methods
Remove the calling code, as follows

public void register(URL url) {
        super.register(url);
        failedRegistered.remove(url);
        failedUnregistered.remove(url);
        try {
            // Sending a registration request to the server side
            doRegister(url);
        } catch (Exception e) {
            Throwable t = e;

            // If the startup detection is opened, the Exception is thrown directly.
            boolean check = getUrl().getParameter(Constants.CHECK_KEY, true)
                    && url.getParameter(Constants.CHECK_KEY, true)
                    && !Constants.CONSUMER_PROTOCOL.equals(url.getProtocol());
            boolean skipFailback = t instanceof SkipFailbackWrapperException;
            if (check || skipFailback) {
                if (skipFailback) {
                    t = t.getCause();
                }
                throw new IllegalStateException("Failed to register " + url + " to registry " + getUrl().getAddress() + ", cause: " + t.getMessage(), t);
            } else {
                logger.error("Failed to register " + url + ", waiting for retry, cause: " + t.getMessage(), t);
            }

            // Record a failed registration request to a failed list, retry regularly
            failedRegistered.add(url);
        }
    }

Next, we analyze the doRegister(url) method
The following code creates a persistent node in zk

@Override
    protected void doRegister(URL url) {
        try {
            zkClient.create(toUrlPath(url), url.getParameter(Constants.DYNAMIC_KEY, true));
        } catch (Throwable e) {
            throw new RpcException("Failed to register " + url + " to zookeeper " + getUrl() + ", cause: " + e.getMessage(), e);
        }
    }

4.3 monitoring Registration Center

Next, we analyze the export(Invoker) method of the RegistryProtocol class. The code of the export part is as follows

  //Create url
  final URL overrideSubscribeUrl = getSubscribedOverrideUrl(registeredProviderUrl);
  //Create a listener
  final OverrideListener overrideSubscribeListener = new OverrideListener(overrideSubscribeUrl, originInvoker);
  //Put in cache
  overrideListeners.put(overrideSubscribeUrl, overrideSubscribeListener);
  //Subscribe to messages
  registry.subscribe(overrideSubscribeUrl, overrideSubscribeListener);

dubbo reference service

1. Reference service entry

Service references are configured in xml with the following information

<! -- add reference remote service configuration -- >
<dubbo:reference id="xxxService" interface="com.xxx.XxxService" />
<! -- use remote services as local services -- >
<bean id="xxxAction" class="com.xxx.XxxAction"> 
    <property name="xxxService" ref="xxxService" />
</bean>

The above configuration information will be resolved to the ReferenceConfig class
The entry class of service reference is ReferenceBean. Let's analyze it and see its implementation first

public class ReferenceBean<T> extends ReferenceConfig<T> implements FactoryBean, ApplicationContextAware, InitializingBean, DisposableBean {}

First look at the afterPropertiesSet() method of the InitializingBean interface
Method to store the configuration information in the member variable of ReferenceConfig
If it is not lazy to load, directly call the getObject() method to create the object

      Map<String, ConsumerConfig> consumerConfigMap = applicationContext == null ? null : BeanFactoryUtils.beansOfTypeIncludingAncestors(applicationContext, ConsumerConfig.class, false, false);
         for (ConsumerConfig config : consumerConfigMap.values()) {
              setConsumer(consumerConfig);
         }

      Map<String, ApplicationConfig> applicationConfigMap = applicationContext == null ? null : BeanFactoryUtils.beansOfTypeIncludingAncestors(applicationContext, ApplicationConfig.class, false, false);
         for (ApplicationConfig config : applicationConfigMap.values()) {
              setApplication(applicationConfig);
         }
      Map<String, ModuleConfig> moduleConfigMap = applicationContext == null ? null : BeanFactoryUtils.beansOfTypeIncludingAncestors(applicationContext, ModuleConfig.class, false, false);
         for (ModuleConfig config : moduleConfigMap.values()) {
              setModule(moduleConfig);
         }
      Map<String, RegistryConfig> registryConfigMap = applicationContext == null ? null : BeanFactoryUtils.beansOfTypeIncludingAncestors(applicationContext, RegistryConfig.class, false, false);
         for (RegistryConfig config : registryConfigMap.values()) {
              if (config.isDefault() == null || config.isDefault().booleanValue()) {
                  registryConfigs.add(config);
              }
         }
         if (registryConfigs != null && !registryConfigs.isEmpty()) {
              super.setRegistries(registryConfigs);
         }
      Map<String, MonitorConfig> monitorConfigMap = applicationContext == null ? null : BeanFactoryUtils.beansOfTypeIncludingAncestors(applicationContext, MonitorConfig.class, false, false);
         for (MonitorConfig config : monitorConfigMap.values()) {
              setMonitor(monitorConfig);
         }
         Boolean b = isInit();
        if (b == null && getConsumer() != null) {
              b = getConsumer().isInit();
        }
        if (b != null && b.booleanValue()) {
              getObject();
        }

Next, look at the interface implementation of FactoryBean. Since it is of FactoryBean type, let's see what objects it can create
The code is as follows

    @Override
    public Object getObject() throws Exception {
        return get();
    }

    //Returns the type of object created
    @Override
    public Class<?> getObjectType() {
        //The type of this object is interfaceClass in ReferenceConfig class
        //<dubbo:reference id="xxxService" interface="com.xxx.XxxService" />
        //That is, in xml dubbo:reference   The value of the interface property of
        return getInterfaceClass();
    }

    @Override
    @Parameter(excluded = true)
    //Whether the created object is singleton
    public boolean isSingleton() {
        return true;
    }

Let's analyze get()
The code is as follows

public synchronized T get() {
    if (destroyed) {
        throw new IllegalStateException("Already destroyed!");
    }
    //This ref is the object to be returned
    if (ref == null) {
        //Create this object without it
        init();
    }
    return ref;
}

Let's analyze the init() method
The main codes are as follows

    // Get the property value corresponding to the interface name from the system variable
    String resolve = System.getProperty(interfaceName);
    String resolveFile = null;
    if (resolve == null || resolve.length() == 0) {
        // Get resolution file path from system properties
        resolveFile = System.getProperty("dubbo.resolve.file");
        if (resolveFile == null || resolveFile.length() == 0) {
            // Load configuration file from specified location
            File userResolveFile = new File(new File(System.getProperty("user.home")), "dubbo-resolve.properties");
            if (userResolveFile.exists()) {
                // Get file absolute path
                resolveFile = userResolveFile.getAbsolutePath();
            }
        }
        if (resolveFile != null && resolveFile.length() > 0) {
            Properties properties = new Properties();
            FileInputStream fis = null;
            try {
                fis = new FileInputStream(new File(resolveFile));
                // Load configuration from file
                properties.load(fis);
            } catch (IOException e) {
                throw new IllegalStateException("Unload ..., cause:...");
            } finally {
                try {
                    if (null != fis) fis.close();
                } catch (IOException e) {
                    logger.warn(e.getMessage(), e);
                }
            }
            // Get the configuration corresponding to the interface name
            resolve = properties.getProperty(interfaceName);
        }
    }
    
	//Create the invoker object corresponding to the interfaceClass interface and cache it
	String[] methods = Wrapper.getWrapper(interfaceClass).getMethodNames();
	//Build a map and store a series of parameters
	appendParameters(map, application);
	appendParameters(map, module);
	appendParameters(map, consumer, Constants.DEFAULT_KEY);
	appendParameters(map, this);
	//Create the final desired object through map
	ref = createProxy(map);

The above code loads the file for
By system variable or dubbo.properties The configuration file populates the fields of ConsumerConfig

2. Create invoker for service reference

Next, analyze the methods in createProxy(map)
The codes are as follows (omitting non major codes)

private T createProxy(Map<String, String> map) {
     // If the user explicitly configured scope=local, isInjvmRefer returns true
    boolean isJvmRefer=InjvmProtocol.getInjvmProtocol().isInjvmRefer(tmpUrl);
    //Local service reference
    if (isJvmRefer) {
        //Constants.LOCAL_ The value of protocol is injvm
        //So the back refprotocol.refer(interfaceClass, url) calls the InjvmProtocol class
        URL url = new URL(Constants.LOCAL_PROTOCOL, NetUtils.LOCALHOST, 0, interfaceClass.getName()).addParameters(map);
        //Create an invoker that calls the local service
        //Call the refer method of InjvmProtocol
        invoker = refprotocol.refer(interfaceClass, url);
    } else {
       // The url is not empty, indicating that the user may want to make a point-to-point call
	   if (url != null && url.length() > 0) { // user specified URL, could be peer-to-peer address, or register center's address.
             String[] us = Constants.SEMICOLON_SPLIT_PATTERN.split(url);
             if (us != null && us.length > 0) {
                    for (String u : us) {
                        URL url = URL.valueOf(u);
                        if (url.getPath() == null || url.getPath().length() == 0) {
                            url = url.setPath(interfaceName);
                        }
                        if (Constants.REGISTRY_PROTOCOL.equals(url.getProtocol())) {
                            urls.add(url.addParameterAndEncoded(Constants.REFER_KEY, StringUtils.toQueryString(map)));
                        } else {
                            urls.add(ClusterUtils.mergeUrl(url, map));
                        }
                    }
                }
            } else { 
             //Load registry
             //At the same time, set Registry to protocol for later calls refprotocol.refer The implementation class of () is RegistryProtocol 
                List<URL> us = loadRegistries(false);
                if (us != null && !us.isEmpty()) {
                    for (URL u : us) {
                        URL monitorUrl = loadMonitor(u);
                        if (monitorUrl != null) {
                            map.put(Constants.MONITOR_KEY, URL.encode(monitorUrl.toFullString()));
                        }
                        urls.add(u.addParameterAndEncoded(Constants.REFER_KEY, StringUtils.toQueryString(map)));
                    }
                }
       }
		
        // The producer has only one service provider
        if (urls.size() == 1) {
            //Create the invoker that calls the remote service. The invoker type is dubboinvoker
            // Calling refer of RegistryProtocol to build Invoker instance
            invoker = refprotocol.refer(interfaceClass, urls.get(0));
        } else {
            // Multiple service providers, stored in the list
            List<Invoker<?>> invokers = new ArrayList<Invoker<?>>();
            URL registryURL = null;
            for (URL url : urls) {
                invokers.add(refprotocol.refer(interfaceClass, url));
                if (Constants.REGISTRY_PROTOCOL.equals(url.getProtocol())) {
                    registryURL = url; // use last registry url
                }
            }
            if (registryURL != null) { 
                URL u = registryURL.addParameter(Constants.CLUSTER_KEY, AvailableCluster.NAME);
                //Select a dubboinvoker according to cluster management
                invoker = cluster.join(new StaticDirectory(u, invokers));
            } else { // not a registry url
                invoker = cluster.join(new StaticDirectory(invokers));
            }
        }
    }
    // create service proxy
    //Here, the proxy object is created by invoker
    return (T) proxyFactory.getProxy(invoker);
}

2.1 local service reference invoker creation

Let's take a look at the refprotocol.refer (interface class, URL) method
The code is as follows

public <T> Invoker<T> refer(Class<T> serviceType, URL url) throws RpcException {
    return new InjvmInvoker<T>(serviceType, url, url.getServiceKey(), exporterMap);
}

Creating the InjvmInvoker object directly is quite simple

2.2 remote service reference invoker creation

In Dubbo protocol refprotocol.refer (interface class, URLs) method

Now let's look at the Dubbo protocol refprotocol.refer (interface class, URLs) method
The code is as follows

public <T> Invoker<T> refer(Class<T> serviceType, URL url) throws RpcException {
    optimizeSerialization(url);
    //Create an invoker of DubboInvoker type. The invoke method of this object will send the call information to the netty server
    DubboInvoker<T> invoker = new DubboInvoker<T>(serviceType, url, getClients(url), invokers);
    invokers.add(invoker);
    return invoker;
}

The main method below is getClients(url). The code is as follows

private ExchangeClient[] getClients(URL url) {
        // whether to share connection
        boolean service_share_connect = false;
        int connections = url.getParameter(Constants.CONNECTIONS_KEY, 0);
        // if not configured, connection is shared, otherwise, one connection for one service
        if (connections == 0) {
            service_share_connect = true;
            connections = 1;
        }

        ExchangeClient[] clients = new ExchangeClient[connections];
        for (int i = 0; i < clients.length; i++) {
            if (service_share_connect) {
                // Get shared client
                clients[i] = getSharedClient(url);
            } else {
                clients[i] = initClient(url);
            }
        }
        return clients;
    }

Next, analyze getSharedClient(url); methods
The code is as follows

private ExchangeClient getSharedClient(URL url) {
        String key = url.getAddress();
        ReferenceCountExchangeClient client = referenceClientMap.get(key);
        if (client != null) {
            if (!client.isClosed()) {
                client.incrementAndGetCount();
                return client;
            } else {
                referenceClientMap.remove(key);
            }
        }

        locks.putIfAbsent(key, new Object());
        synchronized (locks.get(key)) {
            if (referenceClientMap.containsKey(key)) {
                return referenceClientMap.get(key);
            }
			//Initialize client
            ExchangeClient exchangeClient = initClient(url);
            client = new ReferenceCountExchangeClient(exchangeClient, ghostClientMap);
            referenceClientMap.put(key, client);
            ghostClientMap.remove(key);
            locks.remove(key);
            return client;
        }
    }

Next, analyze the initClient(url) method
The code is as follows

   if (url.getParameter(Constants.LAZY_CONNECT_KEY, false)) {
        client = new LazyConnectExchangeClient(url, requestHandler);
   } else {
       //The requestHandler here is the ExchangeHandlerAdapter() in the DubboProtocol class
       //This handler is used to call the service
        client = Exchangers.connect(url, requestHandler);
   }

Next research Exchangers.connect(url, requestHandler) method,
Skip the calling code and directly enter the connect (URL, handler) method of HeaderExchanger class

  public ExchangeClient connect(URL url, ExchangeHandler handler) throws RemotingException {
        //At this point, the handler is wrapped as
        //new DecodeHandler(new HeaderExchangeHandler(new ExchangeHandlerAdapter()))
        return new HeaderExchangeClient(Transporters.connect(url, new DecodeHandler(new HeaderExchangeHandler(handler))), true);
 }

The next analysis Transporters.connect (URL, handler) method
Skip calling code to enter the connect(url, listener) method of NettyTransporter class
The code is as follows

public Client connect(URL url, ChannelHandler listener) throws RemotingException {
    return new NettyClient(url, listener);
}

What did the new NettyClient(url, listener) do next
The code is as follows

 public NettyClient(final URL url, final ChannelHandler handler) throws RemotingException {
       //The handler is wrapped
        super(url, wrapChannelHandler(url, handler));
    }

Let's first look at the wrappers wrapChannelHandler(url,handler) makes for the handler, and then look at the construction method of the parent class of NettyClient. The code of wrapChannelHandler(url,handler) is as follows

protected static ChannelHandler wrapChannelHandler(URL url, ChannelHandler handler) {
    url = ExecutorUtil.setThreadName(url, CLIENT_THREAD_POOL_NAME);
    url = url.addParameterIfAbsent(Constants.THREADPOOL_KEY, Constants.DEFAULT_CLIENT_THREADPOOL);
    return ChannelHandlers.wrap(handler, url);
}

The next analysis ChannelHandlers.wrap(handler, url) method
The code is as follows

public class ChannelHandlers {

    private static ChannelHandlers INSTANCE = new ChannelHandlers();

    public static ChannelHandler wrap(ChannelHandler handler, URL url) {
        return ChannelHandlers.getInstance().wrapInternal(handler, url);
    }

    protected static ChannelHandlers getInstance() {
        return INSTANCE;
    }
    protected ChannelHandler wrapInternal(ChannelHandler handler, URL url) {
        //ExtensionLoader.getExtensionLoader(Dispatcher.class)
        //The implementation class of. Getadapteextension(). Dispatch (handler, URL) is temporarily considered as AllDispatcher
        //new AllChannelHandler(handler) is used in AllDispatcher to wrap the handler
        //Now the handler is wrapped like this
        //new MultiMessageHandler(new HeartbeatHandler(new AllChannelHandler(
        //new DecodeHandler(new HeaderExchangeHandler(new ExchangeHandlerAdapter())))))
        return new MultiMessageHandler(new HeartbeatHandler(ExtensionLoader.getExtensionLoader(Dispatcher.class)
                .getAdaptiveExtension().dispatch(handler, url)));
    }

This is the current handler type
new MultiMessageHandler(new HeartbeatHandler(new AllChannelHandler(new DecodeHandler(new HeaderExchangeHandler(new ExchangeHandlerAdapter())))))
It looks disgusting, but he just needs to use such a handler. We need to remember him and analyze later

After seeing the handler's wrapper, let's see the construction method of the NettyClient's parent class
The main codes are as follows

        try {
            //Create bootstrap configuration information
            doOpen();
        } catch (Throwable t) {
            close();
        }
        try {
            // Connect according to the bootstrap configuration information created
            connect();
        } catch (RemotingException t) {
            if (url.getParameter(Constants.CHECK_KEY, true)) {
                close();
                throw t;
            }
 executor = (ExecutorService) ExtensionLoader.getExtensionLoader(DataStore.class)
                .getDefaultExtension().get(Constants.CONSUMER_SIDE, Integer.toString(url.getPort()));
 ExtensionLoader.getExtensionLoader(DataStore.class)
                .getDefaultExtension().remove(Constants.CONSUMER_SIDE, Integer.toString(url.getPort()));

Let's take a look at the Dopen () method

 protected void doOpen() throws Throwable {
        NettyHelper.setNettyLoggerFactory();
        bootstrap = new ClientBootstrap(channelFactory);
        // config
        // @see org.jboss.netty.channel.socket.SocketChannelConfig
        bootstrap.setOption("keepAlive", true);
        bootstrap.setOption("tcpNoDelay", true);
        bootstrap.setOption("connectTimeoutMillis", getTimeout());
        //Once again, wrap the handler. Now, the handler wraps the following information
        
        //new NettyHandler(new MultiMessageHandler(new HeartbeatHandler(new  
        //AllChannelHandler(new DecodeHandler(new 
        //HeaderExchangeHandler(new ExchangeHandlerAdapter())))))
        final NettyHandler nettyHandler = new NettyHandler(getUrl(), this);
        bootstrap.setPipelineFactory(new ChannelPipelineFactory() {
            @Override
            public ChannelPipeline getPipeline() {
                NettyCodecAdapter adapter = new NettyCodecAdapter(getCodec(), getUrl(), NettyClient.this);
                ChannelPipeline pipeline = Channels.pipeline();
                pipeline.addLast("decoder", adapter.getDecoder());
                pipeline.addLast("encoder", adapter.getEncoder());
                pipeline.addLast("handler", nettyHandler);
                return pipeline;
            }
        });
    }

The above code only creates objects without connection
Let's take a look at the connect() method. The code is as follows

 protected void doConnect() throws Throwable {
        long start = System.currentTimeMillis();
        //Connect to the server
        ChannelFuture future = bootstrap.connect(getConnectAddress());
        try {
            boolean ret = future.awaitUninterruptibly(getConnectTimeout(), TimeUnit.MILLISECONDS);
            if (ret && future.isSuccess()) {
                Channel newChannel = future.getChannel();
                newChannel.setInterestOps(Channel.OP_READ_WRITE);
                try {
                    // Close old channel
                    Channel oldChannel = NettyClient.this.channel; // copy reference
                    if (oldChannel != null) {
                        try {
                            oldChannel.close();
                        } finally {
                            NettyChannel.removeChannelIfDisconnected(oldChannel);
                        }
                    }
                } finally {
                    if (NettyClient.this.isClosed()) {
                        try {
                            }
                            newChannel.close();
                        } finally {
                            NettyClient.this.channel = null;
                            NettyChannel.removeChannelIfDisconnected(newChannel);
                        }
                    } else {
                        NettyClient.this.channel = newChannel;
                    }
                }
            } 
        } finally {
            if (!isConnected()) {
                future.cancel();
            }
        }
    }

So the client's server is up
We need to remember that the handler wrapper information of the client (the same as that of the server) has
new NettyHandler(new MultiMessageHandler(new HeartbeatHandler(new AllChannelHandler(new DecodeHandler(new HeaderExchangeHandler(new ExchangeHandlerAdapter())))))

In RegistryProtocol refprotocol.refer (interface class, URLs) method

Next, we analyze the refer method of the RegistryProtocol class
The code is as follows

  public <T> Invoker<T> refer(Class<T> type, URL url) throws RpcException {
  
        url = url.setProtocol(url.getParameter(Constants.REGISTRY_KEY, Constants.DEFAULT_REGISTRY)).removeParameter(Constants.REGISTRY_KEY);
        Registry registry = registryFactory.getRegistry(url);
        if (RegistryService.class.equals(type)) {
            return proxyFactory.getInvoker((T) registry, type, url);
        }
        Map<String, String> qs = StringUtils.parseQueryString(url.getParameterAndDecoded(Constants.REFER_KEY));
        String group = qs.get(Constants.GROUP_KEY);
        if (group != null && group.length() > 0) {
            if ((Constants.COMMA_SPLIT_PATTERN.split(group)).length > 1
                    || "*".equals(group)) {
                return doRefer(getMergeableCluster(), registry, type, url);
            }
        }
        return doRefer(cluster, registry, type, url);
    }

Let's look at the dorfer method

private <T> Invoker<T> doRefer(Cluster cluster, Registry registry, Class<T> type, URL url) {
    // Create a RegistryDirectory instance
    RegistryDirectory<T> directory = new RegistryDirectory<T>(type, url);
    // Set up registry and protocol
    directory.setRegistry(registry);
    directory.setProtocol(protocol);
    Map<String, String> parameters = new HashMap<String, String>(directory.getUrl().getParameters());
    // Generate url
    URL subscribeUrl = new URL(Constants.CONSUMER_PROTOCOL, parameters.remove(Constants.REGISTER_IP_KEY), 0, type.getName(), parameters);

    // Register the service consumer and create a new node in the consumers directory
    if (!Constants.ANY_VALUE.equals(url.getServiceInterface())
            && url.getParameter(Constants.REGISTER_KEY, true)) {
        registry.register(subscribeUrl.addParameters(Constants.CATEGORY_KEY, Constants.CONSUMERS_CATEGORY,
                Constants.CHECK_KEY, String.valueOf(false)));
    }

    // Subscribe to node data such as providers, configurators, routers, etc
    directory.subscribe(subscribeUrl.addParameter(Constants.CATEGORY_KEY,
            Constants.PROVIDERS_CATEGORY
                    + "," + Constants.CONFIGURATORS_CATEGORY
                    + "," + Constants.ROUTERS_CATEGORY));

    // There may be multiple service providers in a registry, so you need to merge multiple service providers into one
    Invoker invoker = cluster.join(directory);
    ProviderConsumerRegTable.registerConsumer(invoker, url, subscribeUrl, directory);
    return invoker;
}

As mentioned above, the dorfer method creates a RegistryDirectory instance (RegistryDirectory synchronizes the service information of the producer, creates the corresponding DubboInvoker object according to the service that can be called and stores it in RegistryDirectory), generates the consumer url, and registers it with the registry. After registration, the data under the subscription providers, configurators, routers and other nodes will be subscribed.
Since a service may be deployed on multiple servers, the producer will register multiple services in the registry. At this time, the Cluster needs to select an invoker from multiple services to return

3. Create proxy object through invoker

Next, we study ReferenceConfig's proxyFactory.getProxy(invoker) method, which is to make the next proxy for the invoker
Next, let's look at the getProxy method of JavassistProxyFactory

public <T> T getProxy(Invoker<T> invoker, Class<?>[] interfaces) {
    return (T) Proxy.getProxy(interfaces).newInstance(new InvokerInvocationHandler(invoker));
}

The last generated proxy class looks like this

package org.apache.dubbo.common.bytecode;

public class proxy0 implements org.apache.dubbo.demo.DemoService {

    public static java.lang.reflect.Method[] methods;

    private java.lang.reflect.InvocationHandler handler;

    public proxy0() {
    }

    public proxy0(java.lang.reflect.InvocationHandler arg0) {
        handler = $1;
    }

    public java.lang.String sayHello(java.lang.String arg0) {
        Object[] args = new Object[1];
        args[0] = ($w) $1;
        Object ret = handler.invoke(this, methods[0], args);
        return (java.lang.String) ret;
    }
}

Cluster fault tolerance

The official document says that cluster fault tolerance is divided into
Service Directory, service routing Router, Cluster cluster, load balancing

What is the service Directory: when the producer starts, it stores the services that can be called on the zk. After the consumer starts, he pulls the services that can be called on zk, creates an invoker object according to each service information, stores all the invokers in the service Directory, and the information stored in the Directory will change with the service information provided by the producer.

The role of service routing Router: service routing specifies which service providers service consumers can call.
The service route contains a routing rule, which determines the call target of the service consumer. The conditional routing rule information is as follows

host = 10.20.153.10 => host = 10.20.153.11
[service consumer ip] = > [service provider ip]

Explain that the service consumer ip can call the ip of this service provider

Cluste r: in dubbo, the purpose of cluster is to merge multiple service providers into a Cluster Invoker and expose the Invoker to service consumers. In this way, the service consumer only needs to make a remote call through the Invoker. As for which service provider to call and how to handle printing logs or throwing exceptions after the call fails, it is now up to the cluster module to handle it

Load balanceloadbalance: when calling, select an invoker from multiple invokers by weight and policy parameters

Service Directory

Generally, RegistryDirectory is used, and the class diagram is as follows


Directory inherits from Node interface, which contains a method getUrl to get configuration information. The class implementing the interface can provide configuration information to the outside.
AbstractDirectory implements the Directory interface, which contains an important method definition, list(Invocation), which is used to get all the invoke objects. The invoke object here can call the producer's services.
RegistryDirectory implements the NotifyListener interface. When the node information of the registry changes, you can get the change invoke information through this interface method

Where is RegistryDirectory initialized? It must be used by consumers
Initialization in the doRefer method of RegistryProtocol

 private <T> Invoker<T> doRefer(Cluster cluster, Registry registry, Class<T> type, URL url) {
        //Initialization here
        RegistryDirectory<T> directory = new RegistryDirectory<T>(type, url);
        directory.setRegistry(registry);
        directory.setProtocol(protocol);
        Map<String, String> parameters = new HashMap<String, String>(directory.getUrl().getParameters());
        URL subscribeUrl = new URL(Constants.CONSUMER_PROTOCOL, parameters.remove(Constants.REGISTER_IP_KEY), 0, type.getName(), parameters);
        if (!Constants.ANY_VALUE.equals(url.getServiceInterface())
                && url.getParameter(Constants.REGISTER_KEY, true)) {
            registry.register(subscribeUrl.addParameters(Constants.CATEGORY_KEY, Constants.CONSUMERS_CATEGORY,
                    Constants.CHECK_KEY, String.valueOf(false)));
        }
        //Subscribe to the services provided by the server
        directory.subscribe(subscribeUrl.addParameter(Constants.CATEGORY_KEY,
                Constants.PROVIDERS_CATEGORY
                        + "," + Constants.CONFIGURATORS_CATEGORY
                        + "," + Constants.ROUTERS_CATEGORY));

        Invoker invoker = cluster.join(directory);
        ProviderConsumerRegTable.registerConsumer(invoker, url, subscribeUrl, directory);
        return invoker;
    }

Next, we analyze how RegistryDirectory synchronizes producer service information in zk into each invoker object
First look at the list method of the parent class, which can return all the invoker objects. The code is as follows

//All invocation s can be obtained
public List<Invoker<T>> list(Invocation invocation) throws RpcException {
    if (destroyed) {
        throw new RpcException("Directory already destroyed .url: " + getUrl());
    }
    //Template method calls the doList method of subclass
    //In the doList method of RegistryDirectory, if the service is not disabled, get it from the member variable (map < string, list < invoker < T > > > methodinvokermap)
    List<Invoker<T>> invokers = doList(invocation);
    List<Router> localRouters = this.routers; // local reference
    if (localRouters != null && !localRouters.isEmpty()) {
        for (Router router : localRouters) {
            try {
                if (router.getUrl() == null || router.getUrl().getParameter(Constants.RUNTIME_KEY, false)) {
                    //Routing
                    invokers = router.route(invokers, getConsumerUrl(), invocation);
                }
            } catch (Throwable t) {
                logger.error("Failed to execute router: " + getUrl() + ", cause: " + t.getMessage(), t);
            }
        }
    }
    return invokers;
}

Next, look at the notify method of RegistryDirectory. If the service information provided by the producer in zk changes, this method will be called. In the method, the corresponding invoker object will be synchronized according to the changed information. The code is as follows

public synchronized void notify(List<URL> urls) {
        List<URL> invokerUrls = new ArrayList<URL>();
        List<URL> routerUrls = new ArrayList<URL>();
        List<URL> configuratorUrls = new ArrayList<URL>();
        for (URL url : urls) {
            String protocol = url.getProtocol();
            String category = url.getParameter(Constants.CATEGORY_KEY, Constants.DEFAULT_CATEGORY);
            if (Constants.ROUTERS_CATEGORY.equals(category)
                    || Constants.ROUTE_PROTOCOL.equals(protocol)) {
                routerUrls.add(url);
            } else if (Constants.CONFIGURATORS_CATEGORY.equals(category)
                    || Constants.OVERRIDE_PROTOCOL.equals(protocol)) {
                configuratorUrls.add(url);
            } else if (Constants.PROVIDERS_CATEGORY.equals(category)) {
                invokerUrls.add(url);
            } else {
                logger.warn("Unsupported category " + category + " in notified url: " + url + " from registry " + getUrl().getAddress() + " to consumer " + NetUtils.getLocalHost());
            }
        }
        // configurators
        if (configuratorUrls != null && !configuratorUrls.isEmpty()) {
            this.configurators = toConfigurators(configuratorUrls);
        }
        // routers
        if (routerUrls != null && !routerUrls.isEmpty()) {
            List<Router> routers = toRouters(routerUrls);
            if (routers != null) { // null - do nothing
                setRouters(routers);
            }
        }
        List<Configurator> localConfigurators = this.configurators; // local reference
        // merge override parameters
        this.overrideDirectoryUrl = directoryUrl;
        if (localConfigurators != null && !localConfigurators.isEmpty()) {
            for (Configurator configurator : localConfigurators) {
                this.overrideDirectoryUrl = configurator.configure(overrideDirectoryUrl);
            }
        }
        // providers
        refreshInvoker(invokerUrls);
    }

The main method of the above code is refreshinvoker (invokerurl)
The code is as follows

private void refreshInvoker(List<URL> invokerUrls) {
        if (invokerUrls != null && invokerUrls.size() == 1 && invokerUrls.get(0) != null
                && Constants.EMPTY_PROTOCOL.equals(invokerUrls.get(0).getProtocol())) {
            this.forbidden = true; // Forbid to access
            this.methodInvokerMap = null; // Set the method invoker map to null
            destroyAllInvokers(); // Close all invokers
        } else {
            this.forbidden = false; // Allow to access
            Map<String, Invoker<T>> oldUrlInvokerMap = this.urlInvokerMap; // local reference
            if (invokerUrls.isEmpty() && this.cachedInvokerUrls != null) {
                invokerUrls.addAll(this.cachedInvokerUrls);
            } else {
                this.cachedInvokerUrls = new HashSet<URL>();
                this.cachedInvokerUrls.addAll(invokerUrls);//Cached invoker urls, convenient for comparison
            }
            if (invokerUrls.isEmpty()) {
                return;
            }
            Map<String, Invoker<T>> newUrlInvokerMap = toInvokers(invokerUrls);// Translate url list to Invoker map
            Map<String, List<Invoker<T>>> newMethodInvokerMap = toMethodInvokers(newUrlInvokerMap); // Change method name to map Invoker Map
            // state change
            // If the calculation is wrong, it is not processed.
            if (newUrlInvokerMap == null || newUrlInvokerMap.size() == 0) {
                logger.error(new IllegalStateException("urls to invokers error .invokerUrls.size :" + invokerUrls.size() + ", invoker.size :0. urls :" + invokerUrls.toString()));
                return;
            }
            this.methodInvokerMap = multiGroup ? toMergeMethodInvokerMap(newMethodInvokerMap) : newMethodInvokerMap;
            this.urlInvokerMap = newUrlInvokerMap;
            try {
                destroyUnusedInvokers(oldUrlInvokerMap, newUrlInvokerMap); // Close the unused Invoker
            } catch (Exception e) {
                logger.warn("destroyUnusedInvokers error. ", e);
            }
        }
    }

The above code is to convert url information into invoker and store it in the member variables of methodInvokerMap and urlInvokerMap

Service route

When do we need to use Router? When we need to configure the white list blacklist of dubbo service or a service to provide to different consumers. When the service dictionary obtains all the invokers, it will filter out the unavailable invokers through the route
Take a look at the Router diagram


There are two implementation classes
ConditionRouter and ScriptRouter
Let's look at ConditionRouter first
The construction method code of ConditionRouter is as follows

public ConditionRouter(URL url) {
    this.url = url;
    // Get priority and force configuration
    this.priority = url.getParameter(Constants.PRIORITY_KEY, 0);
    this.force = url.getParameter(Constants.FORCE_KEY, false);
    try {
        // Get routing rules
        String rule = url.getParameterAndDecoded(Constants.RULE_KEY);
        if (rule == null || rule.trim().length() == 0) {
            throw new IllegalArgumentException("Illegal route rule!");
        }
        rule = rule.replace("consumer.", "").replace("provider.", "");
        // Positioning = > separator
        int i = rule.indexOf("=>");
        // Get matching rules for service consumers and providers respectively
        String whenRule = i < 0 ? null : rule.substring(0, i).trim();
        String thenRule = i < 0 ? rule.trim() : rule.substring(i + 2).trim();
        // Resolve service consumer matching rules
        Map<String, MatchPair> when = 
            StringUtils.isBlank(whenRule) || "true".equals(whenRule) 
                ? new HashMap<String, MatchPair>() : parseRule(whenRule);
        // Resolve service provider matching rules
        Map<String, MatchPair> then = 
            StringUtils.isBlank(thenRule) || "false".equals(thenRule) 
                ? null : parseRule(thenRule);
        // Assign the resolved matching rules to the when condition and the ncondition member variables respectively
        this.whenCondition = when;
        this.thenCondition = then;
    } catch (ParseException e) {
        throw new IllegalStateException(e.getMessage(), e);
    }
}

According to the above code, according to the incoming url object, the parseRule method is used to parse and assign the parsed information to the member variable. Let's take a look at the parseRule method. The code is as follows

private static Map<String, MatchPair> parseRule(String rule)
        throws ParseException {
    // Defining a set of condition maps
    Map<String, MatchPair> condition = new HashMap<String, MatchPair>();
    if (StringUtils.isBlank(rule)) {
        return condition;
    }
    MatchPair pair = null;
    Set<String> values = null;
    // Matching routing rules through regular expressions, route_ PATTERN = ([&!=,]*)\s*([^&!=,\s]+)
    // This expression does not seem to be well understood. The first expression in parentheses is used to match symbols such as "&", "!", "=" and "," etc.
    // The second bracket is used to match English letters, numbers and other characters. For example:
    //    host = 2.2.2.2 & host != 1.1.1.1 & method = hello
    // The matching results are as follows:
    //     Bracket one bracket two
    // 1.  null       host
    // 2.   =         2.2.2.2
    // 3.   &         host
    // 4.   !=        1.1.1.1 
    // 5.   &         method
    // 6.   =         hello
    final Matcher matcher = ROUTE_PATTERN.matcher(rule);
    while (matcher.find()) {
       	// Get matching results in parenthesis one
        String separator = matcher.group(1);
        // Get the matching result in parenthesis two
        String content = matcher.group(2);
        // The separator is empty, indicating that the matching is at the beginning of the expression
        if (separator == null || separator.length() == 0) {
            // Create MatchPair object
            pair = new MatchPair();
            // Store < match, matchpair > key value pairs, such as < host, matchpair >
            condition.put(content, pair); 
        } 
        
        // If the separator is &, it means that there is a condition next
        else if ("&".equals(separator)) {
            // Try to get MatchPair from condition
            if (condition.get(content) == null) {
                // Did not get MatchPair, recreate one and put it into condition
                pair = new MatchPair();
                condition.put(content, pair);
            } else {
                pair = condition.get(content);
            }
        } 
        
        // The separator is=
        else if ("=".equals(separator)) {
            if (pair == null)
                throw new ParseException("Illegal route rule ...");

            values = pair.matches;
            // Store content in MatchPair's matches collection
            values.add(content);
        } 
        
        //  Separator is= 
        else if ("!=".equals(separator)) {
            if (pair == null)
                throw new ParseException("Illegal route rule ...");

            values = pair.mismatches;
            // Store content in MatchPair's miscatches collection
            values.add(content);
        }
        
        // The separator is,
        else if (",".equals(separator)) {
            if (values == null || values.isEmpty())
                throw new ParseException("Illegal route rule ...");
            // Store the content in the values obtained in the previous step, which may be matches or mismatches
            values.add(content);
        } else {
            throw new ParseException("Illegal route rule ...");
        }
    }
    return condition;
}

What does the above code do? It is to parse the information in the url object, and then return the parsed data
Next, let's look at the route method of ConditionRouter

public <T> List<Invoker<T>> route(List<Invoker<T>> invokers, URL url, Invocation invocation) throws RpcException {
    if (invokers == null || invokers.isEmpty()) {
        return invokers;
    }
    try {
        // First, match the conditions of service consumers. If the matching fails, the url of service consumers does not meet the matching rules,
        // There is no need for subsequent matching, just return to the Invoker list. For example, the following rules:
        //     host = 10.20.153.10 => host = 10.0.0.10
        // This routing rule expects the service consumer with IP 10.20.153.10 to call the service on the machine with IP 10.0.0.10.
        // When the consumer ip is 10.20.153.11, match when returns false, indicating that the current routing rule is not applicable to
        // The current service consumer, at this time, does not need to carry out subsequent matching, just return directly.
        if (!matchWhen(url, invocation)) {
            return invokers;
        }
        List<Invoker<T>> result = new ArrayList<Invoker<T>>();
        // The service provider matching condition is not configured, indicating that the service is disabled for the specified service consumer, that is, the service consumer is in the blacklist
        if (thenCondition == null) {
            logger.warn("The current consumer in the service blacklist...");
            return result;
        }
        // Here we can simply understand Invoker as a service provider. Now we use the service provider matching rule pair 
        // Invoker list for matching
        for (Invoker<T> invoker : invokers) {
            // If the matching is successful, it indicates that the current Invoker conforms to the service provider matching rules.
            // The Invoker is added to the result list
            if (matchThen(invoker.getUrl(), url)) {
                result.add(invoker);
            }
        }
        
        // Returns the matching result. If result is an empty list and force = true, it means that an empty list is forced to be returned,
        // Otherwise, the routing rule with empty routing result will be automatically invalid
        if (!result.isEmpty()) {
            return result;
        } else if (force) {
            logger.warn("The route result is empty and force execute ...");
            return result;
        }
    } catch (Throwable t) {
        logger.error("Failed to execute condition router rule: ...");
    }
    
    // Return as is, when force = false, it means the routing rule is invalid
    return invokers;
}

The above code mainly uses matchWhen and matchThen methods to match the route to see whether the passed parameters can be accessed

summary

dubbo producer initialization process

First, the configuration file is parsed through DubboNamespaceHandler. Each configuration file has a dubbo:service The tag creates a ServiceBean class
Each service bean adds a listener to spring through the setApplicationContext method when the project starts
The listener initializes the producer's services
How the listener initializes the producer's service
First loop through all registries, then through all protocols,
Export each service according to registry and protocol
Create URL object based on configuration information
To export the service, if you want to export the service to the local
-) create the invoker object according to the implementation class ref object and URL object, and cache them
To export the service, if you want to export the service to a remote
-) create the invoker object according to the implementation class ref object and URL object. The invoker is a simple proxy for the ref object
-) create DubboExporter object according to the invoker (used to process the request), and the DubboExporter wraps the invoker object
-) open netty server
-) get the address of the registration center, connect the registration center, check the opening of the registration center, and call back to all listeners in case of any change
-) create a message in the registry for each service
-) and then subscribe to override data

Consumer initialization process

First, the configuration file is parsed through DubboNamespaceHandler. Each configuration file has a dubbo:reference Create a ReferenceBean class
ReferenceBean is a factorybean, through which you can create a singleton proxy class (ref implementation class to call interface)
What kind of proxy object will this factorybean create? Let's analyze it
If it is a local reference, an invoker object is created according to the implementation class (ref)
If it is a remote reference, create a DubboInvoker object based on the implementation class (ref)
- > each service provider creates a DubboInvoker. The invoker method of the DubboInvoker object will call the send method of netty's client (DubboInvoker can make a call request to the server)
- > If a service has multiple providers, multiple dubboinvokers will be created and put into the collection, and then objects under the AbstractClusterInvoker implementation class will be created for cluster management
- > cluster management includes (throwing exception after calling service failure or calling other producers, load balancing policy selection)
Open the netty client, connect the registry, write the service to the registry, and initialize the service dictionary RegistryDirectory
Add listening in the service Dictionary (a DubboInvoker is created for each service on the server side and stored in the service Dictionary).
Create the corresponding proxy object according to the invoker to return. Proxy is to wrap the invoker object.

Service calling procedure

Consumer sends request to producer

DubboProductInterface obj=(DubboProductInterface)application.getBean("DubboProductInterface") 
obj.sayHello();

The above code will create the proxy object of DubboProductInterface interface through getBean() method. Every time the method of this proxy object is called, it will make a remote call. What is the proxy object like?

In the spring container, a proxy object of type DubboProductInterface will be created through the singleton factorybean
 If the producer is not clustered (there is only one provider for a service)
--->In other words, only one producer returns a DubboInvoker object directly based on the service information provided by the producer.
If the producer is a cluster deployment (there are multiple providers for one service)
--->Multiple dubboinvokers will be packaged through AbstractClusterInvoker. After packaging, multiple dubboinvokers will be managed in clusters
 --->(there are several dubboinvokers for a service provided by several producers)
--->Then return the AbstractClusterInvoker object

What does AbstractClusterInvoker's invoke method do?

If the implementation class of the AbstractClusterInvoker called is FailoverClusterInvoker 
His function is to call failure automatically to switch producers, that is, calling producers to call other producers after failure.
The doinvoke method of FailoverClusterInvoker will get the number of retries first (the maximum number of calls after failure)
It is responsible for balancing through the implementation class of LoadBalance (select a return from multiple dubboinvokers),
Call the invoke method in AbstractInvoker, that is, the doinvoke method in DubboInvoker
 If the call fails, it will be called multiple times according to the number of retries. If the number of retries fails, an exception will be thrown.

In DubboInvoker, there are many kinds of calls, including asynchronous return value calls, asynchronous no return values, synchronous calls.
Let's analyze the synchronous call in dobboinvoker's doinvoke method
currentClient.request(inv, timeout).get(); get() here blocks the data waiting to be returned
The calling process is as follows

—> DubboInvoker#doInvoke(Invocation)
  - > referencecounteexchangeclient × request (object, int) - > there is a counter in it
    - > headerexchangeclient (object, int) -- > initialization this.channel = new HeaderExchangeChannel(client);
	                                                     -->Heart rate detection
      - > headerexchangechannel (object, int) -- > package the requested information as a Request object
        —> AbstractPeer#send(Object)				     	 
          - > abstractclient send (object, Boolean) - > connect to Netty server     
		                                                         Store the connected channel object (for example, NioServerSocketChannel)
																 NettyChannel in dubbo
                                       		             -->Get channel                 
														 -->Call the send method of channel     
            —> NettyChannel#send(Object, boolean)        
              —> NioClientSocketChannel#write(Object)    

Coding of parameters sent by consumers to producers

Next, analyze the code calling process

Set the encoder in NettyClient, pipeline.addLast("encoder", adapter.getEncoder());
adapter.getEncoder() creates an internal class InternalEncoder()
encode() method call of InternalEncoder() codec.encode(channel, buffer, msg);
That is, the encodeRequest() method of the encode method of ExchangeCodec
 Send the object information to the server through the channel in the encodeRequest() method

Producer receive request decoding process

The decoding call is as follows

ExchangeCodec -->decode
	ExchangeCodec -->decode(channel, buffer, readable, header)
		ExchangeCodec -->decodeBody(channel, is, header)
			DubboCodec -- "decodeBody()" creates a new DecodeableRpcInvocation() object to return
				DecodeableRpcInvocation    ---> decode()  
				Finally, a DecodeableRpcInvocation object will be returned
 At the end of the decoding process, the sent information is encapsulated into a request object to return, 
The type of this request object is ecodeblerpcinvocation

The producer calls the specified method based on the request information

The next step is to give the request object to the handler for processing. The handler handling the request goes through a series of wrappers. The wrappers are as follows

new NettyHandler(
	new MultiMessageHandler(
		new HeartbeatHandler(
			new AllChannelHandler(
				new DecodeHandler(
					new HeaderExchangeHandler(
						new ExchangeHandlerAdapter()))))))

Let's look at the call stack

NettyHandler - the entry used by the "messagereceived() netty framework to receive requests
	MultiMessageHandler  ->   received
		HeartbeatHandler  ->    received
			AllChannelHandler - received start thread
				ChannelEventRunnable - "run() calls different methods according to the channel type
				    DecodeHandler ->received
						HeaderExchangeHandler - received calls the method, and sends back the result returned by the result of calling the method channel.send(response);
						HeaderExchangeHandler - handlerequest (exchange channel, request); process request
							ExchangeHandlerAdapter ->reply 
							   In other words, DubboProtocol - "reply obtains the corresponding DubboExporter according to the passed parameters
							                                     DubboExporter contains the corresponding invoker
								    AbstractProxyInvoker  ->invoke   								 
											JavassistProxyFactory - "getInvoker can call the specified method

Decoding of parameters sent by consumers to producers

Consumer receives producer response decoding

NettyClient's adapter.getEncoder() method,
Finally, we will go to the encodeResponse(channel, buffer, (Response) msg) method of ExchangeCodec
 Then DubboCodec's decodeBody method returns a new DecodeableRpcResult request object
 DecodeableRpcResult = = "" decode() method, store the parameter in the DecodeableRpcResult object

Consumer receives parameters sent by producer

After decoding, the receiver uses handler to parse DecodeableRpcResult

It's the handler that has been wrapped many times
 Skip some handler s and go directly to the main logic 
HeaderExchangeHandler   received
HeaderExchangeHandler  handleResponse(channel, (Response) message);    
         Remove the message here and put the response parameter passed by the request into the member variable of DefaultFuture
		 Then wake up the user thread

Consumer receives producer response, how to find the thread that consumer initiates the request

Consumer calls producer
When the producer does not return the result of the call, the consumer will block. Where does it block

doinvoke method of DubboInvoker 
    -->  currentClient.request(inv, timeout).get(); get() here will block
 Which is the get method of DefaultFuture

Let's see what DefaultFuture has done

There is a static map in DefaultFuture, which stores the return information request id and DefaultFuture object
 The DefaultFuture static code block will open a daemons thread, and if the corresponding time exceeds a fixed time, the DefaultFuture will be removed from the static map

What does the get method of DefaultFuture do? Simplify the code

 private final Lock lock = new ReentrantLock();
 private final Condition done = lock.newCondition();
 private volatile Response response;
 
 public Object get(int timeout) throws RemotingException {
    //If the return value is empty
    if (!isDone()) {
        long start = System.currentTimeMillis();
        lock.lock();
        try {
            while (!isDone()) {
                //Blocking in lock lock is equivalent to using wait in synchronized and wake up with notify
                //notify in synchronized is a random wake-up and cannot be specified
                //await in lock lock can be specified to wake up
                done.await(timeout, TimeUnit.MILLISECONDS);
                //Whether the response time is exceeded
                if (isDone() || System.currentTimeMillis() - start > timeout) {
                    break;
                }
            }
        } finally {
           //Release lock
            lock.unlock();
        }
    }
    return returnFromResponse();
}

How does the producer return the response to wake up the thread
After decoding, it is handed over to the handler for processing, and the core code is viewed directly
HeaderExchangeHandler —> handleResponse(channel, (Response) message);

static void handleResponse(Channel channel, Response response) throws RemotingException {
    if (response != null && !response.isHeartbeat()) {
        DefaultFuture.received(channel, response);
    }
}

Let's see DefaultFuture.received(channel, response)

public static void received(Channel channel, Response response) {
    try {
        //Get the corresponding DefaultFuture according to the id 
        //Remove DefaultFuture from static map by id
        DefaultFuture future = FUTURES.remove(response.getId());
        if (future != null) {
        	//Call in method done.signal(); wake up
            future.doReceived(response);
        } 
        }
    } finally {
        CHANNELS.remove(response.getId());
    }
}

Let's see future.doReceived(response);

private void doReceived(Response res) {
    lock.lock();
    try {
        response = res;
        if (done != null) {
            //Wake up
            done.signal();
        }
    } finally {
        lock.unlock();
    }
    if (callback != null) {
        invokeCallback(callback);
    }
}

Tags: Dubbo Netty Spring xml

Posted on Thu, 25 Jun 2020 07:08:24 -0400 by jahwobbler