preface

. Net Core provides us with a powerful Configuration configuration system, which is easy to use and highly extensible. Through this Configuration system, we can load data sources such as Json, Xml, Ini into the program, or expand other forms of storage sources by ourselves. What we need to do today is to extend the Etcd data source operation for it in a customized way.

What is Etdc

Before using etcd, let's introduce etcd. I believe many students have heard about it. Etcd It is a highly available and consistent distributed KV storage system, which adopts raft protocol As a consistency algorithm, it is also developed based on GO language. The latest version is v3.4.9. Please refer to the download address for specific version Official GitHub address . I believe that students who have known K8S are familiar with this, it is K8S data management system. The official address is https://etcd.io/.
Before that, I believe you have learned a lot about the storage system. What functions can Etcd achieve? One is for configuration center and service discovery, and the other is for distributed lock and message system. It is based on directory storage, and there is a set of powerful Watch mechanism to monitor the operation changes of nodes and data. Each transaction operation of nodes will have the version information.

Etcd VS Zookeeper

Is it similar to Zookeeper through the above introduction 😂😂😂 There's a lot about Etcd and Zookeeper on the Internet Contrast article , which can be concluded as follows

function	Etcd	Zookeeper
Distributed lock	Yes (node version number information is used)	Yes (temporary nodes and sequential temporary nodes are used)
watcher	Yes	Yes
Consistency algorithm	raft	zab
election	Yes	Yes
Metadata storage	Yes	Yes

Application scenario	Etcd	Zookeeper
Publish and subscribe (configuration center)	Yes (unlimited Watch)	Yes (if it is triggered once, you need to re register the Watch)
Soft load balancing	Yes	Yes
Naming service	Yes	Yes
Service discovery	Yes (based on lease node)	Yes (based on temporary nodes)
Distributed notification / coordination	Yes	Yes
Cluster management and Master election	Yes	Yes
Distributed lock	Yes	Yes
Distributed queue	Yes	Yes

To put it bluntly, Zookeeper is capable, Etcd is also capable. Now that we have Zookeeper, we have to choose Etcd, mainly for the following reasons

• Lighter (Etcd is developed based on GO language, Zookeeper is developed based on Java), easier to use (out of the box)
• Stable reading and writing under high load
• Multi version concurrency control of data model
• Stable watcher function to inform subscribers of the change of monitoring value (Zookeeper's data-based monitoring is one-time, and needs to be re registered after each monitoring)
• The client protocol uses GRPC protocol and supports more languages

In a word, it not only realizes the function of Zookeeper, but also hangs Zookeeper in many aspects 😏😏😏 I can't help trying such a powerful thing.

Use Etcd in. Net Core

Many Etcd client drivers of. Net Core can be found on Nuget. I used the name with the most downloads dotnet-etcd By the way, it's on the GayHub. I'm sorry I got it wrong 😱😱😱 Project address on GitHub I have learned about the basic usage. In fact, we only need two functions to configure the Configuration. One is to Get the data, the other is to change the Watch node (it will be used to update the data). In my opinion, it's very important to study with purpose and boundary in the early stage.

Configuration extension Etcd

We have mentioned before that custom extension Configuration is very convenient. I believe I have learned about it Configuration related source code We are very familiar with each other. We can roughly summarize the following three steps:

• To write the IConfigurationBuilder extension method, we call it AddEtcd
• Write the configuration source information class that implements IConfigurationSource. We call it EtcdConfigurationSource
• Write the configuration data provider class of the ConfigurationSource inherited from the ConfigurationProvider. We call it EtcdConfigurationProvider

Because Microsoft has provided us with some convenience, it is very simple to write. OK, let's start to write the specific implementation code. I'll explain the key points in the code.
First, define the extension class EtcdConfigurationExtensions, which is an extension method for IConfigurationBuilder. The implementation is as follows

public static class EtcdConfigurationExtensions
{
    /// <summary>
    ///AddEtcd extension method
    /// </summary>
    ///< param name = "serveraddress" > etcd address < / param >
    ///< param name = "path" > read path < / param >
    /// <returns></returns>
    public static IConfigurationBuilder AddEtcd(this IConfigurationBuilder builder, string serverAddress,string path)
    {
        return AddEtcd(builder, serverAddress:serverAddress, path: path,reloadOnChange: false);
    }

    /// <summary>
    ///AddEtcd extension method
    /// </summary>
    ///< param name = "serveraddress" > etcd address < / param >
    ///< param name = "path" > read path < / param >
    ///< param name = "reloadonchange" > refresh if data transmission changes < / param >
    /// <returns></returns>
    public static IConfigurationBuilder AddEtcd(this IConfigurationBuilder builder, string serverAddress, string path, bool reloadOnChange)
    {
        return AddEtcd(builder,options => {
            options.Address = serverAddress;
            options.Path = path;
            options.ReloadOnChange = reloadOnChange;
        });
    }

    public static IConfigurationBuilder AddEtcd(this IConfigurationBuilder builder, Action<EtcdOptions> options)
    {
        EtcdOptions etcdOptions = new EtcdOptions();
        options.Invoke(etcdOptions);
        return builder.Add(new EtcdConfigurationSource { EtcdOptions = etcdOptions });
    }
}

Here I also define a POCO of EtcdOptions to carry the configuration properties of reading Etcd

public class EtcdOptions
{
    /// <summary>
    ///Etcd address
    /// </summary>
    public string Address { get; set; }

    /// <summary>
    ///Etcd access user name
    /// </summary>
    public string UserName { get; set; }

    /// <summary>
    ///Etcd access code
    /// </summary>
    public string PassWord { get; set; }

    /// <summary>
    ///Etcd read path
    /// </summary>
    public string Path { get; set; }

    /// <summary>
    ///Whether to refresh and read data changes
    /// </summary>
    public bool ReloadOnChange { get; set; }
}

Next, we define EtcdConfigurationSource, which is very simple to return a configuration providing object

public class EtcdConfigurationSource : IConfigurationSource
{
    public EtcdOptions EtcdOptions { get; set; }

    public IConfigurationProvider Build(IConfigurationBuilder builder)
    {
        return new EtcdConfigurationProvider(EtcdOptions);
    }
}

The real read operations are all in the EtcdConfigurationProvider

public class EtcdConfigurationProvider : ConfigurationProvider
{
    private readonly string _path;
    private readonly bool _reloadOnChange;
    private readonly EtcdClient _etcdClient;

    public EtcdConfigurationProvider(EtcdOptions options)
    {
        //Instantiate EtcdClient
        _etcdClient = new EtcdClient(options.Address,username: options.UserName,password: options.PassWord);
        _path = options.Path;
        _reloadOnChange = options.ReloadOnChange;
    }

    /// <summary>
    ///Override load method
    /// </summary>
    public override void Load()
    {
        //Read data
        LoadData();
        //Whether the data changes and reloads
        if (_reloadOnChange)
        {
            ReloadData();
        }
    }

    private void LoadData()
    {
        //Read data from Etcd
        string result = _etcdClient.GetValAsync(_path).GetAwaiter().GetResult();
        if (string.IsNullOrEmpty(result))
        {
            return;
        }
        //Transform the data structure. Here I use the json format
        //As long as the read Data is assigned to the Data attribute, the real Data read by IConfiguration is the dictionary Data stored in the Data
        Data = ConvertData(result);
    }

    private IDictionary<string,string> ConvertData(string result)
    {
        byte[] array = Encoding.UTF8.GetBytes(result);
        MemoryStream stream = new MemoryStream(array);
        //JsonConfigurationFileParser is used to convert json data into a Configuration readable structure (copied from the JsonConfiguration class library 😄😄😄 )
        return JsonConfigurationFileParser.Parse(stream);
    }

    private void ReloadData()
    {
        WatchRequest request = new WatchRequest()
        {
            CreateRequest = new WatchCreateRequest()
            {
                //You need to convert a format, because the interface of etcd v3 is included in the definition of grpc
                Key = ByteString.CopyFromUtf8(_path)
            }
        };
        //Monitor the change of Etcd node, obtain the change data and update the configuration
        _etcdClient.Watch(request, rsp =>
        {
            if (rsp.Events.Any())
            {
                var @event = rsp.Events[0];
                //You need to convert a format, because the interface of etcd v3 is included in the definition of grpc
                Data = ConvertData(@event.Kv.Value.ToStringUtf8());
                //ConfigurationReloadToken notification needs to be triggered by calling the OnReload method of ConfigurationProvider
                //This allows you to send data change notifications to classes that use Configuration
                //For example, the IOptionsMonitor notifies you of data changes through the ConfigurationReloadToken
                OnReload();
            }
        });
    }
}

Use as follows

builder.AddEtcd("http://127.0.0.1:2379", "service/mydemo", true);

By the way, I would like to recommend an Etcd visual management tool ETCD Manager In order to learn Etcd better.
It's basically over here, isn't it very simple. It's mainly that the design idea of Configuration itself is relatively clear, so it's easy to realize.

summary

All the above codes have been uploaded My GitHub The warehouse also extends the reading of other data sources, such as Consul, Properties files and Yaml files. The implementation ideas are similar, and interested students can consult them by themselves. As the main idea is to explain the implementation, it is possible that many details have not been dealt with yet. I hope you will forgive me. If you have any questions or better suggestions, please contact the comment area for guidance.