Optimization Practice of Zabbix backend storage ES

Scenario analysis

As the historical data of zabbix is stored in elastic search, there is a need to store the monitored historical data as long as possible, preferably for one year. At present, there are three ES nodes, with a daily monitoring historical data volume of 5G. At present, the data can be stored for up to one month, and those over 30 days will be deleted regularly. Each memory is divided into 8G, and all of them use mechanical hard disk , the main partition is 5, and the replica partition is 1. Generally, the query requirements only obtain the historical data of one week. Occasionally, there is a need to query the historical data of one to two months.

Node planning

In order to allow ES to store longer historical data, I increased the number of nodes to 4 nodes, and increased the memory of some nodes. Some nodes use SSD storage

192.168.179.133  200GSSD 4G Memory  tag:hot node.name=es1
192.168.179.134  200GSSD 4G Memory  tag:hot node.name=es2
192.168.179.135  500GHDD 64G Memory  tag:cold node.name=es3
192.168.179.136  500GHDD 64G Memory  tag:cold node.name=es4

Optimization thinking

Data mapping is re modeled, str type data is not segmented, and hot and cold nodes are used to store the data. The first seven days of data index segmentation is designed as two primary and one secondary. The index is stored on the hot node, and the data over seven days will be stored on the cold node, and the number of copies on the cold node is modified to 0. Es provides a shrink api for compression. As ES is a Lucene based search engine, Lucene's index is composed of multiple segments. Each segment will consume file handle, memory and CPU running cycle. Too many segments will make resource consumption larger and search slower. Here, I forced the index fragmentation of the previous day into one segment, and modified the refresh interval to 60s to reduce the frequency of segment generation. Close indexes older than 3 months. All the above operations are performed regularly using the ES management tool, cursor.

Docking operation of zabbix and ES

1. Modify / etc/zabbix/zabbix_server.conf and add the following

ES address can be filled in any node in the cluster

HistoryStorageURL=192.168.179.133:9200
HistoryStorageTypes=str,text,log,uint,dbl
HistoryStorageDateIndex=1

2. Modify / etc/zabbix/web/zabbix.conf.php and add the following

global $DB, $HISTORY;
$HISTORY['url']   = 'http://192.168.179.133:9200';
// Value types stored in Elasticsearch.
$HISTORY['types'] = ['str', 'text', 'log','uint','dbl'];

3. Modify ES configuration file and add labels of hot and cold nodes

vim elasticsearch.yml
Hot node configuration

node.attr.box_type=hot

Cold node configuration

node.attr.box_type=cold

3. Create templates and pipes on es

Templates of each data type need to be created. api information can be obtained according to the elasticsearch.map file. Template definition content has matching index, primary and secondary partition number setting, refresh interval, new index allocation node setting and mapping setting. Here I just take the index of uint and str data as an example

PUT _template/uint_template
{
   "template": "uint*",
   "index_patterns": ["uint*"],
   "settings" : {
      "index" : {
         "routing.allocation.require.box_type": "hot",
         "refresh_interval": "60s",
         "number_of_replicas" : 1,
         "number_of_shards" : 2
      }
   },
   "mappings" : {
      "values" : {
         "properties" : {
            "itemid" : {
               "type" : "long"
            },
            "clock" : {
               "format" : "epoch_second",
               "type" : "date"
            },
            "value" : {
               "type" : "long"
            }
         }
      }
   }
}

PUT _template/str_template
{
   "template": "str*",
   "index_patterns": ["str*"],
   "settings" : {
      "index" : {
         "routing.allocation.require.box_type": "hot",
         "refresh_interval": "60s",
         "number_of_replicas" : 1,
         "number_of_shards" : 2
      }
   },
   "mappings" : {
      "values" : {
         "properties" : {
            "itemid" : {
               "type" : "long"
            },
            "clock" : {
               "format" : "epoch_second",
               "type" : "date"
            },
            "value" : {
               "index" : false,
               "type" : "keyword"
            }
         }
      }
   }
}

The function of defining pipeline is to preprocess the data before writing index, so that it can generate index by day.

PUT _ingest/pipeline/uint-pipeline
{
  "description": "daily uint index naming",
  "processors": [
    {
      "date_index_name": {
        "field": "clock",
        "date_formats": ["UNIX"],
        "index_name_prefix": "uint-",
        "date_rounding": "d"
      }
    }
  ]
}
PUT _ingest/pipeline/str-pipeline
{
  "description": "daily str index naming",
  "processors": [
    {
      "date_index_name": {
        "field": "clock",
        "date_formats": ["UNIX"],
        "index_name_prefix": "str-",
        "date_rounding": "d"
      }
    }
  ]
}

4. Restart zabbix after modification and check whether there is data in zabbix

systemctl restart zabbix-server

Using cursor to operate on Indexes

The official document address of Creator is as follows
https://www.elastic.co/guide/en/elasticsearch/client/curator/5.8/installation.html

1. Install the cursor

pip install -U elasticsearch-curator

2. Create the cursor configuration file

mkdir /root/.curator
vim /root/.curator/curator.yml
---
client:
  hosts:
    - 192.168.179.133
    - 192.168.179.134
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

3. Edit action.yml and define action

Assign indexes older than 7 days to cold nodes

1:
    action: allocation
    description: "Apply shard allocation filtering rules to the specified indices"
    options:
      key: box_type
      value: cold
      allocation_type: require
      wait_for_completion: true
      timeout_override:
      continue_if_exception: false
      disable_action: false
    filters:
    - filtertype: pattern
      kind: regex
      value: '^(uint-|dbl-|str-).*$'
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y-%m-%d'
      unit: days
      unit_count: 7

The index of the previous day is forced to merge, with 1 segment per fragment.

2:
    action: forcemerge
    description: "Perform a forceMerge on selected indices to 'max_num_segments' per shard"
    options:
      max_num_segments: 1
      delay:
      timeout_override: 21600 
      continue_if_exception: false
      disable_action: false
    filters:
    - filtertype: pattern
      kind: regex
      value: '^(uint-|dbl-|str-).*$'
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y-%m-%d'
      unit: days
      unit_count: 1

Modify the cold node to get a copy partition number of 0

  3:
    action: replicas
    description: "Set the number of replicas per shard for selected"
    options:
      count: 0
      wait_for_completion: True
      max_wait: 600
      wait_interval: 10
    filters:
    - filtertype: pattern
      kind: regex
      value: '^(uint-|dbl-|str-).*$'
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y-%m-%d'
      unit: days
      unit_count: 7

Close indexes older than six months

  4:
    action: close
    description: "Close selected indices"
    options:
      delete_aliases: false
      skip_flush: false
      ignore_sync_failures: false
    filters:
     - filtertype: pattern
       kind: regex
       value: '^(uint-|dbl-|str-).*$'
     - filtertype: age
       source: name
       direction: older
       timestring: '%Y-%m-%d'
       unit: days
       unit_count: 180

Delete indexes older than one year

5:
    action: delete_indices
    description: "Delete selected indices"
    options:
      continue_if_exception: False
    filters:
    - filtertype: pattern
      kind: regex
      value: '^(uint-|dbl-|str-).*$'
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y-%m-%d'       
      unit: days
      unit_count: 365

4. Execute the cursor to test

curator action.yml

5. Write the cursor operation into the scheduled task and execute it every day

crontab -e
10 0 * * * curator /root/action.yml

The above is all the practice of storage optimization for zabbix backend storage elastic search. Please refer to the link
https://www.elastic.co/cn/blog/hot-warm-architecture-in-elasticsearch-5-x

Welcome to pay attention to personal company name "master Chen without story"

Tags: Linux Zabbix ElasticSearch vim Unix

Posted on Mon, 09 Mar 2020 01:03:20 -0400 by ClarkF1