ELK: ElasticSearch data backup and restore

I received the task of building ElasticSearch cluster a few days ago. As a novice, I can't do without Baidu's support. I started to check the data and change the configuration. I built the cluster in less than one day (please ignore the time problem...) For the first time, not so skilled.

After several days of stable operation, we suddenly received a call from our leader, and the node hung up (shivering). What's the situation?


Then I checked the cluster status

curl -XGET 'http://127.0.0.1:9200/_cat/health?v'

View node status

curl -XGET 'http://127.0.0.1:9200/_cat/nodes?v'

View index status

curl -XGET 'http://127.0.0.1:9200/_cat/indices?v'

Then, no then

curl -XDELETE ......


At this time, I think how good it is to have backup data...

Background

1. The log is too large and the environment resources are limited, so the previous index should be cleaned regularly.
2. Prevent the deletion by mistake (no deletion of the library and running away ~ ~)
2. For disaster recovery backup, what we need is a real backup of the cluster.

Two, interface

Official document data backup instructions

After opening the website, we can see the obvious English words It's not friendly. Why not display it in Chinese directly

Backing Up Your Cluster


Please ignore one of my English scum~~

Document translation

The original official provides the snapshot API backup interface, so let's start to deal with it
There are generally two ways to use snapshot for cluster backup:
1. Incremental backup: it is the default backup scheme, that is, no index is specified when taking a snapshot, the first snapshot will be a full copy of data, and the subsequent snapshot will retain the difference between the saved snapshot and the new data.
2. Backup specified index: some logs do not need to be saved for a long time, so only taking a snapshot of the index we are interested in can save disk space to some extent.

Now the record is to back up the specified index

3, Specific operation

1.ElasticSearch configuration

Open profile

vim elasticsearch.yml

Add shared directory in configuration file (all nodes need to be configured)

path.repo: ["/data/backups/"]

After configuration, restart the cluster

2. Register shared warehouse

#Create warehouse catalog
mkdir /data/backups/2020-03-09
#Create repository
curl -XPUT http://127.0.0.1:9200/_snapshot/2020-03-09 -d '
{
    "type": "fs",
    "settings": {
        "location": "/data/backups/2020-03-09",
        "max_snapshot_bytes_per_sec": "20mb",
        "max_restore_bytes_per_sec": "20mb"
    }
}'
#2020-03-09 name of shared warehouse
#"type":"fs" specifies that the warehouse type is a shared file system
#"location": "/data/backups" specifies the mount directory
#"Max? Snapshot? Bytes? Per sec": "20mb" controls the current limit when the snapshot is stored in the warehouse. By default, 20mb / S is used
#"Max ﹣ restore ﹣ bytes ﹣ per ﹣ sec": "20mb" controls the current limit when data is recovered from the warehouse. By default, 20mb / S is used

3. Update snapshot configuration

We can update the snapshot configuration through the interface

curl -XPOST http://127.0.0.1:9200/_snapshot/2020-03-09 -d '
{
    "type": "fs",
    "settings": {
        "location": "/data/backups/2020-03-09",
        "max_snapshot_bytes_per_sec": "50mb",
        "max_restore_bytes_per_sec": "50mb"
    }
}'

In this way, the current limiting parameters are modified

4. Backup specified index

curl -XPUT http://127.0.0.1:9200/_snapshot/2020-03-09/my_indices-2020.03.09?wait_for_completion=true -d '
{
    "indices": "my_indices-my_indices-2020.03.09"
}'
#My ﹣ indexes-my ﹣ indexes-2020.03.09 snapshot name
#Wait for completion the default snapshot will enter background execution. Adding this flag can prevent background execution
#'indexes' specifies the index to snapshot

5. Delete the specified index backup

curl -XDELETE http://127.0.0.1:9200/my_indices-my_indices-2020.03.09

4, Complete backup

The backup has been completed. Please record the common operations below

#List warehouse
curl -XGET http://127.0.0.1:9200/_snapshot/*
#Delete repository
curl -XDELETE http://127.0.0.1:9200/_snapshot/2020-03-09
#List snapshots
curl -XGET http://127.0.0.1:9200/_snapshot/2020-03-09/*
#Delete snapshot
curl -XDELETE http://127.0.0.1:9200/_snapshot/2020-03-09/my_indices-my_indices-2020.03.09

5, Backup recovery

We have backed up the index data. How can we recover the index data if it is lost
Fortunately, the government has provided us with an interface

1. Recover all indexes

curl -XPOST "http://127.0.0.1:9200/_snapshot/2020-03-09/_restore?wait_for_completion=true"

2. Restore the specified index

curl -XPOST "http://127.0.0.1:9200/_snapshot/2020-03-09/_restore?wait_for_completion=true" -d '
{
 "indices":"my_indices",
 "ignore_unavailable":"true"
}'

Note: the index needs to be turned off before snapshot recovery

curl -XPOST "http://127.0.0.1:9200/my_indices/_close"

By now, we have finished the data backup and recovery. Don't worry about similar problems in the future!

Published 25 original articles, won praise 14, visited 40000+
Private letter follow

Tags: snapshot curl ElasticSearch less

Posted on Sun, 08 Mar 2020 23:19:10 -0400 by peuge