1. Purpose of document preparation

This document mainly uses HBase snapshot to export the full amount of historical data and restore it to the new HBase cluster, then transforms the original ExportSnapshot class, realizes incremental export by comparing the changed files, and finally realizes the incremental backup and restore of HBase across clusters.

• testing environment

1.CDH7.1.4. Enable Kerberos and hbase 2.2.3

2.CDP7.1.6. Enable Kerberos and hbase 2.2.3

3. User operation using ldapuser1

2. Operation steps

2.1 generate a test table in cdp7.1.4

2.1.1 use the pe command of HBase to generate a 10G table

hbase org.apache.hadoop.hbase.PerformanceEvaluation --compress=SNAPPY --size=10 sequentialWrite 10

Report error due to insufficient permission to create table

The ranger configuration is as follows

2.1.2 test table HDFS size

hadoop fs -du -h /hbase/data/default/

Since the compression format of TestTable table is specified as SNAPPY here, it is only 2.5GB on HDFS

2.1.3 total data volume of TestTable, 10485760 pieces of data in total

count 'TestTable'

2.2 full data backup and restore of HBase across clusters through snapshots

2.2.1 generate TestTable snapshot

snapshot 'TestTable','TestTable-snapshot1'

2.2.2 exporting TestTable snapshot data

On the command line, use the ExportSnapshot that comes with HBase to export snapshots

hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot TestTable-snapshot1 -copy-to hdfs://cdp02:8020/tmp/hbasebackup/TestTable-snapshot1

View exported snapshot data

hadoop fs -ls /tmp/hbasebackup/TestTable-snapshot1

hadoop fs -du -h /tmp/hbasebackup/TestTable-snapshot1

You can see that exporting snapshots actually exports the snapshot information and all data files recorded by snapshots to the. HBase snapshot and archive directories under the specified directory.

The format and content of CDSW product documents have changed significantly.

2.2.3 copy snapshot data to cdp7.1.6 cluster

Because both clusters have Kerberos authentication, mutual trust must be done between the two clusters before using the distcp command (you can contact the cdh operation and maintenance personnel)

hadoop distcp hdfs://cdp02:8020/tmp/hbasebackup/TestTable-snapshot1/.hbase-snapshot/TestTable-snapshot1 hdfs://cdh3.macro.com:8020/hbase/.hbase-snapshot
hadoop distcp hdfs://cdp02:8020/tmp/hbasebackup/TestTable-snapshot1/archive/data/default/TestTable hdfs://cdh3.macro.com:8020/hbase/archive/data/default

Copy error, insufficient permission

The configuration of cluster cdp7.1.6 Ranger is as follows

View snapshot data replicated to cdp7.1.6 cluster

hdfs dfs -ls hdfs://cdh3.macro.com:8020/hbase/.hbase-snapshot
hdfs dfs -ls hdfs://cdh3.macro.com:8020/hbase/archive/data/default
hdfs dfs -du -h hdfs://cdh3.macro.com:8020/hbase/

2.2.4 restore TestTable table using testtable-snapshot 1 snapshot

Open the following permissions, otherwise the problem of insufficient permissions will be reported

If it is not opened, the following error will be reported

list_snapshots
restore_snapshot 'TestTable-snapshot1'

2.2.5 verify whether the recovered table data is consistent with the data at the time of snapshot

The data recovered through the snapshot is consistent with the total number of data at the time of snapshot, and the table content is consistent.

2.3 realize HBase incremental data backup and restore across clusters through snapshots

2.3.1 modifying TestTable data

Modify a piece of data and add a piece of data through hbase shell

put 'TestTable','11111111111111111111111111','info0:AppendTest','1111'
put 'TestTable','00000000000000000000000000','info0:AppendTest','00000'

The total amount of data in TestTable is 10485761

count 'TestTable'

2.3.2 create TestTable snapshot again

snapshot 'TestTable','TestTable-snapshot2'

2.3.3 export incremental data of the second snapshot

This step is mainly to export the incremental data between TestTable-snapshot2 and TestTable-snapshot1 snapshots. The default ExportSnapshot method of HBase is that there is no incremental snapshot export method. Here, the source code is modified on the basis of the original snapshot export to complete the export of incremental data between the two snapshots.

GitHub source address: https://github.com/javaxsky/hbaseexport You can download the compiled jar package and put the balancer.jar and hbase-export-1.0-SNAPSHOT.jar in the directory "/ opt/cloudera/parcels/CDH/lib/hbase/lib"

Execute the following command on the command line to export the incremental data of the two snapshots to HDFS

hbase org.hadoop.hbase.dataExport.ExportSnapshot -snapshot TestTable-snapshot2 -copy-to hdfs://cdp02:8020/tmp/hbasebackup/snapshot2-snapshot1/ -snapshot-old TestTable-snapshot1

Exported data directory

hadoop fs -du -h /tmp/hbasebackup/snapshot2-snapshot1

2.3.4 copy the exported snapshot file to CDP7.1.6 cluster

hadoop distcp hdfs://cdp02:8020/tmp/hbasebackup/snapshot2-snapshot1/.hbase-snapshot/TestTable-snapshot2 hdfs://cdh3.macro.com:8020/hbase/.hbase-snapshot
hadoop distcp hdfs://cdp02:8020/tmp/hbasebackup/snapshot2-snapshot1/archive/data/default/TestTable hdfs://cdh3.macro.com:8020/hbase/archive/data/default

Data directory after importing incremental data

hdfs dfs -du -h hdfs://cdh3.macro.com:8020/hbase/

2.3.5 the command line uses the snapshot to restore the TestTable data and verify the data

disable 'TestTable'
list_snapshots
restore_snapshot 'TestTable-snapshot2'
enable 'TestTable'
get 'TestTable','00000000000000000000000000'
count 'TestTable'

3. Summary

The Snapshot of HBase only involves the record of metadata, not the copy of data

The export snapshot of HBase is performed at the HDFS level, which will not cause additional burden on the Master and RegionServer services of HBase.

The rewritten ExportSnapshot incremental snapshot export is realized through the difference between the two snapshot file lists. You only need to export the files with differences.

Exporting snapshot data using ExportSnapshot does not cause the export data to expand, which is basically the same as the size of the original HBase table with snapshot compression enabled.

When modifying the ExportSnapshot provided with HBase, you need to obtain the source code according to the corresponding HBase version for modification. The Packages path of ExportSnapshot of different versions has been changed.

When transferring SnapShot files across clusters, it is recommended to use the Cloudera enterprise version function BDR to realize file copy transmission across clusters.