27. Data governance of yiee data operation system of Duoyi Education - atlas deployment and use

Catalog 1, atlas Compilation and packaging 2, atlas installation configuration 1. Compilation environment 2. Compilation steps 3. Installation steps 4. Hive hook configuration 5. Operation test 3, atlas configuration hive hook 4, Introduction to atlas         1,Base Search         ...

Posted on Tue, 11 Feb 2020 03:02:02 -0500 by bensonang

Ambari 2.7.0 + hdp3.1.4.0 installation, hdfs data backup and recovery, hive data backup and recovery, hbase data backup and recovery

Catalog 1 Ambari + HDP offline installation 1.1 INTRODUCTION 1.1.1 introduction to ambari 1.1.2 HDP 1.1.3 HDP-UTILS 1.2 address of ambari official website 1.3 Ambari and HDP Downloads 1.4 system requirements 1.4.1 software requirements 1.5 modify the maximum number of open files 1.6 cluster node plann ...

Posted on Fri, 07 Feb 2020 06:30:42 -0500 by ryanyoungsma

Basic commands of Hbase Shell

Article directory 1, Enter HBase command line 2, Operation of HBase table 3, create 4, View table list 5, View table details desc 6, Modify the definition of table alter 1. Add a column cluster 2. Delete a column cluster 3. Add column cluster hehe and delete column cluster myInfo 4. Clear table trunc ...

Posted on Wed, 05 Feb 2020 05:22:58 -0500 by djddb

How to integrate Hive and HBase

Version Description: HDP: 3.0.1.0 Hive: 3.1.0 HBase: 2.0.0 I. Preface Before learning HBase, we had doubts. Although HBase can store hundreds of millions or billions of rows of data, it is not very friendly for data analysis. It only provides a simple quick query ability based on Key values, and cannot perform a large number of conditional qu ...

Posted on Fri, 31 Jan 2020 14:23:36 -0500 by buck2bcr

Chapter 6 HBase API Operations - Data Operations and Data Migration

Previous: Chapter 6 HBase API Operations (2) 1. Encapsulation of data Encapsulate data using multithreaded thread security First, create a tool class: HbaseUtil Code implementation: package studey.bigdate.util; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hbase.HBaseConfig ...

Posted on Fri, 24 Jan 2020 02:00:18 -0500 by kuma

Flink reads Kafka data Sink to MySQL and HBase databases

Flink reads Kafka data Sink to MySQL and HBase databases Flink transfers the stream data Sink to the database. Generally, it needs to implement its own custom Sink. The following example demonstrates the Sink to MySQL and HBase examples. Insert a code slice here import java.util.Properties import org.ap ...

Posted on Mon, 20 Jan 2020 09:21:47 -0500 by like_php

Two ways for Spark to read and write HBase (RDD, DataFrame)

Write data using saveAsHadoopDataset import org.apache.hadoop.hbase.{HBaseConfiguration, HTableDescriptor, TableName} import org.apache.hadoop.hbase.client.{HBaseAdmin, Put, Result} import org.apache.hadoop.hbase.io.ImmutableBytesWritable import org.apache.hadoop.hbase.mapreduce.TableInputFormat //import org.apache.hadoop.hbase.mapreduce.TableO ...

Posted on Sun, 05 Jan 2020 11:31:26 -0500 by nileshn

How can multithreading be used in the Flink operator to ensure that no data is lost?

Analyzing Pain Points The author has a Flink task on-line that consumes Kafka data. After data conversion, a third-party API is called inside Flink's Ink Operator to report the data to the third-party's data analysis platform.The batch synchronization API is used here, that is, a third-party interface can be requested every 50 data requests to ...

Posted on Wed, 25 Dec 2019 22:59:57 -0500 by benpaxton777

[Hbase learning notes] 1.Hbase Standalone Mode

I think the data is quite interesting this year. I plan to work hard in this direction This is mainly based on manjaro. It took a long time to install this system, and gradually felt that this distribution is more comfortable than Ubuntu. Install JDK In fact, the shell command is not important. It is mainly to install JDK, and then set JAVA_HOM ...

Posted on Sun, 15 Dec 2019 15:10:07 -0500 by KnottyAlder

Cluster Snapshot Practice in Distributed Graph Database Nebula Graph

1 Overview 1.1 Requirement Background Nebula Graph, a graphics database, will have a large amount of data and high frequency of business processing in the production environment. It will inevitably cause human, hardware or business processing errors in the actual operation. Some serious errors will cause the cluster to not function properly or ...

Posted on Tue, 10 Dec 2019 08:24:41 -0500 by Jeremysr