The running in period of Flink and Hive

There is a lot of feedback from readers. Please refer to the previous article< Hive is finally waiting, Flink >When Flink is deployed and Hive is integrated, there are some bug s and compatibility problems. Although waiting, it is not available. So I added this article as a sister article. review In the previous article, the author use ...

Posted on Mon, 25 May 2020 03:57:37 -0400 by soccerstar_23

Submit Spark tasks remotely to yarn cluster

Reference article:How to submit spark tasks to yarn cluster remotely in idea Several modes of running spark tasks: 1, local mode, write code in idea and run directly. 2,standalone mode, need to jar package program, upload to cluster, spark-submit submit to cluster run 3,yarn mode (local,client,cluster) as above, also requires jar packa ...

Posted on Thu, 21 May 2020 20:09:40 -0400 by twilightnights

Using IDEA to submit MapReduce jobs to pseudo distributed Hadoop remotely

Environmental Science VirtualBox 6.1 IntelliJ IDEA 2020.1.1 Ubuntu-18.04.4-live-server-amd64 jdk-8u251-linux-x64 hadoop-2.7.7 Install pseudo distributed Hadoop Install pseudo distributed reference: Hadoop installation tutorial single machine / pseudo distributed configuration Hadoop 2.6.0 (2.7.1) / Ubuntu 14.04 (16.04) Let's not go over it he ...

Posted on Thu, 14 May 2020 03:50:06 -0400 by Helios

Analysis of Hadoop YARN ResourceManager crash caused by data limit of ZooKeeper node

This problem makes us encounter again. It happens infrequently, but once it happens, it will cause resource manager service crash, ZK registration watch too many and other problems. It has always been a hindrance to not completely solve this problem, so based on the previous two times of analysis and reading the latest version of Hadoop 3.2.1 c ...

Posted on Sun, 10 May 2020 10:38:53 -0400 by FireWhizzle

Several ways of reading and writing spark articles by HBase

1. Overview of how HBase is read and written It is mainly divided into: The way pure Java API s read and write HBase s; How Spark reads and writes HBase s; How Flink reads and writes HBase s; HBase is read and written through Phoenix; The first is the more original and efficient operation mode provided by HBase itself. The second and ...

Posted on Sat, 09 May 2020 23:05:48 -0400 by israfel

Hadoop High Availability Environment Setup (QJM)

1. Virtual Machine Configuration host name NN DN ZK ZKFC JNN RM NM node1 * * * * node2 * * * * * * * node3 * * * * node4 * * * 2. Hadoop High Availability (HA) Implementation (QJM) 1. Modify profile on node1 host Modify core-site.xml [root@node1 ~]# vi /opt/hadoop/etc/hadoop/core-site.xml#Modify to the following:<co ...

Posted on Thu, 07 May 2020 14:02:49 -0400 by jotate

Hadoop? MapReduce compression case

Hadoop? MapReduce compression case Compress at map output modify Hadoop'wordcount word statistics engineering Just modify the main method of MyWordCount: package com.blu.mywordcount; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; imp ...

Posted on Sun, 26 Apr 2020 12:42:10 -0400 by mbarmawi

Installation of JDK, Hadoop, Hbase and GUI on Linux Server (Ubuntu 18.04)

Article Directory 0. Preparations 1. Get Ali Cloud ECS Server 2. Create Hadoop users(viii) 3. Configure SSH passwordless login 4. Install JDK 4.1 Method 1:_Manual installation (recommended)_) 4.1.1 Process 4.1.2 Download Linux JDK8 on PC 4.1.3 Use WinSCP to upload compressed packag ...

Posted on Fri, 17 Apr 2020 02:57:09 -0400 by zgkhoo

High Availability Deployment of Docker Registry Based on kubernetes

Highly available deployment of Docker Registry based on kubernetes https://blog.lecury.cn/2017/04/14/Be based on kubernetes Of Docker-Registry High Availability Deployment/   Write before It has been more than half a year for services to maintain the generated environment in the kubernetes cluster. We have encountered a l ...

Posted on Wed, 15 Apr 2020 20:03:54 -0400 by Joshua4550

Reader's Letter | Just after the cluster of HBases was built, Phoenix started and HBases completely collapsed. What was the cause? (Resolved)

Foreword: Some friends and friends have discussed some problems with me before, and I think these problems are very valuable. So I want to open a question and answer column in this public number to facilitate technical exchange and sharing. The column name is "Letter from Reader".If you encounter a problem that is difficult to solve b ...

Posted on Mon, 06 Apr 2020 23:53:14 -0400 by Formula