Use and installation of sqoop

SQOOP installation and use SQOOP installation 1. Upload and unzip Installation package download address http://archive.apache.org/dist/sqoop/1.4.7/ tar -zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /usr/local/soft/ 2. Modify folder name mv sqoop-1.4.7.bin__hadoop-2.6.0/ sqoop-1.4.7 3. Modify profile # Switch to the sqoop profile ...

Posted on Mon, 06 Dec 2021 17:35:45 -0500 by CoreyR

JVM quick start

1. JVM exploration? Several interview questions: 1. Would you please talk about your understanding of jvm? Java 8 virtual machine and previous changes update? 2. What is oom? What is stack overflow? How to analyze? 3. What are the common tuning parameters for the jvm? 4. How to capture memory snapshots and analyze Dump files? 5. What ...

Posted on Mon, 06 Dec 2021 15:16:03 -0500 by Im Jake

Senior big data Development Engineer - Hadoop learning notes

Hadoop advanced level YARN: Hadoop resource scheduling system What is YARN Apache Hadoop YARN(Yet Another Resource Negotiator) is a sub project of Hadoop, which is introduced to separate Hadoop 2.0 resource management and computing components.YARN has enough universality, and customers support other distributed computing modes. Analys ...

Posted on Sun, 05 Dec 2021 18:08:55 -0500 by snpo123

HADOOP learning notes

rpm -qa | grep vim // See what packages vim commands are in appear vim-minimal-7.4.160-4.el7.x86_64 yum install -y vim* //Install vim related packages rpm -qa | grep vim see HADOOP learning notes 1, Install virtual machine (CentOS) 2, Modify host name (host name of current virtual machine) 1. View the current host name Command: hos ...

Posted on Sun, 05 Dec 2021 08:12:39 -0500 by Rizla

3, Hive's DML data operation

3.1 data import 3.1.1 loading data into the table (Load) (1) Grammar load data [local] inpath 'path of data' [overwrite] into table student [partition (partcol1=val1,...)]; load data: indicates loading data Local: indicates loading data from local to hive table; Otherwise, load data from HDFS to hive table inpath: indicates the path to load d ...

Posted on Sun, 05 Dec 2021 02:14:14 -0500 by shinyo

Flink independent cluster deployment and HA deployment

Scene description 172.19.9.202 master node JobManager master / slave 172.19.9.201 slave node TaskManager master / slave 172.19.9.203 slave node TaskManager master / slave 1, SSH master node and slave node settings should be unified ssh-keygen -t rsa -P "" Do not set password cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys ...

Posted on Thu, 02 Dec 2021 16:30:34 -0500 by angelssin

Hadoop pseudo-distributed deployment and common operations

hadoop pseudo-distributed deployment Having deployed hadoop 2.x before, now do a deployment of hadoop 3.x. hadoop has three components: hdfs for storing data, mapreduce for computing (jobs), yarn for resources (cpu memory) and job scheduling. In fact, the official hadoop website has deployment steps: https://hadoop.apache.org/docs/stable/h ...

Posted on Thu, 02 Dec 2021 12:09:53 -0500 by mikes1471

Hive processes Json data

Data processing in Json format Json data format is a commonly used data format, such as embedded point data, business end data and front and rear end calls. Therefore, it is necessary for us to learn the processing method of this data format Prepare data cat json.data {"movie":"1193","rate":"5","timeStamp":"978300760","uid":"1"} {"movie":"661", ...

Posted on Wed, 01 Dec 2021 09:41:51 -0500 by coderb

Hadoop fully distributed installation -- hadoop-2.7.3

catalogue 1, Installation preparation   2: Installing hadoop on the master node 3, Installing hadoop from a node   4, Start hadoop 5, Verify installation 1, Installation preparation 1. Three virtual machines are required: the master node is Hadoop 001, the slave nodes are Hadoop 002 and Hadoop 003; hadoop001,hadoop002,hado ...

Posted on Wed, 01 Dec 2021 09:27:56 -0500 by Slip

Docker + Hadoop build fully distributed

Reading this article requires a certain Linux foundation, which is not suitable for novices. It only provides ideas System deployment reference resources Docker Download: https://docs.docker.com/desktop/windows/install/ Docker installation: https://www.runoob.com/docker/windows-docker-install.html Deploying Centos7 Docker domestic sourc ...

Posted on Mon, 22 Nov 2021 22:48:50 -0500 by Al42