Linux CentOS 7.5 builds a highly available Hadoop distributed cluster environment

1.Linux environment preparation 1.1 turn off the firewall (all three virtual machines execute) firewall-cmd --state #View firewall status systemctl start firewalld.service #Turn on the firewall systemctl stop firewalld.service #Turn off firewall systemctl disable firewalld.service #Do not start firewall 1.2 configure static ...

Posted on Sun, 14 Jun 2020 05:40:55 -0400 by chris9902

HA mode of building Hadoop environment

1, Overview of Hadoop HA 1. Ha (high availability), that is, high availability (service interruption is not allowed). 2. The key strategy to achieve high availability is to eliminate single point of failure. Strictly speaking, ha should be divided into HA mechanism of each component: ha of HDFS and ha ...

Posted on Thu, 12 Mar 2020 01:05:33 -0400 by ReverendDexter

Building of Hadoop cluster stand-alone version-3. Hadoop cluster version

Build a simple cluster version based on Hadoop stand-alone version Article directory 0. planning 1. Clone virtual machine 2. Start the virtual machine and change the static ip 3. Modify ip mapping in hosts file 4. Modify hostname 5. Modify hadoop configuration file 5.1 core site.xml modification 5.2 ...

Posted on Tue, 25 Feb 2020 22:40:06 -0500 by stargate03

HADOOP installation configuration

1. Decompression (bigdata users) Links: https://pan.baidu.com/s/19MIN_uoSwHF6F1C14OWp_g Password: j2c4 tar -zvxf hadoop-2.7.2.tar.gz 2. Environment variable configuration vi .bash_profile HADOOP_HOME=/home/hadoopadmin/hadoop-2.7.2 PATH=$PATH:$HADOOP_HOME/bin export HADOOP_HOME //Save exit source .bash_profile //Refresh environment variab ...

Posted on Mon, 30 Dec 2019 10:19:16 -0500 by kettle_drum

Kylin configures Spark and builds Cube

HDP version: 2.6.4.0Kylin version: 2.5.1 Machine: Three CentOS-7,8G memory In addition to MapReduce, Kylin's computing engine also has a faster SPARK. This article tests the speed at which Spark builds a Cube with Kylin's own example, kylin_sales_cube. 1. Configure Kylin's Spark parameters Before running Spark cubing, it is recommended that yo ...

Posted on Tue, 24 Sep 2019 12:08:38 -0400 by nitediver

Haoop Self-study Diary--3. Haoop HDFS Operation

hadoop self-study Diary - 3.hadoop hdfs operation Setting up the Environment I use a simple windows 7 notebook and use VirtualBox to create a Centos virtual machine to install Hadoop. VirtualBox: 6.0.8 r130520 (Qt5.6.2) CentOS: CentOS Linux release 7.6.1810 (Core) jdk: 1.8.0_202 hadoop: 2.6.5 Clus ...

Posted on Thu, 25 Jul 2019 03:12:49 -0400 by Tory

Haoop Self-study Diary--2. Haoop Cluster Environment Construction

Haoop Self-study Diary --- 2. Haoop Cluster Environment Construction Setting up the Environment I use a simple windows 7 notebook and use VirtualBox to create a Centos virtual machine to install Hadoop. VirtualBox: 6.0.8 r130520 (Qt5.6.2) CentOS: CentOS Linux release 7.6.1810 (Core) jdk: 1.8.0_202 h ...

Posted on Wed, 24 Jul 2019 07:23:53 -0400 by PHPilliterate

centos7 installation Hadoop+hbase+hive steps

centos7 installation Hadoop+hbase+hive steps I. IP, DNS, Host Name linux static IP, DNS, host name configuration Two, Hadoop #### 1. IP allocation #### 2. Install jdk8 (four) yum list java* yum install -y java-1.8.0-openjdk-devel.x86_64 The default jre jdk installation path is under / usr/lib/jvm #### 3. Configure jdk environment variables (f ...

Posted on Thu, 21 Mar 2019 14:06:52 -0400 by drkstr

Secondary NameNode directory structure, nameNode fault recovery

Secondary NameNode directory structure, nameNode failure recovery 1. Secondary NameNode directory structure Secondary NameNode is an auxiliary daemon used to monitor the status of HDFS, which takes snapshots of HDFS metadata at regular intervals. See the Second aryNameNode directory structure in / opt/module/hadoop-2.7.2/dat ...

Posted on Tue, 29 Jan 2019 02:51:15 -0500 by Petrushka

spark Lost executor on YARN

The execution script appears: 15/07/30 10:18:13 ERROR cluster.YarnScheduler: Lost executor 8 on myhost1.com: remote Rpc client disassociated 15/07/30 10:18:13 ERROR cluster.YarnScheduler: Lost executor 6 on myhost2.com: remote Rpc client disassociated The reason for this is that yarn resources are not enough. Increasing resources ...

Posted on Sun, 06 Jan 2019 23:21:09 -0500 by jmansa