hadoop high availability installation (HA)

High availability hadoop installation HA

Write at the beginning

The premise of this article is that the installation of hadoop pseudo distribution is completed.
hadoop high availability fool installation (fool to most details)
Completion of a college assignment

Let's first introduce the general needs of high availability hadoop?

High availability hadoop generally requires four linux virtual machines to be turned on and seven nodes to be hadoop HA.
Three of them are zookeeper cluster, three are journal node cluster, two are zkfc, two are namenode, three are datanode, two are ResourceManager, three are nodemanager
So these four virtual machines have multiple functions.

For the convenience of operation, you can download the xshell software as the case may be and directly connect with the virtual machine for operation

So we set up four linux virtual machines, named node01, node02, node03, node04 (which have been solved in hadoop pseudo Distributed installation)

Installation procedure

1. Distribute JDK

In the pseudo Distributed installation, the preliminary construction of four virtual machines has been completed. We first distribute the jdk in node01 to the other three.

//It's a symbol, not a symbol. It's the key on the left of the horizontal number key 1
scp jdk-7u67-linux-x64.rpm node02:`pwd`
scp jdk-7u67-linux-x64.rpm node03:`pwd`
scp jdk-7u67-linux-x64.rpm node04:`pwd`

Then we input the command ll (lowercase L) under each virtual machine to check whether there are jdk files under the paths of all virtual machines. The success picture is as follows:

If xshell is installed, you can click the three horizontal bars in the lower right corner of the software

Click all sessions, and then directly enter ll in the command bar below, all machines can execute the command

1.1 install jdk

Execute rpm installation commands on node02, 03 and 04 respectively

rpm -i jdk-7u67-linux-x64.rpm

Enter command on node01

cd /etc

In this directory, the profile file is distributed to node02, 03, 04.

scp profile node04:`pwd`

Refresh the file by using all session columns of Xshell, or entering the sub input instructions of each machine

source /etc/profile

Use the Xshell all session bar, or enter each machine to input instructions

jps

Check whether the jdk of the three machines 02, 03 and 04 is installed.
The success picture is as follows:

2. Synchronization time

Use the Xshell all session bar, or enter each machine to input instructions

date

To view the current time of each machine, if the time difference between machines is too large, some processes in the cluster will not run.
If the time is not synchronized, the instruction is used for time synchronization.
First, use the Xshell all session bar, or enter each machine to input instructions

yum -y install ntp

Install the time synchronizer. And then each performs a synchronization command

ntpdate time1.aliyun.com

This code is for time synchronization with Alibaba cloud server.

3. Configuration file check before installation

Use the Xshell all session bar, or enter each machine to input instructions

cat /etc/sysconfig/network

Check that HOSTNAME is correct.
Use the Xshell all session bar, or enter each machine to input instructions

cat /etc/hosts

Check whether the IP mapping is correct. If not, you can modify the file.
Success picture:

Use the Xshell all session bar, or enter each machine to input instructions

cat /etc/sysconfig/selinux

Check if your SELINUX=disabled is modified.
Use the Xshell all session bar, or enter each machine to input instructions

service iptables status

See if the firewall is turned off.
The success picture is as follows:

4. Secret key free settings of NN and other three machines

1. Execute instructions in home directory

 ll  -a

Check whether there is a. ssh file. If not, enter the command

ssh localhost

Remember! Remember to input the command exit after ssh localhost, because the node0X entered after ssh is not the node0X you input the command, but two things.
Home directory is to input the command cd, that is, when you are in [node02 ~], you are in the home directory.

The figure is as follows:

2. When you have finished executing, have the. ssh file, and then input the command

cd .ssh
//Reenter command
ll


3. There will be many files. We will send the public key of node01 to the other three computers

scp id_dsa.pub node02:`pwd`/node01.pub
scp id_dsa.pub node03:`pwd`/node01.pub
scp id_dsa.pub node04:`pwd`/node01.pub

The schematic diagram is as follows:

If there is no. ssh file on node02, node03 and node04, remember ssh localhost and don't forget to exit

4. In the. ssh directory of node02, we can see node01.pub, and then we append the public key to authorized_keys

cat node01.pub >> authorized_keys

And on node01, ssh node02 to see if it's secret key free. Remember to exit later
After the test is successful, add node01.pub to node05 and node06

5. No secret key between two NN

1.node01 and node02 are keyless to each other: now node01 can log in node02 with the secret key, so node02 can log in node01 with the secret key

ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

Then try ssh localhost.
Redistribute to node01:

scp  id_dsa.pub   node01:`pwd`/node02.pub

Under the. ssh directory of node01

cat node02.pub >> authorized_keys

Finally, verify whether you can log in to node01 without secret key on node02

6. Modify some configuration information of namenode

Let's go to node01 first.
We first go to the hdfs.xml Documents. The folder directive is

cd /opt/ldy/hadoop-2.6.5/etc/hadoop/
//Reenter command
ll

We can find the directory under this folder

Input command

vi hdfs-site.xml

This profile appears

We remove the configuration of snn existing in the folder (ignore it if not), and then configure the file

//Press the I key on the keyboard
i
//To modify the document
//After the modification is completed, press esc key, and then input: wq to complete the modification and exit
esc
:wq
//Then configure the file in < configuration >
 <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>node01:50090</value>
    </property>

<property>
        <name>dfs.nameservices</name>
        <value>mycluster</value>
</property>

<property>
        <name>dfs.replication</name>
        <value>3</value>
</property>

<property>
        <name>dfs.ha.namenodes.mycluster</name>
        <value>nn1,nn2</value>
</property>

<property>
  <name>dfs.namenode.rpc-address.mycluster.nn1</name>
  <value>node01:8020</value>
</property>
<property>
  <name>dfs.namenode.rpc-address.mycluster.nn2</name>
  <value>node02:8020</value>
</property>
<property>
  <name>dfs.namenode.http-address.mycluster.nn1</name>
  <value>node01:50070</value>
</property>
<property>
  <name>dfs.namenode.http-address.mycluster.nn2</name>
  <value>node02:50070</value>
</property>

<property>
  <name>dfs.namenode.shared.edits.dir</name>
  <value>qjournal://node01:8485;node02:8485;node03:8485/mycluster</value>
</property>

<property>
  <name>dfs.journalnode.edits.dir</name>
  <value>/var/ldy/hadoop/ha/jn</value>
</property>


<property>
  <name>dfs.client.failover.proxy.provider.mycluster</name>
  <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
  <name>dfs.ha.fencing.methods</name>
  <value>sshfence</value>
</property>
<property>
  <name>dfs.ha.fencing.ssh.private-key-files</name>
  <value>/root/.ssh/id_dsa</value>
</property>

You can copy and paste directly. If there is any change, you can modify the corresponding attribute.
And then we configure the core-site.xml Property.
Still in our folder.

vi core-site.xml
//Then configure the file in < configuration >
<!– Cluster name mycluster-->
<property>
        <name>fs.defaultFS</name>
        <value>hdfs://mycluster</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/var/ldy/hadoop/pseudo</value>
    </property>
<property>
   <name>ha.zookeeper.quorum</name>
   <value>node02:2181,node03:2181,node04:2181</value>
</property>

Then we go to the slave file.

vi slaves
//Modify configuration
node04
node05
node06

We have installed hadoop in node01 in the pseudo distribution, and then we complete the file configuration of hadoop here. Now we distribute hadoop to the other three machines.

cd /opt

Enter this folder, and distribute the self named ldy folder in which hadoop is placed to node02, node03, node04

scp -r ldy/ node02:`pwd`
scp -r ldy/ node03:`pwd`
scp -r ldy/ node04:`pwd`

Then hdfs-site.xml And core-site.xml Send to node02, node03, node04 respectively

cd /opt/ldy/hadoop-2.6.5/etc/hadoop/
scp hdfs-site.xml core-site.xml node02:`pwd`
scp hdfs-site.xml core-site.xml node03:`pwd`
scp hdfs-site.xml core-site.xml node04:`pwd`

This completes the configuration and installation of hadoop.
Then we install zookeeper

7.zookeeper installation

Let's first go to node02, put the zookeeper installation package in the home directory, and then unzip zookeeper
Here are some.

tar xf zookeeper-3.4.6.tar.gz -C /opt/ldy

Then modify the configuration file of zookeeper

cd /opt/ldy/zookeeper-3.4.6/conf
ll

We can see zoo_sample.cfg Documents. We're going to give zookeeper's zoo_sample.cfg Change the name. It is recommended to copy zoo here_ sample.cfg File, so that if there is an error in the configuration file, the source file is reused. So let's copy the file first

cp zoo_sample.cfg zoo.cfg
//zoo_sample.cfg  Renamed zoo.cfg

And then we modify it zoo.cfg

vi zoo.cfg
dataDir=/var/ldy/zk
//And add at the end
server.1=node02:2888:3888
server.2=node03:2888:3888
server.3=node04:2888:3888
//Among them, 2888 primary and secondary communication ports and 3888 are the ports for election mechanism when the primary is suspended

Then we distribute zookeeper to other nodes

scp -r zookeeper-3.4.6/ node03:`pwd`
scp -r zookeeper-3.4.6/ node04:`pwd`

And in each machine down / opt/ldy / ll see if the distribution is successful.
We continue to create paths in the configuration file for each machine.

mkdir -p /var/ldy/zk

For node2:

echo 1 > /var/ldy/zk/myid
cat /var/ldy/zk/myid

For node03:

echo 2 > /var/ldy/zk/myid
cat /var/ldy/zk/myid

For node04:

echo 3 > /var/ldy/zk/myid
cat /var/ldy/zk/myid

Then we configure the file in / etc/profile

cd /etc/profile
export ZOOKEEPER_HOME=/opt/ldy/zookeeper-3.4.6
export PATH=$PATH:/usr/java/jdk1.7.0_67/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin

Then distribute / etc/profile to other node03 and node04

scp /etc/profile node05:/etc
scp /etc/profile node06:/etc

In node02, node03 and node04, source /etc/profile is used to refresh files.
After verification, input zkCli.s and press TAB key to incomplete the name zkCli.sh That's it.
And then we start zookeeper, in the entire session

zkServer.sh start

Then use

zkServer.sh status

Check the status of each zookeeper node. The picture is as follows

If zookeeper can't run, you can go to Java in / etc/profile_ Home change
It's an absolute path.

cd /etc
vi  profile


All three machines need to be changed

8. Start the journal node

We started the journal node to synchronize the data between the two namenode s
Start the journal node on node01, node02 and node03.

hadoop-daemon.sh start journalnode

We can use the command jps to check whether it starts

We randomly choose a namenode to execute

hdfs namenode –format

Another namenode does not need to be executed, otherwise the clusterID changes and the cluster cannot be found. The success is as follows

Then start to format the namenode

hadoop-daemon.sh start namenode

Enter jps and another namenode will appear
Then another namenode needs to synchronize the data. Let's enter the following command

hdfs namenode -bootstrapStandby

Enter jps, and you have namenode
Then we need to format zkfc

hdfs zkfc -formatZK

We execute on node02 zkCli.sh Open the zookeeper client to see if Hadoop HA is open

zkCli.sh
ls /


Then we start the hdfs cluster on node03

start-dfs.sh

If a node is not opened, you can go to the log under the node to view it

If all are turned on, input jps, and the pictures of the four machines are as follows




Access node01:50070 and node02:50070 with a browser. They must be in active state and standby state.


At this point, hadoop high availability is basically installed
hadoop shutdown instruction

//Shutdown cluster command:
stop-dfs.sh
//To turn off the zookeeper command:
zkServer.sh stop

Then we will prepare for MapReduce.

9. MapReduce configuration

We go to the hadoop directory of node01

cd /opt/ldy/hadoop-2.6.5/etc/hadoop/]
ll

You can see that there is a mapred-site.xml.template Documents. For security, we also copy the file for a backup. Rename the copied file

cp mapred-site.xml.template mapred-site.xml

And then we're at mapred-site.xml Add the following property

vi mapred-site.xml
<property>
       <name>mapreduce.framework.name</name>
       <value>yarn</value>
</property>

Yarn in the same directory- site.xml Add property to

<property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
<property>
   <name>yarn.resourcemanager.ha.enabled</name>
   <value>true</value>
 </property>
 <property>
   <name>yarn.resourcemanager.cluster-id</name>
   <value>cluster1</value>
 </property>
 <property>
   <name>yarn.resourcemanager.ha.rm-ids</name>
   <value>rm1,rm2</value>
 </property>
 <property>
   <name>yarn.resourcemanager.hostname.rm1</name>
   <value>node03</value>
 </property>
 <property>
   <name>yarn.resourcemanager.hostname.rm2</name>
   <value>node04</value>
 </property>
 <property>
   <name>yarn.resourcemanager.zk-address</name>
   <value>node02:2181,node03:2181,node04:2181</value>
 </property>

Then we send these two modified configuration files to node02, node03 and node04

scp mapred-site.xml yarn-site.xml node04:`pwd`
scp mapred-site.xml yarn-site.xml node05:`pwd`
scp mapred-site.xml yarn-site.xml node06:`pwd`

Since node03 and node04 are resource managers, they should be keyless to each other. First, log in to node04 on node03
Generate the secret key in the. ssh directory of node03

cd .ssh
ssh-keygen -t dsa -P '' -f ./id_dsa
//And add it to authorized_keys
cat id_dsa.pub >> authorized_keys

Use ssh localhost to verify whether a password is required, and don't forget to exit
Then distribute the public key of node03 to node04

scp id_dsa.pub node04:`pwd`/node03.pub

In the. ssh directory of node06, add node05.pub

cat node03.pub >> authorized_keys

ssh node04 on node03, is it secret key free
Same as node04

//Generate the key in the. ssh directory of node04
	ssh-keygen -t dsa -P '' -f ./id_dsa
//And add it to authorized_keys
	cat id_dsa.pub >> authorized_keys
//Use ssh localhost to verify whether you need a password. Don't forget to exit
//Distribute node04's public key to node03
	scp id_dsa.pub node05:`pwd`/node06.pub
//In the. ssh directory of node03, add node06.pub
	cat node04.pub >> authorized_keys
//ssh node03 on node04 to see if it is key free

1. Start zookeeper, all sessions zkServer.sh start
2. Start hdfs and start on node01- dfs.sh
3. Start yarn and start on node01- yarn.sh
4. Start resource manager on node03 and 04 respectively

yarn-daemon.sh start resourcemanager

5. All sessions jps, see the process is not complete
The picture is as follows




Visit node05:8088 in the browser to view the content managed by resource manager

In this way, the sub installation is basically completed
Finally, shut down the cluster

//Shut down the cluster:
//node01: 
stop-dfs.sh 
//node01: (stop nodemanager)
stop-yarn.sh 
//node03,node04: 
yarn-daemon.sh stop resourcemanager
//Node02,03,04: 
zkServer.sh stop 

Tags: ssh Hadoop xml Zookeeper

Posted on Mon, 15 Jun 2020 03:30:10 -0400 by ludachris