Build a simple cluster version based on Hadoop stand-alone version

0. planning

This time, a host and two slaves are set up, that is, there are only two node nodes

You can also let the host have node nodes, and then you will say

host name	cluster planning
master	namenode,secondarynamenode,ResourceManager
slave1	Datanode, NodeManager
slave2	Datanode, NodeManager

Clone a stand-alone virtual machine first

1. Clone virtual machine

Right click > Manage > clone and select create full clone

If you want to clone several clusters, I will clone three, one host and two slaves

2. Start the virtual machine and change the static ip

Modify the IP addresses of the three virtual machines: master, slave1 and slave2

Or change ifcfg-eno16777736 file in / etc / sysconfig / network scripts folder

vim /etc/sysconfig/network-scripts/ifcfg-eno16777736

The changes are as follows:

As long as the ip of this virtual machine is not used

Then save and restart the service

systemctl restart network

3. Modify ip mapping in hosts file

Write the ip address and host name of the three hosts, find the hosts file and edit it

vim /etc/hosts

192.168.27.101 master
192.168.27.102 slave1
192.168.27.103 slave2

It is better to reboot the virtual machine reboot to make it effective after the changes, so that no exception will occur when ssh keyless login is finally configured

4. Modify hostname

Modify the host names of the master, slave1 and slave2 hosts respectively

hostnamectl set-hostname master
hostnamectl set-hostname slave1
hostnamectl set-hostname slave2

5. Modify hadoop configuration file

Go to / usr/hadoop/hadoop-2.9.2/etc/hadoop/ and modify core-site.xml and yarn-site.xml

All three virtual machines need to be changed, or only one can be modified and then sent

Change the previous host name to the current host master

5.1 core site.xml modification

Change to

5.2 modification of yarn-site.xml

Change to

5.3 changing the slaves file

Or under this folder, change the slaves file. All three virtual machines need to be changed

vim slaves

Changed to:

# localhost

slave1
slave2

It needs to be explained here that if you want the host to be a node, then you need to add the host name. This file tells hadoop whose datanode to start

If it's sent, it's

#It will directly cover the previous
[root@master .ssh]# scp -r /usr/hadoop root@slave1:/usr/
[root@master .ssh]# scp -r /usr/hadoop root@slave2:/usr/

6. Cluster version ssh keyless login

All three virtual machines need to be operated

# Each machine first uses ssh to generate a. ssh folder in the home directory
ssh master # master is the host name

#Each machine first uses ssh to generate a. ssh folder in the home directory
[root@slave1 ~]# ssh slave1
The authenticity of host 'slave1 (192.168.27.102)' can't be established.
ECDSA key fingerprint is 26:e5:4a:4a:55:1e:ee:cf:51:23:0d:d8:75:f0:0d:02.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave1,192.168.27.102' (ECDSA) to the list of known hosts.
root@slave1's password: 
Last login: Tue Feb 25 17:59:40 2020 from 192.168.27.1


# Enter ~ /. ssh (~ is essentially to create a password free for root)
[root@slave1 ~]# cd ~/.ssh/

# Enter the following command and enter all the way to generate the public key and secret key
[root@slave1 .ssh]# ssh-keygen -t rsa -P ''
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
1b:21:bc:c6:59:a2:df:89:4a:68:34:1b:2f:7d:c2:a5 root@slave1
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|     .           |
|      + o        |
|     o * .       |
|  + . * S        |
| . O = o +       |
|  = E + +        |
| . o +           |
|    
+-----------------+
[root@slave1 .ssh]# cp id_rsa.pub authorized_keys

If more authorized ABCD keys is as follows:

[root@slave1 .ssh]# more authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDKdHLDeZMQnYttxmR86WxyK627XksHxmiqovo/tWYcdpVN1DcAKRNvDa5dr0Yxer0E1O
qol0fIKu1Qak+Rv7y8K1HymTFuo6E0UWNS6MM3jn0Bv2F+9gN4tn+VrtSn2HOqnnI4g+1u4aPvxYc9GGU5xyQ/KkXiQc3xdmx136OkqzBJj
NNwf41ekU1clzpZcVnFkDf4bB6hLf56XGVJWnyHHSBxbU85PeCgih/iXyABxh+LwwCFHMoDdpZyi4p/+qPpHpyWy7ipEUnGK162u7dKdtLn
TL554h2gP/vpK4zxQRmC1XubM5w4nUFEEWeZquU9viyEReoDLo4uzddtESZ root@slave1

@After that, your host name means normal. Otherwise, repeat the previous steps to overwrite

Then merge the authorized keys files of slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave slave Slave

#Merge all the authorized keys files (the easiest way is to append the contents of the other three slave hosts to the master host)
[root@slave1 .ssh]# cat ~/.ssh/authorized_keys | ssh root@master 'cat >> ~/.ssh/authorized_keys'
[root@slave2 .ssh]# cat ~/.ssh/authorized_keys | ssh root@master 'cat >> ~/.ssh/authorized_keys'


#View the contents of the authorized keys file on the master as follows
[root@master .ssh]# more authorized_keys 

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDx53xU7H7mRHSi6jLFKxiCaTeUpkj1x64kLFmyfAs0ruccZt3EpIYiaCGg6O68sKOX32M
T+UzvxePJ/rLbo+ZVV8FG+jH4wb7MMdp1lBtbspFytAeGNrAgIW5VOX81CDTaV6hvVrJCl6aXKpc5d4xnHwkTMrQ2Tv6Jqe3C5HR+xdn7rU
KFEsIzzJoD8KwudZkCVlzFX+B+UcDdaaOZsrrzPR8TpXRjUHSDQ6nbjMKtDh08jrndtRQd1F9hmfkOBevj9WHoe7G5Ngh5ssq9rPflV7Gk/
SMwe7QtrKPIa7JiXulrlaPm109c8R5H/fHUehjYl1HcP2P67QcdJ1VYkK7V root@master
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDKdHLDeZMQnYttxmR86WxyK627XksHxmiqovo/tWYcdpVN1DcAKRNvDa5dr0Yxer0E1O
qol0fIKu1Qak+Rv7y8K1HymTFuo6E0UWNS6MM3jn0Bv2F+9gN4tn+VrtSn2HOqnnI4g+1u4aPvxYc9GGU5xyQ/KkXiQc3xdmx136OkqzBJj
NNwf41ekU1clzpZcVnFkDf4bB6hLf56XGVJWnyHHSBxbU85PeCgih/iXyABxh+LwwCFHMoDdpZyi4p/+qPpHpyWy7ipEUnGK162u7dKdtLn
TL554h2gP/vpK4zxQRmC1XubM5w4nUFEEWeZquU9viyEReoDLo4uzddtESZ root@slave1
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH7qis2UUgPsaT4LkdQylwC66Rm+VDEBe7dIKo52LVlqL4ws9IZ9YRV+htvTda+PWbQ9L
ppf+tZRfM1Cz5MbEvcvaYDRsfsqz8KVVAtT71bRmZC5p8q+OgcOsq1o4BWdAl+zHH+rTAizjabELOzVnhATnA52AxSrFki/mc8QDitzTcFF
2NUeskhZC5BhOL5miF/+mTc1Y/VG3lzB4rhnPJVnA1bay9HalaBaxVnx3+OZhJojUKxRGs7qUZih+OVxqMvMdHeIthvk+TwUkt/EfOMmwzW
glK8X6Xtk0w0AP+z8d90mwFL8BMV937RWsR2nMpOf5dPQjnVDqOM+mGN82z root@slave2

Then distribute the authorized keys on the host

#Distribute the authorized keys file on the master to other hosts
[root@master ~]# scp ~/.ssh/authorized_keys root@slave1:~/.ssh/
[root@master ~]# scp ~/.ssh/authorized_keys root@slave2:~/.ssh/

Verify password free success

7. Reformat namenode

All three virtual machines need

hadoop namenode -format

8. Start Hadoop

Start start-all.sh directly on the host master, followed by the opportunity

[root@master .ssh]# start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [master]
master: starting namenode, logging to /usr/hadoop/hadoop-2.9.2/logs/hadoop-root-namenode-master.out
slave1: starting datanode, logging to /usr/hadoop/hadoop-2.9.2/logs/hadoop-root-datanode-slave1.out
slave2: starting datanode, logging to /usr/hadoop/hadoop-2.9.2/logs/hadoop-root-datanode-slave2.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/hadoop/hadoop-2.9.2/logs/hadoop-root-secondarynamenode-master.out
starting yarn daemons
starting resourcemanager, logging to /usr/hadoop/hadoop-2.9.2/logs/yarn-root-resourcemanager-master.out
slave2: starting nodemanager, logging to /usr/hadoop/hadoop-2.9.2/logs/yarn-root-nodemanager-slave2.out
slave1: starting nodemanager, logging to /usr/hadoop/hadoop-2.9.2/logs/yarn-root-nodemanager-slave1.out

Then view jps on the host and slave respectively, and the result should be the same as the plan

host name	cluster planning
master	namenode,secondarynamenode,ResourceManager
slave1	Datanode, NodeManager
slave2	Datanode, NodeManager

9. Verify that the Hadoop cluster is started successfully

Check ip:50070 of http: / / host master and ip:8088 of http: / / host master respectively

Local:

• http://192.168.27.101:50070
• http://192.168.27.101:8088

If so, the startup is successful. So far, the simple hadoop cluster version has been built