Construction of Hadoop running environment

1. Environment preparation of template virtual machine

1) ip address 192.168.112.100, host name Hadoop 100, memory 2G, hard disk 50G

2) Install EPEL release

[root@hadoop100 ~]# yum install -y epel-release

Note: Extra Packages for Enterprise Linux is an additional software package for the "red hat" operating system, which is applicable to RHEL, CentOS and Scientific Linux. As a software warehouse, most rpm packages cannot be found in the official repository)

Note: if the minimum system version is installed on Linux, the following tools need to be installed; If you are installing Linux Desktop Standard Edition, you do not need to perform the following operations

  • Net tool: toolkit collection, including ifconfig and other commands
[root@hadoop100 ~]# yum install -y net-tools 
  • vim: Editor
[root@hadoop100 ~]# yum install -y vim

3) Configure network open ifcfg-ens33 file

[root@hadoop100 ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33

Add the following

IPADDR=192.168.112.100
GATEWAY=192.168.112.2
DNS1=192.168.112.2

4) Add hostname

[root@hadoop100 ~]# vim /etc/hostname

Enter the following

hadoop100

5) Configure the Linux clone host name mapping hosts file and open / etc/hosts

[root@hadoop100 ~]# vim /etc/hosts

Add the following

192.168.112.100 hadoop100
192.168.112.101 hadoop101
192.168.112.102 hadoop102
192.168.112.103 hadoop103
192.168.112.104 hadoop104
192.168.112.105 hadoop105
192.168.112.106 hadoop106
192.168.112.107 hadoop107
192.168.112.108 hadoop108

6) Modify the host mapping file (hosts file) of windows

Add the following

192.168.112.100 hadoop100
192.168.112.101 hadoop101
192.168.112.102 hadoop102
192.168.112.103 hadoop103
192.168.112.104 hadoop104
192.168.112.105 hadoop105
192.168.112.106 hadoop106
192.168.112.107 hadoop107
192.168.112.108 hadoop108

7) View the virtual network editor of Linux virtual machine, edit - > virtual network editor - > VMnet8

8) View the IP address of the Windows system adapter VMware Network Adapter VMnet8

2. Turn off the firewall. Turn off the firewall and start it automatically

[root@hadoop100 ~]# systemctl stop firewalld
[root@hadoop100 ~]# systemctl disable firewalld.service

Note: during enterprise development, the firewall of a single server is usually turned off. The company will set up a very secure firewall

3. Create a ramelon user and modify the password of the ramelon user

[root@hadoop100 ~]# useradd ramelon
[root@hadoop100 ~]# passwd ramelon

4. Configure the atguigu user to have root permission, which is convenient for sudo to execute the command with root permission later

[root@hadoop100 ~]# vim /etc/sudoers

Note: the ramelon line should not be placed directly under the root line, because all users belong to the wheel group. You first configured ramelon to have a password free function, but when the program runs to the% wheel line, the function is overwritten and a password is required. So ramelon needs to be placed under the% wheel line.

5. Create a folder in the / opt directory and modify the home and group

1. Create the module and software folders in the / opt directory

[root@hadoop100 ~]# mkdir /opt/module
[root@hadoop100 ~]# mkdir /opt/software

2. Modify that the owner and group of the module and software folders are atguigu users

[root@hadoop100 ~]# chown ramelon:ramelon /opt/module 
[root@hadoop100 ~]# chown ramelon:ramelon /opt/software

3. View the owner and group of the module and software folders

6. Uninstall the JDK that comes with the virtual machine

Note: if your virtual machine is minimized, you do not need to perform this step.

[root@hadoop100 ~]# rpm -qa | grep -i java | xargs -n1 rpm -e --nodeps 

Ø

  • rpm -qa: query all installed rpm packages
  • grep -i: ignore case
  • xargs -n1: means that only one parameter is passed at a time
  • rpm -e – nodeps: force software uninstallation

7. Restart the virtual machine

[root@hadoop100 ~]# reboot

8. Clone virtual machine

Using the template machine Hadoop 100, clone three virtual machines: Hadoop 102, Hadoop 103, Hadoop 104

Note: when cloning, close Hadoop 100 first

9. Install JDK

Note: before installing the JDK, be sure to delete the JDK of the virtual machine in advance.

1) Use the XShell transport tool to import the JDK into the software folder under the opt directory

2) Enter the software directory

[ramelon@hadoop102 ~]$ cd /opt/software/

3) Unzip the JDK to the / opt/module directory

[ramelon@hadoop102 software]$ tar -zxvf jdk-8u301-linux-x64.tar.gz -C /opt/module/

4) Remember to check whether the decompression is successful

[ramelon@hadoop102 software]$ cd /opt/module/

5) Configure JDK environment variables

[ramelon@hadoop102 /]$ sudo vim /etc/profile.d/my_env.sh

Add the following

#JAVA_HOME
export JAVA_HOME=/opt/module/jdk1.8.0_301
export PATH=$PATH:$JAVA_HOME/bin

6) source click the / etc/profile file to make the new environment variable PATH effective

[ramelon@hadoop102 ~]$ source /etc/profile

7) Test for successful installation

[ramelon@hadoop102 ~]$ java -version

Note: restart (if java -version can be used, there is no need to restart)

[ramelon@hadoop102 ~]$ sudo reboot

10. Install Hadoop

Hadoop download address:

https://archive.apache.org/dist/hadoop/common/hadoop-3.1.3/

https://archive.apache.org/dist/hadoop/common/hadoop-2.7.2/

1) Use the XShell file transfer tool to import hadoop-3.1.3.tar.gz into the software folder under the opt directory

2) Enter the software directory

[ramelon@hadoop102 ~]$ cd /opt/software/

3) Unzip the installation file under / opt/module

[ramelon@hadoop102 software]$ tar -zxvf hadoop-3.1.3.tar.gz -C /opt/module/

4) Remember to check whether the decompression is successful

[ramelon@hadoop102 software]$ cd /opt/module/

5) Add Hadoop to environment variable

[ramelon@hadoop102 /]$ sudo vim /etc/profile.d/my_env.sh

Add the following in my_ Add the following at the end of env.sh file: (shift+g)

#HADOOP_HOME
export HADOOP_HOME=/opt/module/hadoop-3.1.3
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin

6) source click the / etc/profile file to make the new environment variable PATH effective

[ramelon@hadoop102 ~]$ source /etc/profile

7) Test for successful installation

[ramelon@hadoop102 ~]$ hadoop -version

Restart (restart the virtual machine if the Hadoop command cannot be used)

[ramelon@hadoop102 ~]$ sudo reboot

Tags: Linux Hadoop vim

Posted on Fri, 17 Sep 2021 11:54:16 -0400 by phpfre@k*