Hadoop pseudo distributed architecture

preparation

{
Create virtual machine
Install CentOS7 image files
Configure IP
}
These three steps can search other blogs, which will not be described in detail in this article.
Directly enter the Java installation and Hadoop configuration of Linux

Required files

jdk-8u77-linux-x64.tar.gz
hadoop-2.7.3.tar.gz

Software similar to Xshell connecting virtual machine, which is used to transfer files to the specified directory of virtual machine.

Xshell connect virtual machine



Click the connect button
Follow the prompts to enter the user name and password of your virtual machine
My username is xwk

In this way, the connection is successful. The configuration of our files is carried out in Xshell, because Xshell can be copied during vi editing!

Start transferring Java and Hadoop files to the specified directory of the virtual machine

New software folder

[root@master xwk]# cd /home/xwk/
[root@master xwk]# mkdir software
[root@master xwk]# ls
software
[root@master xwk]# cd software/
[root@master software]#

Give the folder sufficient permissions, otherwise the transfer may fail.

[root@master software]# chmod 777 /home/xwk/software/
[root@master software]#

Click the green button

Just drag the two files from the computer on the left to the interface of the virtual machine on the right

ls to check whether the file was transferred successfully

[root@master software]# ls
hadoop-2.7.3.tar.gz  jdk-8u77-linux-x64.tar.gz
[root@master software]#

Unzip the two compressed packages

[root@master software]# tar -zxf jdk-8u77-linux-x64.tar.gz 
[root@master software]# tar -zxf hadoop-2.7.3.tar.gz 
[root@master software]# ls
hadoop-2.7.3  hadoop-2.7.3.tar.gz  jdk1.8.0_77  jdk-8u77-linux-x64.tar.gz
[root@master software]# 

Modify / etc/profile file

vim /etc/profile

If present

[root@master software]# vim /etc/profile
bash: vim: Command not found
[root@master software]#

implement

[root@master software]# yum -y install vim-enhanced

Continue editing file

vim /etc/profile

Next, insert the following configuration information into the file
Xshell can be pasted directly. If you knock with your hand, you must see the letters clearly and don't input them wrong.

JAVA_HOME=/home/xwk/software/jdk1.8.0_77
JRE_HOME=/home/xwk/software/jdk1.8.0_77/jre

PATH=$JAVA_HOME/bin:$PATH
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export JAVA_HOME PATH CLASSPATH
export HADOOP_HOME=/home/xwk/software/hadoop-2.7.3
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/lib
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib"

Make the configuration file effective.

source /etc/profile

Check whether Java is successfully installed

[root@master software]# java -version
java version "1.8.0_77"
Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)
[root@master software]# 

Configure Hadoop files

All files are configured in the following directory

cd /home/xwk/software/hadoop-2.7.3/etc/hadoop/

Configure the core-site.xml file

vim core-site.xml
<configuration>
<property>
        <name>hadoop.tmp.dir</name>
        <value>/home/xwk/software/hadoop-2.7.3/tmp</value>
        <description>A base for other temporary directiories.</description>
</property>
<property>
         <name>fs.default.name</name>
         <value>hdfs://localhost:9000</value>
</property>
</configuration>

Configure hadoop-env.sh file

vim hadoop-env.sh
export HADOOP_HEAPSIZE=2000
export JAVA_HOME=/home/xwk/software/jdk1.8.0_77
export HADOOP_IDENT_STRING=$USER

Configuration yarn-env.sh file

vim yarn-env.sh
export JAVA_HOME=/home/xwk/software/jdk1.8.0_77

Configure mapred-site.xml file

Because this file didn't exist before, you have to copy it first

cp mapred-site.xml.template mapred-site.xml
vim mapred-site.xml
<configuration>
<property>
        <name>mapred.job.tracker</name>
        <value>localhost:9001</value>
</property>
<property>
         <name>mapreduce.cluster.local.dir</name>
         <value>/home/xwk/software/hadoop-2.7.3/hadoop_dir/mapred/local</value>
</property>
<property>
         <name>mapreduce.jobtracker.system.dir</name>
         <value>/home/xwk/software/hadoop-2.7.3/hadoop_dir/mapred/system</value>
</property>
</configuration>

Configure hdfs-site.xml file

vim hdfs-site.xml
<configuration>
<property>
        <name>dfs.namenode.name.dir</name>
        <value>/home/xwk/software/hadoop-2.7.3/hadoop_dir/dfs/name</value>
</property>
<property>
         <name>dfs.datanode.data.dir</name>
         <value>/home/xwk/software/hadoop-2.7.3/hadoop_dir/dfs/data</value>
</property>
</configuration>

format

cd /home/xwk/software/hadoop-2.7.3
hadoop namenode -format

start-up

cd /home/xwk/software/hadoop-2.7.3/sbin

You need to constantly enter your password for confirmation

./start-all.sh


Enter jps and the following results show that the startup is successful

[root@master sbin]# jps
11205 NameNode
11909 NodeManager
12006 Jps
11479 SecondaryNameNode
11625 ResourceManager
11325 DataNode
[root@master sbin]#

If you edit all in Xshell, you should run the following command again in the virtual machine (this is very important!!!)

Make configuration effective

source /etc/profile

Turn off the firewall (this is important)

Once closed
[root@master sbin]# systemctl stop firewalld.service
 Disable firewall startup
[root@master sbin]# systemctl disable firewalld.service
[root@master sbin]# 

Browser authentication

Input separately

http://192.168.10.20:8088/
http://192.168.10.20:50070/



If you have reached this step, Congratulations, your Hadoop pseudo distributed construction has been successful!

Close (this is important!!!)

You need to constantly enter your password for confirmation
Be sure to shut down Hadoop before shutting down the virtual machine!!!
Otherwise, unexpected errors will occur during the next boot!!!

./stop-all.sh

The following results should occur when you enter jps

[root@master sbin]# jps
12006 Jps
[root@master sbin]#

Tags: Linux Big Data Hadoop

Posted on Fri, 10 Sep 2021 13:52:12 -0400 by Wien