catalogue

Create hadoop user

Create a hadoop password and enter it twice (the password is not displayed)

Add administrator privileges for hadoop users

Update apt (installation package management tool)

Install change profile vim (vi enhanced)

Install SSH server(SSH is a security protocol specially provided for remote login sessions and other network services)

Log in to this computer

Configure SSH password less login

OpenJDK:

Unzip into / usr/local

Modify the hadoop-env.sh file as follows:

Insert export Java in the first line_ HOME=/usr/lib/jvm/java-8-openjdk-amd64​

Then modify the core-site.xml in turn   and   hdfs-site.xml, directly under the directory in the previous step

Modify profile   core-site.xml, which will

Modify to the following configuration

Similarly, modify the configuration file   hdfs-site.xml, modify to the following configuration

After modifying the above configuration, next, start hadoop. It needs to be formatted for the first time, and it won't be needed in the future.

  Then run jps to view the process

These four processes must occur, but there are also accidents

Unexpected situation

1. Jps does not have a NameNode

  2. The Jps does not have a DataNode

After successful startup, you can access the Web interface http://localhost:50070 View NameNode and Datanode information, and view files in HDFS online.

Create hadoop user

sudo useradd -m hadoop -s /bin/bash

Create a hadoop password and enter it twice (the password is not displayed)

sudo passwd hadoop

Add administrator privileges for hadoop users

sudo adduser hadoop sudo

Update apt (installation package management tool)

sudo apt-get update

Install change profile vim (vi enhanced)

sudo apt-get install vim

Install SSH server(SSH is a security protocol specially provided for remote login sessions and other network services)

sudo apt-get install openssh-server

Log in to this computer

ssh localhost

Configure SSH password less login

exit                           # Exit ssh localhost just now
cd ~/.ssh/                     # If you do not have this directory, first execute ssh localhost
ssh-keygen -t rsa              # There will be a prompt. Just press enter
cat ./id_rsa.pub >> ./authorized_keys  # Join authorization

OpenJDK:

sudo apt-get install openjdk-8-jre openjdk-8-jdk      #Install jdk without downloading
sudo vim ~/.bashrc                                    #Edit configuration
#export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64   #Insert this paragraph in the first line
source ~/.bashrc                                      #Configuration takes effect immediately  
echo $JAVA_HOME                                       #Verify environment variable values
Java -version                                         #View version number

Hadoop 2 passed http://mirror.bit.edu.cn/apache/hadoop/common/ Or http://mirrors.cnnic.cn/apache/hadoop/common/   Format: hadoop-2.x.x.tar.gz  

Unzip into / usr/local

sudo tar -zxf ~/(Location of installation package)/hadoop-2.x.x.tar.gz -C /usr/local   

cd /usr/local/
sudo mv ./hadoop-2.6.0/ ./hadoop                  # Change the folder name to hadoop
sudo chown -R hadoop ./hadoop                     # Modify file permissions
cd /usr/local/hadoop       
./bin/hadoop version                              #Check hadoop version

Modify the hadoop-env.sh file as follows:

Insert export Java in the first line_ HOME=/usr/lib/jvm/java-8-openjdk-amd64

Then modify the core-site.xml in turn   and   hdfs-site.xml, directly under the directory in the previous step

Modify profile   core-site.xml, which will

<configuration>
​​​​​​​</configuration>

Modify to the following configuration

<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:/usr/local/hadoop/tmp</value>
        <description>Abase for other temporary directories.</description>
    </property>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

Similarly, modify the configuration file   hdfs-site.xml, modify to the following configuration

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/usr/local/hadoop/tmp/dfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/usr/local/hadoop/tmp/dfs/data</value>
    </property>
</configuration>

After modifying the above configuration, next, start hadoop. It needs to be formatted for the first time, and it won't be needed in the future.

cd /usr/local/hadoop
./bin/hdfs namenode -format             #Format HDFS file system in hadoop
./sbin-dfs.sh                           #Start hadoop dfs

  Then run jps to view the process

These four processes must occur, but there are also accidents

Unexpected situation

1. Jps does not have a NameNode

./sbin/stop-dfs.sh             #Stop hadoop dfs process
rm -r tmp                      #Delete tmp
cd bin/
hadoop namenode -format        #Format
./sbin/start-dfs.sh            #Restart hadoop

  2. The Jps does not have a DataNode

./sbin/stop-dfs.sh                 #Stop hadoop dfs process
rm -r ./tmp/                       #Delete tmp
./bin/hdfs namenode -format        #Format
./sbin/start-dfs.sh                #Restart hadoop

After successful startup, you can access the Web interface http://localhost:50070 View NameNode and Datanode information, and view files in HDFS online.