Installation of JDK, Hadoop, Hbase and GUI on Linux Server (Ubuntu 18.04)

Article Directory

0. Preparations

First of all, you need to prepare the following (VNC Viewer is perfect for little novices like me who can't see the GUI and panic):

When choosing JDK, Hadoop, HBase versions, be aware of their compatibility

Official Documents (Search for content in the document: Ctrl + F)

1. Get Ali Cloud ECS Server

System: Ubuntu 18.04

During epidemic prevention and control(viii), available to students A 6-Month Free Aliyun ECS server

Acquire conditions:

  • Account is authenticated by real name & student certification
  • Complete a test of 10 multiple choice questions (more than 6 correct answers)

2. Create Hadoop users(viii)

# Create a hadoop user and use/bin/bash as a shell
sudo useradd -m hadoop -s /bin/bash  

# Set password for hadoop user, enter password twice
sudo passwd hadoop         

# Administrator privileges for hadoop users to facilitate deployment
sudo adduser hadoop sudo       

# Switch current user to hadoop user
su - hadoop             

# Update apt of hadoop users to facilitate subsequent software installation
sudo apt-get update

3. Configure SSH passwordless login

# 1. Server-side Installation
sudo apt-get install openssh-server
 
# Client Installation
# Ubuntu has SSH client installed by default
# sudo apt-get install openssh-client


------------------------
# 2. Enter the host directory, generate the local secret key and set up the secret-free login
# >> Note that instead of using the root user to generate the secret key, you want to use the user you want to set << < here
cd /home/hadoop/
 
ssh-keygen -t rsa -P ""
 
# 2.1 Append public keys to authorized_keys file
cat .ssh/id_rsa.pub >> .ssh/authorized_keys
# 2.2 Give authorized_keys file permissions
chmod 600 .ssh/authorized_keys


------------------------
# 3. Verify success
ssh localhost

4. Install JDK

4.1 Method 1:_Manual installation (recommended)_)

4.1.1 Process

  1. Computer Download Linux_JDK8
  2. Use WinSCP Upload the compressed package to the server
  3. Unzip and install JDK8

4.1.2 Download Linux JDK8 on PC

Get Linux_JDK8 (Download requires login to Oracle account)

4.1.3 Use WinSCP to upload compressed packages to the server

Get WinSCP

  1. Fill in server information
  1. Connect to Server
  1. Upload Compressed Packet

4.1.4 Unzip and install JDK8

# 1. Create a directory
sudo mkdir /usr/lib/jvm

------------------------
# 2. Unzip the JDK into the directory
# >> Note that the file names of the following commands match their own JDK package names < <<
sudo tar -zxvf jdk-8u241-linux-x64.tar.gz -C /usr/lib/jvm

------------------------
# 3. Modify environment variables
# 3.1 Edit.bashrc
# Related actions:'i': Enter edit mode,'esc': Exit edit mode,': wq': Save and exit
sudo vi ~/.bashrc

# 3.2 Add the following at the end of the file
# >> Note that the catalog and JDK are consistent with what you created < < < < < < < < < < < < < < < < < < < < < < <
export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_241
export JRE_HOME=${JAVA_HOME}/jre  
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib  
export PATH=${JAVA_HOME}/bin:$PATH 

# 3.3 Make environment variables take effect immediately
source ~/.bashrc

------------------------
# 4. The system registers this JDK
# >> Note that the JDK is consistent with what you've created < < < < <.
sudo update-alternatives --install /usr/bin/java java /usr/lib/jvm/jdk1.8.0_241/bin/java 300

------------------------
# 5. Check to see if the installation was successful
java -version

4.2 Method 2: Command line installation (not recommended!_)

I see many JDK command line installation tutorials written like this, but I did not install JDK in this way until the end...😭😤

# 1. Installation Dependency
sudo apt-get install python-software-properties
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update

# 2. Install jdk1.8
sudo apt-get install oracle-java8-installer

4.2.1 Problems encountered during installation

  1. E: Package 'python-software-properties' has no installation candidate

    # >> Solutions < <*
    # System >= Ubuntu 12.10 Use'software-properties-common'instead of'python-software-properties'
    sudo apt-get install software-properties-common
    
  2. add-apt-repository command not found

    # >> Solutions < <*
    # Install software-properties-common
    apt-get install software-properties-common
    
  3. E: Package 'oracle-java8-installer' has no installation candidate

    # >> Trying any of the following will not work < < <
    # 1. Add dependencies
    # 2. Update apt-get
    # 3. Replace apt-get with aptitude
    # 4. sudo update-alternatives --config java    # This method is used for switching between multiple versions of jdk
    
    
    # >> Solutions < <*
    # 1. Manual Installation
    #   (See Method 2: Manual Installation for details)
    
    # 2. Replace with OpenJDK
    # 2.1 Update package list
    sudo apt-get update
    # 2.2 Install openjdk
    sudo apt-get install openjdk-8-jdk
    # 2.3 Check to see if the installation was successful
    java -version
    

5. Graphical interface

5.1 Process

  1. Server opens ports used by VNC
  2. Server Installation Ubuntu Desktop and VNC Server
  3. Client Installation VNC Viewer
  4. Connect

5.2 Server opens ports used by VNC

  1. Enter Security Group
  1. Add Security Group Rule

⚠Note: If the port is not open, the following prompts will appear when connecting with the VNC Viewer:

5.3 Server Install Ubuntu Desktop and VNC Server

# 1. Update apt
sudo apt-get update && sudo apt-get upgrade -y

------------------------
# 2. Install desktop
# During installation, enter'y'and return if prompted to change the system file to a new version
# The installation process lasts approximately 10 minutes
sudo apt-get install ubuntu-desktop gnome-panel gnome-settings-daemon metacity nautilus gnome-terminal

------------------------
# 3. Install VNC Server
sudo apt-get install vnc4server

------------------------
# 4. Start VNC Server and create a session
# >> Note that there is a space between'vncserver'and':' < <
# Next you need to set the password
vncserver :1

------------------------
# 5. Close Session
vncserver -kill :1

------------------------
# 6. Edit ~/.vnc/xstartup
# This file will not be created until VNC Sever is first started
nano ~/.vnc/xstartup

# > nano part operation:'ctrl + o'save,'Enter' confirm,'ctrl + x'exit < < <
# 7. Change the file to:

#!/bin/sh

# Uncomment the following two lines for normal desktop:
# unset SESSION_MANAGER
# exec /etc/X11/xinit/xinitrc

[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &
x-terminal-emulator -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &
x-window-manager &

gnome-panel &
gnome-settings-daemon &
metacity &
nautilus &
# 8. Open VNC Server again
vncserver :1

5.4 Client Installation VNC Viewer (omitted)

Get VNC Viewer

5.5 Connection(vii)

  1. Enter Server IP +:1, Enter
  1. Enter the password (password set after vncserver:1 was first entered on the command line) and click OK
  1. _Great success!(viii)
Test the network again~

6. Install Hadoop_

6.1 Process

  1. Computer Download Hadoop 2.7.1 (omitted)
  2. Upload Hadoop compressed packages to a Linux server using WinSCP (same as 4.1.3, omitted)
  3. Unzip and install Hadoop 2.7.1

6.2 Unzip and install Hadoop 2.7.1

# 1. Decompression
# 1.1 Unzip to/usr/local directory
sudo tar -zxvf  hadoop-2.7.1.tar.gz -C /usr/local

# 1.2 Enter / usr/local directory
cd /usr/local

# 1.3 Renamed hadoop
sudo mv hadoop-2.7.1 hadoop

# 1.4 Modify file permissions
sudo chown -R hadoop ./hadoop


------------------------
# 2. Configuring environment variables
# 2.1 Edit.bashrc
vim ~/.bashrc

# 2.2 Add the following
# Hadoop Environment
export HADOOP_HOME=/usr/local/hadoop
export CLASSPATH=$($HADOOP_HOME/bin/hadoop classpath):$CLASSPATH
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

# 2.3 Make environment variables take effect immediately
source ~/.bashrc

# 2.4 Check to see if the installation was successful
hadoop version
# 3. Add jdk path in hadoop-env.sh
# 3.1 Enter the / usr/local/hadoop/etc/hadoop directory
cd /usr/local/hadoop/etc/hadoop

# 3.2 Edit hadoop-env.sh
vim hadoop-env.sh

# 3.3 Modify the JAVA_HOME value
# Before modification: export JAVA_HOME=${JAVA_HOME}
export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_241
# 4. Still in the / usr/local/hadoop/etc/hadoop directory, modify both profiles

# 4.1 Modify the core-site.xml file
# 4.1.1 Edit core-site.xml
vim core-site.xml

# 4.1.2 Add the following configuration
<configuration>
        <property>
             <name>hadoop.tmp.dir</name>
             <value>file:/usr/local/hadoop/tmp</value>
             <description>Abase for other temporary directories.</description>
        </property>
        <property>
             <name>fs.defaultFS</name>
             <value>hdfs://localhost:9000</value>
        </property>
</configuration>

----------------
# 4.2 Modify the hdfs-site.xml file
# 4.2.1 Edit core-site.xml
vim hdfs-site.xml

# 4.2.2 Add the following configuration
<configuration>
        <property>
             <name>dfs.replication</name>
             <value>1</value>
        </property>
        <property>
             <name>dfs.namenode.name.dir</name>
             <value>file:/usr/local/hadoop/tmp/dfs/name</value>
        </property>
        <property>
             <name>dfs.datanode.data.dir</name>
             <value>file:/usr/local/hadoop/tmp/dfs/data</value>
        </property>
</configuration>


------------------------
# 5. Start hadoop
# 5.1 Enter/usr/local/hadoop
cd /usr/local/hadoop

# 5.2 Perform NameNode Formatting
./bin/hdfs namenode –format

# 5.3 Start Namenode and Datanode processes
./sbin/start-dfs.sh

# 5.4 View startup results
jps

A solution without NameNode process under 6.3 jps_

# 1. Turn off dfs
stop-dfs.sh

# 2. Format namenode
hadoop namenode -format

# 3. Restart Services
start-all.sh

7. Install HBase

7.1 Process

  1. Computer Download HBase 1.6.0 (omitted)
  2. Uploading HBase compressed packages to a Linux server using WinSCP (same as 4.1.3, omitted)
  3. Unzip and install HBase 1.6.0

7.2 Unzip and install HBase 1.6.0

# 1. Decompression
# 1.1 Unzip to/usr/local directory
sudo tar -zxf hbase-1.6.0-bin.tar.gz  -C/usr/local  

# 1.2 Enter / usr/local directory
cd /usr/local

# 1.3 Rename
mv ./hbase-1.6.0 ./hbase

# 1.4 Modify permissions
sudo chown -R hadoop:hadoop  ./hbase  


------------------------
# 2. Configure command line environment variables
# 2.1 Edit/etc/profile
vim /etc/profile

# 2.2 Add environment variables
export HBASE_HOME=/usr/local/hbase
export PATH=$HBASE_HOME/bin:$PATH
# 3. Modify hbase's configuration file/conf/hbase-env.sh
# 3.1 Edit hbase-env.sh
vim /usr/local/hbase/conf/hbase-env.sh

# 3.2 Add the following
export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_241
export HBASE_MANAGES_ZK=true
# 4. Edit the.Xml configuration file conf/hbase-site.xml
# 4.1 Edit hbase-site.xml
vim /usr/local/hbase/conf/hbase-site.xml

# 4.2 Add the following
<configuration>
        <property>
                <name>hbase.rootdir</name>
                <value>hdfs://localhost:9000/hbase</value>
                <description>The location where the data is stored.</description>
        </property>
        <property>
                <name>hbase.cluster.distributed</name>
                <value>true</value>
        </property>
        <property>
                <name>hbase.zookeeper.quorum</name>
                <value>localhost</value>
        </property>
        <property>
                <name>dfs.replication</name>
                <value>1</value>
                <description>The number of copies specified is 1 because of pseudo-distributed.</description>
        </property>
</configuration>
# 5. Start HBase
# >> Before starting HBase, make sure Hadoop is functioning properly <<<
# >> Command line input jps, showing 4 Java processes <<<<*
# >> jps, NameNode, DataNode, SecondaryNameNode <<< respectively
start-hbase.sh

# After starting HBase, executing jps again adds three new Java processes
# HMaster, HRegionServer, HQuorumPeer

The browser accesses localhost:16010 and enters the Web administration page

⚠NOTE: Ports prior to HBASE1.0 are 60010

The HBase database on the Ubuntu Linux server is now installed!___

8. References

Fifteen original articles have been published. 6. Visits 388
Private letter follow

Tags: Hadoop HBase sudo vnc

Posted on Fri, 17 Apr 2020 02:57:09 -0400 by zgkhoo