Hadoop cluster management continued

catalogue

1, Hadoop cluster management

1. Access file system cluster

2, Add, repair and reduce nodes (manual management, not automatic management by the master)

1. Reinitialize the cluster

2. Overview of new datanode

3. Add new nodes and installation

4. Repair nodes

5. Delete node

3, NFS gateway

1. Overview of NFS gateway

2.NFS gateway architecture diagram

3. Purchase virtual machine and prepare

4.HDFS cluster authorization

5. Build nfsgw gateway service

  6.mount verification

seven   Error reporting processing

 

1, Hadoop cluster management

1. Access file system cluster

1) web page access

        - It can be viewed, read and written

2) Command line view

        - Can view, read and write

        / usr/local/hadoop/bin/hadoop fs - command

[root@hadoop1 hadoop]# ./bin/hadoop fs -mkdir /aaa
[root@hadoop1 hadoop]# ./bin/hadoop fs -mkdir /input
[root@hadoop1 hadoop]# ./bin/hadoop fs -ls /
Found 2 items
drwxr-xr-x   - root supergroup          0 2021-09-13 21:05 /aaa
drwxr-xr-x   - root supergroup          0 2021-09-13 21:06 /input

  View files and directories

-If the access path does not specify a prefix, read the fs.defauItFS parameter by default - access the local file system, use

/usr/local/hadoop/bin/hadoop fs -Is hdfs://hadoop1:9000//usr/local/hadoop/bin/hadoop fs -Is /

● create folders

        /usr/local/hadoop/bin/hadoop fs -mkdir /input

● create documents

        /usr/local/hadoop/bin/hadoop fs -touchz /input/tfile

hdfs management command

●. Upload files

        / usr/local/hadoop/bin/hadoop fs -put *.txt /input / ● download files

        / usr/local/hadoop/bin/hadoop fs -get /tfile /tmp/tfile ● delete files

        / usr/local/hadoop/bin/hadoop fs -rm /tmp/tfile ● delete folder

        /usr/local/hadoop/bin/hadoop fs -rmdir /dir1

  • Data analysis read the previous article

CSDNhttps://mp.csdn.net/mp_blog/creation/editor/1202159

●. Create a directory and upload files

        /usr/local/hadoop/bin/hadoop fs -mkdir /input

        /sr/local/hadoop/bin/hadoop fs -put *.xt /input/

● analyze the uploaded files and find out the words that appear most frequently

        /sr/local/hadoop/bin/Hadoop jar

        /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce - examples-2.7.7.jar wordcount /input /output

● view results

        /us/loca/hadoop/bin/hadoop fs -cat /output/*

2, Add, repair and reduce nodes (manual management, not automatic management by the master)

1. Reinitialize the cluster

Warning: this method will lose all data

Stop cluster   / usr/local/hadoop/sbin/stop-all.sh
Delete all nodes   / var/hadoop/*
Reformat / usr/local/hadoop/bin/hdfs namenode -format on Hadoop 1
Start cluster   / usr/local/hadoop/sbin/start-all.sh

[root@hadoop1 ~]# /usr/local/hadoop/sbin/stop-all.sh
[root@hadoop1 ~]# for i in hadoop1 node-{0001..0003};do
                      ssh ${i} 'rm -rf /var/hadoop/*'
                  done
[root@hadoop1 ~]# /usr/local/hadoop/bin/hdfs namenode -format
[root@hadoop1 ~]# /usr/local/hadoop/sbin/start-all.sh

2. Overview of new datanode

         ● new nodes in HDFS

                - Start a new system and set SSH password free login

                - Modify / etc/hosts on all nodes and add the host information of the new node - install the Java runtime environment (java-1. 8.0-open JDK deve 1)

                - Copy / usr/ local/hadoop of NamNode to the new node

                - Synchronize profiles to all machines

         ● start DataNode at the new node

                /usr/local/hadoop/sbin/hadoop-daemon.sh start datanode

         ● set bandwidth and balance data

                /usr/local/hadoop/bin/hdfs dfsadmin -setBalancerBandwidth500000000

                /usr/local/hadoop/sbin/start-balancer.sh

         ● check status

                /usr/local/hadoop/bin/hdfs dfsadmin -report

Since nodemanager is not responsible for saving data, it is relatively simple to add nodes, and the environment configuration is consistent with datanode

         ● add new node

                /usr/local/hadoop/sbin/yarn-daemon.sh start nodemanager

         ● view nodes (ResourceManager)

                /usr/local/hadoop/bin/yarn node -list

3. Add new nodes and installation

1) Purchase virtual machine
host      IP address                     to configure
newnode      192.168.1.54      Minimum configuration: 2-core 2G

2) Execute on Hadoop 1

[root@hadoop1 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub 192.168.1.54
[root@hadoop1 ~]# vim /etc/hosts
192.168.1.50    hadoop1
192.168.1.51    node-0001
192.168.1.52    node-0002
192.168.1.53    node-0003
192.168.1.54    newnode
#Synchronous data
[root@hadoop1 ~]# for i in node-{0001..0003} newnode;do
                      rsync -av /etc/hosts ${i}:/etc/
                  done 
[root@hadoop1 ~]# rsync -aXSH /usr/local/hadoop newnode:/usr/local/

3) Execute on the newnode node

#Environment for installing java
[root@newnode ~]# yum install -y java-1.8.0-openjdk-devel
#You can delete the log first and start the service to be the latest
[root@newnode hadoop]# rm -rf logs/
#Start service
[root@newnode ~]# /usr/local/hadoop/sbin/hadoop-daemon.sh start datanode
#Maximum synchronization bandwidth
[root@newnode ~]# /usr/local/hadoop/bin/hdfs dfsadmin -setBalancerBandwidth  500000000
#Start data balancing
[root@newnode ~]# /usr/local/hadoop/sbin/start-balancer.sh
#Add node
[root@newnode ~]# /usr/local/hadoop/sbin/yarn-daemon.sh start nodemanager
#verification
[root@newnode ~]# jps
1186 DataNode      
1431 NodeManager    #Provides data operations
1535 Jps
[root@newnode hadoop]#  ./bin/yarn node -list
21/09/13 22:04:46 INFO client.RMProxy: Connecting to ResourceManager at hadoop1/192.168.1.50:8032
Total Nodes:4
         Node-Id	     Node-State	Node-Http-Address	Number-of-Running-Containers
 node-0001:33821	        RUNNING	   node-0001:8042	                           0
   newnode:46167	        RUNNING	     newnode:8042	                           0
 node-0002:40099	        RUNNING	   node-0002:8042	                           0
 node-0003:35670	        RUNNING	   node-0003:8042	                           0

4) Validate cluster (execute on Hadoop 1)

[root@hadoop1 ~]# /usr/local/hadoop/bin/hdfs dfsadmin -report
... ...
-------------------------------------------------
Live datanodes (4):
[root@hadoop1 ~]# /usr/local/hadoop/bin/yarn node -list
... ...
Total Nodes:4

4. Repair nodes

● repairing nodes is relatively simple, and the steps are the same as adding nodes

        - Note: the ip and host name of the new node should be consistent with that of the damaged node

● if the data of the old node is lost, the new node can recover the data automatically

● data will be recovered automatically after online. If the amount of data is very large, it will take -- some time

● process of repairing nodes and adding new nodes -- to

5. Delete node

1) datanode deletion steps

        - Migrate data

        - Delete node

Configure HDFS site. XMI

        - Delete host list dfs. hosts. exc lude

<property>

< name> dfs.hosts.exclude </name>

<value> /usr/local/hadoop/etc/hadoop/exclude < /value ></property>
 

● edit delete list file

[root@localhost ~]# cat /usr/local/hadoop/etc/hadoop/exclude

newnode

You can migrate data after changing the configuration file on namenode. There is no need to synchronize the configuration file

/usr/local/hadoop/bin/hdfs dfsadmin -refreshNodes

● view status    

/usr/local/hadoop/bin/hdfs dfsadmin -report

2) Upload data example

Check the data first

[root@hadoop1 ~]# /usr/local/hadoop/bin/hdfs dfsadmin -report
Configured Capacity: 168561606656 (156.99 GB)
Present Capacity: 148584251392 (138.38 GB)
DFS Remaining: 148583858176 (138.38 GB)
DFS Used: 393216 (384 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

-------------------------------------------------
Live datanodes (4):

Name: 192.168.1.51:50010 (node-0001)
......
DFS Used: 159744 (156 KB)
......

Name: 192.168.1.52:50010 (node-0002)
......
DFS Used: 131072 (128 KB)
......


Name: 192.168.1.53:50010 (node-0003)
......
DFS Used: 65536 (64 KB)
......


Name: 192.168.1.54:50010 (newnode)
......
DFS Used: 36864 (36 KB)
......

Upload data

[root@hadoop1 ~]# ls
hadoop-2.7.7.tar.gz  mydata
[root@hadoop1 ~]# du -h mydata 
1.3G	mydata
[root@hadoop1 ~]# 
[root@hadoop1 ~]# /usr/local/hadoop/bin/hadoop fs -put mydata /
[root@hadoop1 ~]# rm -rf hadoop-2.7.7.tar.gz mydata

View data

[root@hadoop1 ~]# /usr/local/hadoop/bin/hdfs dfsadmin -report
Configured Capacity: 168561606656 (156.99 GB)
Present Capacity: 148584062680 (138.38 GB)
DFS Remaining: 145938518016 (135.92 GB)
DFS Used: 2645544664 (2.46 GB)
DFS Used%: 1.78%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

-------------------------------------------------
Live datanodes (4):

Name: 192.168.1.51:50010 (node-0001)
......
DFS Used: 405958677 (387.15 MB)
......

Name: 192.168.1.52:50010 (node-0002)
......
DFS Used: 946995249 (903.13 MB)
......

Name: 192.168.1.53:50010 (node-0003)
......
DFS Used: 511043394 (487.37 MB)
......

Name: 192.168.1.54:50010 (newnode)
......
DFS Used: 781547344 (745.34 MB)
......

Configure data migration hdfs-site.xml (done on Hadoop 1, no synchronization required)

[root@hadoop1 ~]# vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml
    <property>
        <name>dfs.hosts.exclude</name>
        <value>/usr/local/hadoop/etc/hadoop/exclude</value>
    </property>

Configure the list of excluded hosts and migrate data (executed on Hadoop 1)

# Add newnode to delete profile
[root@hadoop1 ~]# echo newnode >/usr/local/hadoop/etc/hadoop/exclude
# Migrate data
[root@hadoop1 ~]# /usr/local/hadoop/bin/hdfs dfsadmin -refreshNodes
# View the status. You can go offline only when the node status is decentralized
[root@hadoop1 ~]# /usr/local/hadoop/bin/hdfs dfsadmin -report

Offline node (newnode execution)

[root@newnode ~]# /usr/local/hadoop/sbin/hadoop-daemon.sh stop datanode
[root@newnode ~]# /usr/local/hadoop/sbin/yarn-daemon.sh stop nodemanager

3, NFS gateway

1. Overview of NFS gateway

● purpose of NFS gateway

        - Users can browse the HDFS file system through the local NFSv3 client compatible with the operating system

        - Users can stream data directly through the mount point

        - Allows HDFS to be mounted as part of the client file system

        - File attachment is supported, but random write (nolock) is not supported

        - At present, NFS gateway only supports NFSv3 and TCP protocol (vers=3, proto=tcp)

 

●   Proxy user

      - The proxy user is an authorized user for NFS gateway to access the cluster

                 Add proxy users on NameNode and NFSGW

                 The UID, GID and user name of the proxy user must be identical

        - If the user UID, GID and user name cannot be maintained for special reasons, we need to configure the static mapping relationship of nfs. map

-For example:

        uid 10 100 # Map the remote UID 10 to the local UID 100

        gid 11 101 # Map the remote GID 11 to the local GID 101

2.NFS gateway architecture diagram

3. Purchase virtual machine and prepare

host      IP address      to configure
nfsgw      192.168.1.55      Minimum configuration: 1 core 1G

HDFS user authorization
Both Hadoop 1 and nfsgw need to add users

[root@hadoop1 ~]# groupadd -g 800 nfsuser
[root@hadoop1 ~]# useradd  -g 800 -u 800 -r -d /var/hadoop nfsuser
#----------------------------------------------------------------------------------------
[root@nfsgw ~]# groupadd -g 800 nfsuser
[root@nfsgw ~]# useradd  -g 800 -u 800 -r -d /var/hadoop nfsuser

4.HDFS cluster authorization

[root@hadoop1 ~]# vim /usr/local/hadoop/etc/hadoop/core-site.xml
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hadoop1:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/var/hadoop</value>
    </property>
    <property>
        <name>hadoop.proxyuser.nfsuser.groups</name>
        <value>*</value>
    </property>
    <property>
        <name>hadoop.proxyuser.nfsuser.hosts</name>
        <value>*</value>
    </property>
</configuration>
[root@hadoop1 ~]# /usr/local/hadoop/sbin/stop-all.sh
[root@hadoop1 ~]# for i in node-{0001..0003};do
                      rsync -avXSH /usr/local/hadoop/etc ${i}:/usr/local/hadoop/
                  done
[root@hadoop1 ~]# /usr/local/hadoop/sbin/start-dfs.sh
[root@hadoop1 ~]# jps
5925 NameNode
6122 SecondaryNameNode
6237 Jps
[root@hadoop1 ~]# /usr/local/hadoop/bin/hdfs dfsadmin -report
... ...
-------------------------------------------------
Live datanodes (3):

5. Build nfsgw gateway service

1)nfsgw gateway service

        - por tmap service (conflicts with system rpcb ind) - nfs3 service (conflicts with System NFS)

        - Uninstall rpcb ind and NFS utilsyum remove y rpcbind NFS utils of NFSGW

Building Gateway Services (continued 1)

● NFS gateway has dual roles. It is both an HDFS client program and an NFS server. Therefore, NFS gateway must be able to access all nodes in the cluster and configure / etc/hosts

[root@nfsgw ~]# yum remove -y rpcbind nfs-utils
[root@nfsgw ~]# vim /etc/hosts
192.168.1.50    hadoop1
192.168.1.51    node-0001
192.168.1.52    node-0002
192.168.1.53    node-0003
192.168.1.55    nfsgw
[root@nfsgw ~]# yum install -y java-1.8.0-openjdk-devel
 rsync -aXSH --delete hadoop1:/usr/local/hadoop /usr/local/

2) Build gateway service

● nfs. exports authorization

I. nfs. exports. allowed. hosts

        - By default, export can be mounted by any client. To better control access, you can set properties. The value and string correspond to the machine name and access policy, which are separated by spaces. The format of the machine name can be a single host, a Java regular expression, or an IPv4 address

        - Use rw or ro to specify read-write or read-only permissions for the export directory.

        - The default setting is read-only

[root@nfsgw ~]# vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml
<configuration>
    <property>
        <name>dfs.namenode.http-address</name>
        <value>hadoop1:50070</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>hadoop1:50090</value>
    </property>
    <property>
        <name>dfs.replication</name>
        <value>2</value>
    </property>
    <property>
        <name>dfs.hosts.exclude</name>
        <value>/usr/local/hadoop/etc/hadoop/exclude</value>
    </property>
    <property>#Default shared root
        <name>nfs.exports.allowed.hosts</name> #Similar to / etc/exports
        <value>* rw</value>  #Similar to / dir * (RW) read / write
    </property>
    <property>
        <name>nfs.dump.dir</name> 
        <value>/var/nfstmp</value>#Temporary documents
    </property>
</configuration>

3) Build gateway service

● nfs. dump temporary directory configuration

        - nfs. dump. dir

        - The user needs to update the file dump directory parameters. NFS clients often rearrange write operations, and sequential write operations will randomly arrive at NFS gateway. This directory is often used to temporarily store unordered writes. For each file, unordered writes are dumped when they accumulate in memory beyond a certain threshold (e.g., 1M). To ensure that there is enough space for the directory, use the memory cache first, and use the hard disk when the memory is insufficient

        - NFS gateway needs to be restarted after setting this attribute

[root@nfsgw ~]# mkdir /var/nfstmp #Create staging directory
[root@nfsgw ~]# chown nfsuser.nfsuser /var/nfstmp
#Give permission to write logs
[root@nfsgw ~]# rm -rf /usr/local/hadoop/logs/*
[root@nfsgw ~]# setfacl -m user:nfsuser:rwx /usr/local/hadoop/logs
[root@nfsgw ~]# getfacl /usr/local/hadoop/logs
[root@nfsgw ~]# cd /usr/local/hadoop/
#Start service
[root@nfsgw hadoop]# ./sbin/hadoop-daemon.sh --script ./bin/hdfs start portmap
[root@nfsgw hadoop]# jps
1376 Portmap
1416 Jps
#Delete temporary file
[root@nfsgw hadoop]# rm -rf /tmp/.hdfs-nfs
[root@nfsgw hadoop]# sudo -u nfsuser ./sbin/hadoop-daemon.sh --script ./bin/hdfs start nfs3
[root@nfsgw hadoop]# sudo -u nfsuser jps
1452 Nfs3
1502 Jps

  6.mount verification

Mount NFS gateway

         ● at present, NFS can only use v3 version         

                - vers=3

         ● only TCP is used as the transmission protocol

                - proto=tcp

         ● random writing NLM is not supported

                 -nolock

         ● disable time update of access t ime

                -noatime

         ● disable acl extension permissions

                - noac I

         ● synchronous write to avoid reordering write

                -sync

[root@newnode ~]# yum install -y nfs-utils
[root@newnode ~]# showmount -e 192.168.1.55
Export list for 192.168.1.55:
/ *
[root@newnode ~]# mount -t nfs -o vers=3,proto=tcp,nolock,noacl,noatime,sync 192.168.1.55:/ /mnt/
[root@newnode ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
192.168.1.55:/  118G   15G  104G  13% /mnt

seven   Error reporting processing

  Error 1. When the file is lost and there is no backup, you can only use the format

 

[root@hadoop1 ~]# /usr/local/hadoop/sbin/stop-all.sh
[root@hadoop1 ~]# for i in hadoop1 node-{0001..0003};do
                      ssh ${i} 'rm -rf /var/hadoop/*'
                  done
[root@hadoop1 ~]# /usr/local/hadoop/bin/hdfs namenode -format
[root@hadoop1 ~]# /usr/local/hadoop/sbin/start-all.sh

  Common errors

● the root user is required to start portmap

● to start nfs3, you need to use the proxy user set in the core site ● you must authorize the proxy user

        - / If var/nfstmp is not authorized to upload files, an error will occur

        - / usr/ loca 1/hadoop/ logs is not authorized and cannot see the error log ● you must start portmap before starting nfs3

● if portmap is restarted, nfs3 must also be restarted after the restart

Tags: Python Big Data Hadoop

Posted on Fri, 19 Nov 2021 01:29:18 -0500 by anita999

Hot Tags

©2023 Programmer Help