catalogue
2, Add, repair and reduce nodes (manual management, not automatic management by the master)
3. Add new nodes and installation
2.NFS gateway architecture diagram
3. Purchase virtual machine and prepare
5. Build nfsgw gateway service
seven Error reporting processing
1, Hadoop cluster management
1. Access file system cluster
1) web page access
- It can be viewed, read and written
2) Command line view
- Can view, read and write
/ usr/local/hadoop/bin/hadoop fs - command
[root@hadoop1 hadoop]# ./bin/hadoop fs -mkdir /aaa [root@hadoop1 hadoop]# ./bin/hadoop fs -mkdir /input [root@hadoop1 hadoop]# ./bin/hadoop fs -ls / Found 2 items drwxr-xr-x - root supergroup 0 2021-09-13 21:05 /aaa drwxr-xr-x - root supergroup 0 2021-09-13 21:06 /input
View files and directories
-If the access path does not specify a prefix, read the fs.defauItFS parameter by default - access the local file system, use
/usr/local/hadoop/bin/hadoop fs -Is hdfs://hadoop1:9000//usr/local/hadoop/bin/hadoop fs -Is /
● create folders
/usr/local/hadoop/bin/hadoop fs -mkdir /input
● create documents
/usr/local/hadoop/bin/hadoop fs -touchz /input/tfile
hdfs management command
●. Upload files
/ usr/local/hadoop/bin/hadoop fs -put *.txt /input / ● download files
/ usr/local/hadoop/bin/hadoop fs -get /tfile /tmp/tfile ● delete files
/ usr/local/hadoop/bin/hadoop fs -rm /tmp/tfile ● delete folder
/usr/local/hadoop/bin/hadoop fs -rmdir /dir1
- Data analysis read the previous article
CSDNhttps://mp.csdn.net/mp_blog/creation/editor/1202159
●. Create a directory and upload files
/usr/local/hadoop/bin/hadoop fs -mkdir /input
/sr/local/hadoop/bin/hadoop fs -put *.xt /input/
● analyze the uploaded files and find out the words that appear most frequently
/sr/local/hadoop/bin/Hadoop jar
/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce - examples-2.7.7.jar wordcount /input /output
● view results
/us/loca/hadoop/bin/hadoop fs -cat /output/*
2, Add, repair and reduce nodes (manual management, not automatic management by the master)
1. Reinitialize the cluster
Warning: this method will lose all data
Stop cluster / usr/local/hadoop/sbin/stop-all.sh
Delete all nodes / var/hadoop/*
Reformat / usr/local/hadoop/bin/hdfs namenode -format on Hadoop 1
Start cluster / usr/local/hadoop/sbin/start-all.sh
[root@hadoop1 ~]# /usr/local/hadoop/sbin/stop-all.sh [root@hadoop1 ~]# for i in hadoop1 node-{0001..0003};do ssh ${i} 'rm -rf /var/hadoop/*' done [root@hadoop1 ~]# /usr/local/hadoop/bin/hdfs namenode -format [root@hadoop1 ~]# /usr/local/hadoop/sbin/start-all.sh
2. Overview of new datanode
● new nodes in HDFS
- Start a new system and set SSH password free login
- Modify / etc/hosts on all nodes and add the host information of the new node - install the Java runtime environment (java-1. 8.0-open JDK deve 1)
- Copy / usr/ local/hadoop of NamNode to the new node
- Synchronize profiles to all machines
● start DataNode at the new node
/usr/local/hadoop/sbin/hadoop-daemon.sh start datanode
● set bandwidth and balance data
/usr/local/hadoop/bin/hdfs dfsadmin -setBalancerBandwidth500000000
/usr/local/hadoop/sbin/start-balancer.sh
● check status
/usr/local/hadoop/bin/hdfs dfsadmin -report
Since nodemanager is not responsible for saving data, it is relatively simple to add nodes, and the environment configuration is consistent with datanode
● add new node
/usr/local/hadoop/sbin/yarn-daemon.sh start nodemanager
● view nodes (ResourceManager)
/usr/local/hadoop/bin/yarn node -list
3. Add new nodes and installation
1) Purchase virtual machine
host IP address to configure
newnode 192.168.1.54 Minimum configuration: 2-core 2G
2) Execute on Hadoop 1
[root@hadoop1 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub 192.168.1.54 [root@hadoop1 ~]# vim /etc/hosts 192.168.1.50 hadoop1 192.168.1.51 node-0001 192.168.1.52 node-0002 192.168.1.53 node-0003 192.168.1.54 newnode #Synchronous data [root@hadoop1 ~]# for i in node-{0001..0003} newnode;do rsync -av /etc/hosts ${i}:/etc/ done [root@hadoop1 ~]# rsync -aXSH /usr/local/hadoop newnode:/usr/local/
3) Execute on the newnode node
#Environment for installing java [root@newnode ~]# yum install -y java-1.8.0-openjdk-devel #You can delete the log first and start the service to be the latest [root@newnode hadoop]# rm -rf logs/ #Start service [root@newnode ~]# /usr/local/hadoop/sbin/hadoop-daemon.sh start datanode #Maximum synchronization bandwidth [root@newnode ~]# /usr/local/hadoop/bin/hdfs dfsadmin -setBalancerBandwidth 500000000 #Start data balancing [root@newnode ~]# /usr/local/hadoop/sbin/start-balancer.sh #Add node [root@newnode ~]# /usr/local/hadoop/sbin/yarn-daemon.sh start nodemanager #verification [root@newnode ~]# jps 1186 DataNode 1431 NodeManager #Provides data operations 1535 Jps [root@newnode hadoop]# ./bin/yarn node -list 21/09/13 22:04:46 INFO client.RMProxy: Connecting to ResourceManager at hadoop1/192.168.1.50:8032 Total Nodes:4 Node-Id Node-State Node-Http-Address Number-of-Running-Containers node-0001:33821 RUNNING node-0001:8042 0 newnode:46167 RUNNING newnode:8042 0 node-0002:40099 RUNNING node-0002:8042 0 node-0003:35670 RUNNING node-0003:8042 0
4) Validate cluster (execute on Hadoop 1)
[root@hadoop1 ~]# /usr/local/hadoop/bin/hdfs dfsadmin -report ... ... ------------------------------------------------- Live datanodes (4): [root@hadoop1 ~]# /usr/local/hadoop/bin/yarn node -list ... ... Total Nodes:4
4. Repair nodes
● repairing nodes is relatively simple, and the steps are the same as adding nodes
- Note: the ip and host name of the new node should be consistent with that of the damaged node
● if the data of the old node is lost, the new node can recover the data automatically
● data will be recovered automatically after online. If the amount of data is very large, it will take -- some time
● process of repairing nodes and adding new nodes -- to
5. Delete node
1) datanode deletion steps
- Migrate data
- Delete node
Configure HDFS site. XMI
- Delete host list dfs. hosts. exc lude
<property> < name> dfs.hosts.exclude </name> <value> /usr/local/hadoop/etc/hadoop/exclude < /value ></property>
● edit delete list file
[root@localhost ~]# cat /usr/local/hadoop/etc/hadoop/exclude newnode
You can migrate data after changing the configuration file on namenode. There is no need to synchronize the configuration file
/usr/local/hadoop/bin/hdfs dfsadmin -refreshNodes
● view status
/usr/local/hadoop/bin/hdfs dfsadmin -report
2) Upload data example
Check the data first
[root@hadoop1 ~]# /usr/local/hadoop/bin/hdfs dfsadmin -report Configured Capacity: 168561606656 (156.99 GB) Present Capacity: 148584251392 (138.38 GB) DFS Remaining: 148583858176 (138.38 GB) DFS Used: 393216 (384 KB) DFS Used%: 0.00% Under replicated blocks: 0 Blocks with corrupt replicas: 0 Missing blocks: 0 Missing blocks (with replication factor 1): 0 ------------------------------------------------- Live datanodes (4): Name: 192.168.1.51:50010 (node-0001) ...... DFS Used: 159744 (156 KB) ...... Name: 192.168.1.52:50010 (node-0002) ...... DFS Used: 131072 (128 KB) ...... Name: 192.168.1.53:50010 (node-0003) ...... DFS Used: 65536 (64 KB) ...... Name: 192.168.1.54:50010 (newnode) ...... DFS Used: 36864 (36 KB) ......
Upload data
[root@hadoop1 ~]# ls hadoop-2.7.7.tar.gz mydata [root@hadoop1 ~]# du -h mydata 1.3G mydata [root@hadoop1 ~]# [root@hadoop1 ~]# /usr/local/hadoop/bin/hadoop fs -put mydata / [root@hadoop1 ~]# rm -rf hadoop-2.7.7.tar.gz mydata
View data
[root@hadoop1 ~]# /usr/local/hadoop/bin/hdfs dfsadmin -report Configured Capacity: 168561606656 (156.99 GB) Present Capacity: 148584062680 (138.38 GB) DFS Remaining: 145938518016 (135.92 GB) DFS Used: 2645544664 (2.46 GB) DFS Used%: 1.78% Under replicated blocks: 0 Blocks with corrupt replicas: 0 Missing blocks: 0 Missing blocks (with replication factor 1): 0 ------------------------------------------------- Live datanodes (4): Name: 192.168.1.51:50010 (node-0001) ...... DFS Used: 405958677 (387.15 MB) ...... Name: 192.168.1.52:50010 (node-0002) ...... DFS Used: 946995249 (903.13 MB) ...... Name: 192.168.1.53:50010 (node-0003) ...... DFS Used: 511043394 (487.37 MB) ...... Name: 192.168.1.54:50010 (newnode) ...... DFS Used: 781547344 (745.34 MB) ......
Configure data migration hdfs-site.xml (done on Hadoop 1, no synchronization required)
[root@hadoop1 ~]# vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml <property> <name>dfs.hosts.exclude</name> <value>/usr/local/hadoop/etc/hadoop/exclude</value> </property>
Configure the list of excluded hosts and migrate data (executed on Hadoop 1)
# Add newnode to delete profile [root@hadoop1 ~]# echo newnode >/usr/local/hadoop/etc/hadoop/exclude # Migrate data [root@hadoop1 ~]# /usr/local/hadoop/bin/hdfs dfsadmin -refreshNodes # View the status. You can go offline only when the node status is decentralized [root@hadoop1 ~]# /usr/local/hadoop/bin/hdfs dfsadmin -report
Offline node (newnode execution)
[root@newnode ~]# /usr/local/hadoop/sbin/hadoop-daemon.sh stop datanode [root@newnode ~]# /usr/local/hadoop/sbin/yarn-daemon.sh stop nodemanager
3, NFS gateway
1. Overview of NFS gateway
● purpose of NFS gateway
- Users can browse the HDFS file system through the local NFSv3 client compatible with the operating system
- Users can stream data directly through the mount point
- Allows HDFS to be mounted as part of the client file system
- File attachment is supported, but random write (nolock) is not supported
- At present, NFS gateway only supports NFSv3 and TCP protocol (vers=3, proto=tcp)
● Proxy user
- The proxy user is an authorized user for NFS gateway to access the cluster
Add proxy users on NameNode and NFSGW
The UID, GID and user name of the proxy user must be identical
- If the user UID, GID and user name cannot be maintained for special reasons, we need to configure the static mapping relationship of nfs. map
-For example:
uid 10 100 # Map the remote UID 10 to the local UID 100
gid 11 101 # Map the remote GID 11 to the local GID 101
2.NFS gateway architecture diagram
3. Purchase virtual machine and prepare
host IP address to configure
nfsgw 192.168.1.55 Minimum configuration: 1 core 1G
HDFS user authorization
Both Hadoop 1 and nfsgw need to add users
[root@hadoop1 ~]# groupadd -g 800 nfsuser [root@hadoop1 ~]# useradd -g 800 -u 800 -r -d /var/hadoop nfsuser #---------------------------------------------------------------------------------------- [root@nfsgw ~]# groupadd -g 800 nfsuser [root@nfsgw ~]# useradd -g 800 -u 800 -r -d /var/hadoop nfsuser
4.HDFS cluster authorization
[root@hadoop1 ~]# vim /usr/local/hadoop/etc/hadoop/core-site.xml <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://hadoop1:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/var/hadoop</value> </property> <property> <name>hadoop.proxyuser.nfsuser.groups</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.nfsuser.hosts</name> <value>*</value> </property> </configuration> [root@hadoop1 ~]# /usr/local/hadoop/sbin/stop-all.sh [root@hadoop1 ~]# for i in node-{0001..0003};do rsync -avXSH /usr/local/hadoop/etc ${i}:/usr/local/hadoop/ done [root@hadoop1 ~]# /usr/local/hadoop/sbin/start-dfs.sh [root@hadoop1 ~]# jps 5925 NameNode 6122 SecondaryNameNode 6237 Jps [root@hadoop1 ~]# /usr/local/hadoop/bin/hdfs dfsadmin -report ... ... ------------------------------------------------- Live datanodes (3):
5. Build nfsgw gateway service
1)nfsgw gateway service
- por tmap service (conflicts with system rpcb ind) - nfs3 service (conflicts with System NFS)
- Uninstall rpcb ind and NFS utilsyum remove y rpcbind NFS utils of NFSGW
Building Gateway Services (continued 1)
● NFS gateway has dual roles. It is both an HDFS client program and an NFS server. Therefore, NFS gateway must be able to access all nodes in the cluster and configure / etc/hosts
[root@nfsgw ~]# yum remove -y rpcbind nfs-utils [root@nfsgw ~]# vim /etc/hosts 192.168.1.50 hadoop1 192.168.1.51 node-0001 192.168.1.52 node-0002 192.168.1.53 node-0003 192.168.1.55 nfsgw [root@nfsgw ~]# yum install -y java-1.8.0-openjdk-devel rsync -aXSH --delete hadoop1:/usr/local/hadoop /usr/local/
2) Build gateway service
● nfs. exports authorization
I. nfs. exports. allowed. hosts
- By default, export can be mounted by any client. To better control access, you can set properties. The value and string correspond to the machine name and access policy, which are separated by spaces. The format of the machine name can be a single host, a Java regular expression, or an IPv4 address
- Use rw or ro to specify read-write or read-only permissions for the export directory.
- The default setting is read-only
[root@nfsgw ~]# vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml <configuration> <property> <name>dfs.namenode.http-address</name> <value>hadoop1:50070</value> </property> <property> <name>dfs.namenode.secondary.http-address</name> <value>hadoop1:50090</value> </property> <property> <name>dfs.replication</name> <value>2</value> </property> <property> <name>dfs.hosts.exclude</name> <value>/usr/local/hadoop/etc/hadoop/exclude</value> </property> <property>#Default shared root <name>nfs.exports.allowed.hosts</name> #Similar to / etc/exports <value>* rw</value> #Similar to / dir * (RW) read / write </property> <property> <name>nfs.dump.dir</name> <value>/var/nfstmp</value>#Temporary documents </property> </configuration>
3) Build gateway service
● nfs. dump temporary directory configuration
- nfs. dump. dir
- The user needs to update the file dump directory parameters. NFS clients often rearrange write operations, and sequential write operations will randomly arrive at NFS gateway. This directory is often used to temporarily store unordered writes. For each file, unordered writes are dumped when they accumulate in memory beyond a certain threshold (e.g., 1M). To ensure that there is enough space for the directory, use the memory cache first, and use the hard disk when the memory is insufficient
- NFS gateway needs to be restarted after setting this attribute
[root@nfsgw ~]# mkdir /var/nfstmp #Create staging directory [root@nfsgw ~]# chown nfsuser.nfsuser /var/nfstmp #Give permission to write logs [root@nfsgw ~]# rm -rf /usr/local/hadoop/logs/* [root@nfsgw ~]# setfacl -m user:nfsuser:rwx /usr/local/hadoop/logs [root@nfsgw ~]# getfacl /usr/local/hadoop/logs [root@nfsgw ~]# cd /usr/local/hadoop/ #Start service [root@nfsgw hadoop]# ./sbin/hadoop-daemon.sh --script ./bin/hdfs start portmap [root@nfsgw hadoop]# jps 1376 Portmap 1416 Jps #Delete temporary file [root@nfsgw hadoop]# rm -rf /tmp/.hdfs-nfs [root@nfsgw hadoop]# sudo -u nfsuser ./sbin/hadoop-daemon.sh --script ./bin/hdfs start nfs3 [root@nfsgw hadoop]# sudo -u nfsuser jps 1452 Nfs3 1502 Jps
6.mount verification
Mount NFS gateway
● at present, NFS can only use v3 version
- vers=3
● only TCP is used as the transmission protocol
- proto=tcp
● random writing NLM is not supported
-nolock
● disable time update of access t ime
-noatime
● disable acl extension permissions
- noac I
● synchronous write to avoid reordering write
-sync
[root@newnode ~]# yum install -y nfs-utils [root@newnode ~]# showmount -e 192.168.1.55 Export list for 192.168.1.55: / * [root@newnode ~]# mount -t nfs -o vers=3,proto=tcp,nolock,noacl,noatime,sync 192.168.1.55:/ /mnt/ [root@newnode ~]# df -h Filesystem Size Used Avail Use% Mounted on 192.168.1.55:/ 118G 15G 104G 13% /mnt
seven Error reporting processing
Error 1. When the file is lost and there is no backup, you can only use the format
[root@hadoop1 ~]# /usr/local/hadoop/sbin/stop-all.sh [root@hadoop1 ~]# for i in hadoop1 node-{0001..0003};do ssh ${i} 'rm -rf /var/hadoop/*' done [root@hadoop1 ~]# /usr/local/hadoop/bin/hdfs namenode -format [root@hadoop1 ~]# /usr/local/hadoop/sbin/start-all.sh
Common errors
● the root user is required to start portmap
● to start nfs3, you need to use the proxy user set in the core site ● you must authorize the proxy user
- / If var/nfstmp is not authorized to upload files, an error will occur
- / usr/ loca 1/hadoop/ logs is not authorized and cannot see the error log ● you must start portmap before starting nfs3
● if portmap is restarted, nfs3 must also be restarted after the restart