1, Experimental purpose
- Understand the role of HDFS in Hadoop architecture;
- Proficient in using HDFS to operate common Shell commands;
- Familiar with Java API s commonly used in HDFS operation.
2, Experimental platform
- Operating system: Linux (CentOS recommended);
- Hadoop version: 2.6.1;
- JDK version: 1.7 or above;
- Java IDE: Eclipse.
3, Experimental steps
(1) The following functions are realized by programming, and the same tasks are completed by using the Shell command provided by Hadoop:
Outputting the contents of the specified file in HDFS to the terminal;
#!/bin/bash hadoop fs -ls / read -p "please select file you want to output: " filename if hadoop fs -test -e /$filename then hadoop fs -cat /$filename else echo "the file not exist, output failed" fi
Java implementation: |
Display the read-write permission, size, creation time, path and other information of the file specified in HDFS;
read -p "please select file you want to check: " filename if hadoop fs -test -e /$filename then hadoop fs -ls -h /$filename else echo "the file not exist, output failed" fi
Java implementation: |
Given a directory in HDFS, output the read-write permission, size, creation time, path and other information of all files in the directory. If the file is a directory, recursively output the relevant information of all files in the directory;
read -p "enter a path: " path hadoop fs -ls -R $path
Java implementation: |
Provide a path to a file in HDFS to create and delete the file. If the directory where the file is located does not exist, the directory will be created automatically;
#!/bin/bash read -p "enter a path: " path if $(hadoop fs -test -d $path); then read -p "make choie (delete or append)" choice if [ $choice == "delete" ] then read -p "enter filename: " filename hadoop fs -rm -r {$path}/{$filename} else read -p "enter filename: " filename hadoop fs -touchz $path/$filename fi else echo "path does not exist, created" hadoop fs -mkdir -p $path read -p "make choie (delete or append)" choice if [ $choice == "delete" ] then read -p "enter filename: " filename hadoop fs -rm -r {$path}/{$filename} else read -p "enter filename: " filename hadoop fs -touchz $path/$filename fi fi
Java implementation: |
Provide a path to the directory of HDFS, and create and delete the directory. When creating a directory, if the directory where the directory file is located does not exist, the corresponding directory will be automatically created; When deleting a directory, the user specifies whether to delete the directory when the directory is not empty;
#!/bin/bash read -p "enter a path: " path if $(hadoop fs -test -d $path); then read -p "make choie (delete or add):" choice if [ $choice == "delete" ] then read -p "enter deletefilename: " filename isEmpty=$(hadoop fs -count $path/$filename | awk '{print $2}') if [[ $isEmpty -eq 0 ]] then hadoop fs -rm -r $path/$filename else read -p "Not an empty directory,continue(yes or no): " choice2 if [ choice2 == "yes" ] then hadoop fs -rm -r $path/$filename fi fi else read -p "enter creatfilename: " filename hadoop fs -touchz $path/$filename fi else echo "path does not exist, created" hadoop fs -mkdir -p $path read -p "make choie (delete or append)" choice if [ $choice == "delete" ] then read -p "enter filename: " filename hadoop fs -rm -r {$path}/{$filename} else read -p "enter filename: " filename hadoop fs -touchz $path/$filename fi fi
Java implementation: |
Add content to the file specified in HDFS, and the content specified by the user is added to the beginning or end of the original file;
#!/bin/bash read -p "Enter the file path where you want to add content: " path if hadoop fs -test -f $path then read -p "make choie (head or end)" choice if [ $choice == "head" ] then read -p "enter content: " filename hadoop fs -copyFromLocal -f $filename $path else read -p "enter content: " filename hadoop fs -appendToFile $filename $path fi else echo "file does not exist, created" hadoop fs -mkdir -p $path/../ hadoop fs -touchz $path read -p "make choie (head or end)" choice if [ $choice == "head" ] then read -p "enter content: " filename hadoop fs -copyFromLocal -f $filename $path else read -p "enter content: " filename hadoop fs -appendToFile $filename $path fi fi
Java implementation: |
Delete the file specified in HDFS;
read -p "enter a path you want to delete: " path hadoop fs -rm -R $path
Java implementation: |
In HDFS, move files from the source path to the destination path.
read -p "enter sourcepath: " sourcepath read -p "enter target path: " targetpath hadoop fs -mv $sourcepath $targetpath
Java implementation: |
Optional:
(2) Program and implement a class "MyFSDataInputStream", which inherits "org.apache.hadoop.fs.FSDataInputStream". The requirements are as follows: implement the method "readLine()" to read the specified file in HDFS by line. If it reads the end of the file, it will return null, otherwise it will return the text of one line of the file.
(3) Check the Java help manual or other materials, complete the programming with "java.net.URL" and "org.apache.hadoop.fs.FsURLStreamHandlerFactory", and output the text of the specified file in HDFS to the terminal.
Java implementation: |
- Experimental summary and problems
1. Learn to use what to do;
2. What problems did you encounter during the experiment? How to solve it?
3. What problems remain unresolved? What may have caused it.