1 data source: the last product about simple statistics of mobile traffic

2. Requirements: sort in reverse order according to the total flow value, and then get the output

3. General logic

(1) FlowSort class: for serialization and deserialization, the implementation of sorting logic interface

(2) FlowSortMapper class: encapsulating data

(3) FlowSortReducer class: exchange key and value, and then encapsulate and write them (only key (parameter) can be sorted, not value)

(4) FlowSortDriver class; for driver operation (with specific comments below the code), k and value will be swapped back when the output type is set

4. Code

(1)FlowSort

package flowsort;

import org.apache.hadoop.io.WritableComparable;

import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;

public class FlowSort implements WritableComparable<FlowSort> {
    //Up flow flow
    private long upFlow;
    //Lower flow flow
    private long downFlow;
    //sum flow
    private long sumFlow;
    //Space parameter structure
    public FlowSort() {
    }

    @Override
    public String toString() {
        return upFlow +"\t"
                 +downFlow +"\t"+
                sumFlow
                ;
    }

    //Serialization method
    public void write(DataOutput out) throws IOException {

       out.writeLong(upFlow);
        out.writeLong(downFlow);
        out.writeLong(sumFlow);
    }
    //Deserialization method
    public void readFields(DataInput in) throws IOException {
         this.upFlow=in.readLong();
         this.downFlow=in.readLong();
         this.sumFlow=in.readLong();
    }

    //Sorting logic
    public int compareTo(FlowSort o) {
        int result=0;
        if (sumFlow>o.getSumFlow()) {
            result = -1;
        }else if (sumFlow<o.getSumFlow()){
            result=1;

        }else{
            result=0;        }
        return result;
    }

    public long getUpFlow() {
        return upFlow;
    }

    public void setUpFlow(long upFlow) {
        this.upFlow = upFlow;
    }

    public long getDownFlow() {
        return downFlow;
    }

    public void setDownFlow(long downFlow) {
        this.downFlow = downFlow;
    }

    public long getSumFlow() {
        return sumFlow;
    }

    public void setSumFlow(long sumFlow) {
        this.sumFlow = sumFlow;
    }
}

(2)FlowSortMapper

package flowsort;

import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

public class FlowSortMapper extends Mapper<LongWritable, Text,FlowSort,Text> {

    Text k = new Text();
    FlowSort v = new FlowSort();
    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {

        //Get a row of data
        String line = value.toString();
        //Cut string
        String[] filds = line.split("\t");
        //Encapsulate objects
        //Encapsulate key
        String phonenumber = filds[0];
        //Get upstream traffic
        long upflow=Long.parseLong(filds[1]);
        //Get downstream traffic
        long downflow=Long.parseLong(filds[2]);
        //Get total traffic
        long sumflow=Long.parseLong(filds[3]);



        k.set(phonenumber);
        v.setUpFlow(upflow);
        v.setDownFlow(downflow);
        v.setSumFlow(sumflow);

        //Write
        context.write(v,k);

    }
}

(3)FlowSortReducer

package flowsort;


import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

import java.io.IOException;

public class FlowSortReducer extends Reducer<FlowSort, Text,Text,FlowSort> {
    @Override
    protected void reduce(FlowSort key, Iterable<Text> values, Context context) throws IOException, InterruptedException {


        for (Text phoneNumber : values) {
            context.write(phoneNumber,key);

        }
    }
}

(4)FlowSortdriver

package flowsort;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;


import java.io.IOException;

public class FlowSortDrivers {
    public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
        //0 encapsulates the I / O path
        args =new String[]{"C:/Users/input","C:/Users/output"};
        System.setProperty("hadoop.home.dir","E:/hadoop-2.7.2/");
        //Get job object
        Configuration conf = new Configuration();
        Job job = Job.getInstance(conf);
        //Set jar load path
        job.setJarByClass(FlowSortDrivers.class);
        //Associate mapper and reducer
        job.setMapperClass(FlowSortMapper.class);
        job.setReducerClass(FlowSortReducer.class);
        //Set map final output type
        job.setMapOutputKeyClass(FlowSort.class);
        job.setMapOutputValueClass(Text.class);
        //Set final output type
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(FlowSort.class);
        //Set input road force
        FileInputFormat.setInputPaths(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));

        //Submit job
        boolean result = job.waitForCompletion(true);
        System.exit(result ? 0 : 1);

    }
}

5. Results

6. My error: there is no problem with the path, but the job object is not completed, and the ouput has no content

(1) Misguided bag, it's really fatal

(2) Wrong function name.....

Oh, I hope I don't make these low-level mistakes in the future

2020-05-26

21:30:13

FlowSort