Imitating Baidu net disk to realize file slice upload and breakpoint continuation on web side

Unable to restrain the emotions of the early morning, the humid voice of a rough man's shy rose opened quietly. Is it my aesthetic view that tiktok has seen many times in harmony?

Many of the functions I wrote are independent in my daily development, and I will consolidate and comb the overall implementation process (HMM Hello, I'm Hello, everyone, it's really good). This side mainly refers to the upload function of Baidu online disk web end to realize the core file slice upload and breakpoint continue transfer function. In order not to delay the time of the unnecessary guests, I will first post a practical effect picture. If there is a picture with truth, I will continue to see if I can see it. If I can't see it, please continue to try ~ ~!

 

It's still the old rule. Before developing functions, we need to understand the process and principle of implementation, so that we can achieve twice the result with half the effort. Let's enter the process as follows:

1. Competitive product analysis

We open the web end of Baidu online disk, upload a file first, let's look at the request mode in the debugging tool and the operation process when uploading

A professional file management system like Baidu online disk is sure to upload slices when uploading large files. It can be seen from the above that it calls 5 times at the same time to upload slices one by one. It's a bit like a thread, but it doesn't support multithreading in the browser. In addition to websocket, it's a pseudo multithread. At the same time, it turns on 5 tasks in the for loop and executes them Put in a new one after one.

The xhr object (XMLHttpRequest) is used to track the upload progress. In fact, the process is very simple without too much complex logic. The problems encountered are as follows:

(1) How is file slicing implemented?

The direct Math function at the front cuts the file.

(2) How does this pause and continue work?

Here by observing Baidu online disk, in fact, its pause is not a real pause, but a fake through the style, but the actual upload process is still in progress.

(3) How is breakpoint continuation implemented?

When the file is uploaded in segments, record the segment id of the successfully uploaded file in the server, and use the file information encrypted by MD5 as the index. If the same file is detected, it will return to the last segment point and then download.

2. Development and function realization

Make clear the above problems, and the development and implementation will be much easier. Let's talk about the code, first look at the front-end core code:

//Record the current file upload status
        var init={};
        (function(){
            //Upload picture
            $("#file_1").on("change", function() {
                var file = $(this).get(0).files[0];  //File object
                var fname = file.name;        //file name
                var totalsize = file.size;      //Total size
                //Conversion M and Kb
                if (totalsize > 1024 * 1024){
                    var tosize = (Math.round(totalsize * 100 / (1024 * 1024)) / 100).toString() + 'M';
                }
                else{
                    var tosize = (Math.round(totalsize * 100 / 1024) / 100).toString() + 'KB';
                }
                //Generate random string
                var rndstr=randomString(16);
                //Add the current upload global management status
                init[rndstr]=false;
                //Upload file html
                var html='<li class="info-con-li info-li-'+rndstr+'">\n' +
                    '                <div class="process" style="width: 0%;"></div>\n' +
                    '                <div class="info">\n' +
                    '                    <div class="file-name" title="'+fname+'">\n' +
                    '                        <div class="file-icon fileicon-small-zip"></div>\n' +
                    '                        <span class="name-text">'+fname+'</span>\n' +
                    '                    </div>\n' +
                    '                    <div class="file-size">'+tosize+'</div>\n' +
                    '                    <div class="file-status">\n' +
                    '                        <span class="waiting">Queuing</span>\n' +
                    '                        <span class="prepare">Ready to upload</span>\n' +
                    '                        <span class="uploading" data-pernum="0" data-upsize="0" data-pretime="0" style="display: block">\n' +
                    '                            <em class="precent">0.00%</em>\n' +
                    '                            <em class="speed">(0kb/s)</em>\n' +
                    '                        </span>\n' +
                    '                        <span class="error"><em></em><i>Server error</i><b></b></span>\n' +
                    '                        <span class="pause"><em></em><i>Paused</i></span>\n' +
                    '                        <span class="cancel"><em></em><i>Cancelled</i></span>\n' +
                    '                        <span class="success"><em></em><i></i></span>\n' +
                    '                    </div>\n' +
                    '                    <div class="file-operate">\n' +
                    '                        <em class="operate-pause"></em>\n' +
                    '                        <em class="operate-continue"></em>\n' +
                    '                        <em class="operate-retry"></em>\n' +
                    '                        <em class="operate-remove"></em>\n' +
                    '                    </div>\n' +
                    '                </div>\n' +
                    '            </li>';
                $(".info-con-ul").append(html);
                //Pause upload
                $(".info-li-"+rndstr).find(".operate-pause").unbind('click').click(function(){
                    console.log("Pause upload");
                    //xhr.abort();
                    $(this).hide();
                    $(this).parent().parent().find(".uploading").hide();
                    $(this).parent().parent().find(".pause").show();
                    $(this).parent().find(".operate-continue").show();
                });
                //Continue to upload
                $(".info-li-"+rndstr).find(".operate-continue").unbind('click').click(function(){
                    console.log("Continue to upload")
                    $(this).hide();
                    $(this).parent().parent().find(".uploading").show();
                    $(this).parent().parent().find(".pause").hide();
                    $(this).parent().find(".operate-pause").show();
                    console.log(init[rndstr])
                    if (!init[rndstr]){
                        $.ajax({
                            url: '/getrecord',
                            type: "POST",
                            data: {
                                "rndstr":rndstr
                            },
                            success:function(data){
                                console.log(data)
                                if (data.code==200){
                                    fileUpload(file,data.zoneid,rndstr);
                                }else{
                                    console.log(data.msg)
                                }

                            }
                        });
                    }
                });
                var i=0;
                var xhr='';
                fileUpload(file,i,rndstr,xhr);
            });

            //Upload file method
            function fileUpload(file,i,rndstr,xhr){
                //Modify the current upload status
                init[rndstr]=true;
                $(".info-li-"+rndstr).find(".uploading").data("upsize",0);
                var fname = file.name;        //file name
                var totalsize = file.size;      //Total size
                var filesize = 0.5 * 1024 * 1024;    //Take 500k as a slice
                var filenum = Math.ceil(totalsize / filesize);  //Total number of pieces
                //for(var i = 0;i < filenum;++i) {
                    //Calculate the start and end positions of each piece
                    var start = i * filesize;
                    var end = Math.min(totalsize, start + filesize);
                    var curfile=file.slice(start,end);
                    var data = new FormData();
                    data.append('file', curfile);
                    data.append('filename', fname);
                    data.append('filenum', filenum);
                    data.append('rndstr', rndstr);
                    data.append('zoneid', i);
                    $.ajax({
                        url: '/upload',
                        type: "POST",
                        processData: false,
                        contentType: false,
                        data: data,
                        success:function(){
                            init[rndstr]=false;
                            //Dealing with xhr objects with CSS
                            if ($(".info-li-"+rndstr).find(".operate-continue").css('display') == 'block') {
                                xhr.abort();
                            }else{
                                i += 1;
                                if (i < filenum) {
                                    fileUpload(file, i, rndstr,xhr);
                                }else{
                                    $(".info-li-"+rndstr).find(".file-status span").hide();
                                    $(".info-li-"+rndstr).find(".file-operate em").hide();
                                    $(".info-li-"+rndstr).find(".success").show();
                                }
                            }
                        },
                        xhr: function(){
                            xhr = $.ajaxSettings.xhr();
                            if(uploadProgress && xhr.upload) {
                                xhr.upload.addEventListener("progress" , function(){uploadProgress(event,".info-li-"+rndstr,totalsize)}, false);
                                return xhr;
                            }
                        }
                    })
                //}
            }
            //Upload progress tracking
            function uploadProgress(evt,classname,totalsize){
                //Get the size and time of the last uploaded file
                var preupsize=parseFloat($(classname).find(".uploading").data("upsize"));
                var pretime=parseFloat($(classname).find(".uploading").data("pretime"));
                var loaded = evt.loaded;     //Uploaded size
                var per = (loaded-preupsize)/totalsize;  //Percentage Uploaded
                var nowtime = new Date().getTime();      //Get current time
                //Calculate the upload speed based on the current and last upload size and time
                var speed =(loaded-preupsize)/1024/((nowtime-pretime)/1000);
                var speedStr = (speed >= 1024) ? (speed/1024) + "":speed + "";
                speedStr=toDecimal2(speedStr);
                speedStr= speedStr+ (speed>1024?" Mb/s" : " Kb/s");
                //assignment
                $(classname).find(".speed").html('('+speedStr+')');
                $(classname).find(".uploading").data("upsize",loaded);
                $(classname).find(".uploading").data("pretime",nowtime);
                //Current total file upload size
                var cursize=parseFloat($(classname).find(".uploading").data("pernum"));
                cursize+=per;
                cursize=cursize>1?1:cursize;
                var curper=toDecimal2(cursize*100)+"%";
                //Progress bar does not execute when paused, but the process continues
                if ($(classname).find(".operate-continue").css('display') == 'none') {
                    $(classname).find(".process").css('width', curper);
                }
                $(classname).find(".uploading").data("pernum",cursize);
                $(classname).find(".precent").html(curper);
             }

             //Two decimal places shall be reserved for the system. For example: 2, 00 will be added after 2, i.e. 2.00
             function toDecimal2(x) {
                var f = parseFloat(x);
                if (isNaN(f)) {
                return false;
                }
                var f = Math.round(x*100)/100;
                var s = f.toString();
                var rs = s.indexOf('.');
                if (rs < 0) {
                rs = s.length;
                s += '.';
                }
                while (s.length <= rs + 2) {
                s += '0';
                }
                return s;
             }
             //Generate random string
            function randomString(len) {
              len = len || 32;
              var $chars = 'ABCDEFGHJKMNPQRSTWXYZabcdefhijkmnprstwxyz2345678';
              var maxPos = $chars.length;
              var pwd = '';
              for (var i = 0; i < len; i++) {
                pwd += $chars.charAt(Math.floor(Math.random() * maxPos));
              }
              return pwd;
            }
        })()

Here, I simply use jquery to create an anonymous function method for execution. There are notes on it that can be understood basically. If you don't understand, you can leave a message to me. For some reasons, I combine the assignment of parameters to the page to save the data. Students who are interested can set some local variables in the global variable allocation. I directly use the code of the page on Baidu online disk Transfer file style. If you need complete code, it will be provided later.

I still use the Sanic framework of Python on the server side. There are many advantages of using python. It's very fast and convenient to build the framework No more nonsense. Look at the code:

#Upload file interface
@app.route('/upload',methods=["POST"])
async def upload(request):
    args = request.args if request.method == 'GET' else request.form
    filename = args.get('filename', '')    #file name
    filenum = args.get('filenum', 0)       #Total number of file segments
    rndstr = args.get('rndstr', '')        #Random string (unique identification)
    zoneid = args.get('zoneid', 0)         #Current fragment id
    file = request.files.get('file')       #File flow
    path = os.path.abspath(os.path.dirname(__file__))
    #Basic folder path
    basepath="{}\\files\\{}".format(path,rndstr)
    datapath="{}\\data".format(basepath)
    if not os.path.exists(datapath):
        os.makedirs(datapath)
    #Save file fragment
    with open("{}\\{}".format(datapath,zoneid), 'wb') as f:
         f.write(file.body)
    #Record the currently completed fragment id
    with open("{}\\record.txt".format(basepath), 'w',encoding="utf-8") as f:
         txt="{}|{}".format(zoneid,rndstr)
         f.write(txt)
    #Judge whether the file has been uploaded
    if int(filenum)==int(zoneid)+1:
        u=Upload()
        u.mergefiles(rndstr,filename,int(filenum))
    msg={"code":200,"msg":"Start upload"}
    return json(msg)

#Get the progress of file upload and save
@app.route('/getrecord',methods=["POST"])
async def getrecord(request):
    args = request.args if request.method == 'GET' else request.form
    rndstr = args.get('rndstr', '')
    if rndstr!="":
        path = os.path.abspath(os.path.dirname(__file__))
        basepath = "{}\\files\\{}".format(path, rndstr)
        if os.path.exists("{}\\record.txt".format(basepath)):
            with open("{}\\record.txt".format(basepath), 'r') as f:
                result=f.read()
                print(result)
            if result!="":
                zoneid=int(result.split("|")[0])+1
                msg = {"code": 200, "msg": "Request succeeded", "zoneid": zoneid}
        else:
            msg = {"code": 203, "msg": "Creating file"}
    else:
        msg = {"code": 204, "msg": "Request exception"}
    return json(msg)

Here we write casually. Some of them are not very standard. In the later stage, we can optimize the code format by ourselves. There is also an upload class:

#coding:utf-8
import os

class Upload():
    def __init__(self):
        pass

    #Merge files
    def mergefiles(self,rndstr,filename,filenum):
        path = os.path.abspath(os.path.dirname(os.path.dirname(__file__)))
        with open('{}\\files\\{}\\{}'.format(path,rndstr,filename), 'wb') as file:  # create a new file
            for i in range(filenum):
                try:
                    filename = '{}\\files\\{}\\data\\{}'.format(path,rndstr,i)
                    source_file = open(filename, 'rb')  # Open each tile in order
                    file.write(source_file.read())  # Read fragment content and write new file
                    source_file.close()
                except IOError:
                    break
                #Delete merged tiles
                #os.remove(filename)

if __name__=='__main__':
    u=Upload()
    u.mergefiles("e4aMQGZZkYi7iJem","Icon file.zip",2)

The above code basically realizes the function of uploading files on Baidu online disk. Due to the time relationship, I just basically realize the logic function here. For example, I only open a task cycle to call and so on. In terms of details, if we want to achieve real practicality, there are still some places that need to be optimized by ourselves. Let's look at the uploaded files:

3. Optimization and improvement

In fact, the above part only realizes the basic functions. In the process of uploading, it also involves many problems, such as the verification of interface users, the realization of multi task upload, MD5 encryption of file information, the complete implementation of breakpoint renewal, etc. in fact, the above method can realize the breakpoint renewal function through getrecord, but I am not the file when I transfer the unique value of the file MD5 encryption information is a randomly generated string. If the page is refreshed, it will be downloaded again. Therefore, the name, size, last modification time and type progress of the file are generally used to determine the uniqueness of MD5 encryption. This is left to the guests to operate by themselves. There is also the realization of second transmission of Baidu online disk. Here I will mention that the basic idea is basically the root According to the MD5 information of the file to match the database, if it matches the successful one that has been uploaded, copy it directly. The above only represents my personal point of view. What Marx said is good: practice is the truth.

Well, serious time always passes so fast, let's continue to code happily ~ ~!

I have worked hard for such a long time. What I have never changed my original intention is to accompany and assist my fellow procedural apes to recover from their lovelorn, abandoned and forced life, turn grief into power, and move to the realm of Daniel as soon as possible, so as to realize the realm of postnatal wild flowers.

If the white prostitute brothers need complete code, please pay attention to the reply of the official account: file upload.

Pay attention to the official account and surpass the ordinary to achieve self.

Tags: Fragment Python JSON JQuery

Posted on Tue, 16 Jun 2020 23:48:01 -0400 by Kyori