Implementation of multi thread file download based on rxjava 2.0 + retrofit2.0

preface

I wrote an article before Implementation of file download based on RxJava2.0+Retrofit2.0 (with progress, non overwriting ResponseBody and interceptor) , it is a file downloaded by one-way single task, not a breakpoint download. Breakpoint download can be single threaded or multi-threaded. However, multi-threaded download requires files to support breakpoint continuation, and the request header parameter Range is required. Whether single thread breakpoint download or multi-threaded breakpoint download, the principle is the same. The file needs to be divided into several parts. Each part adopts one thread to download. The difference is that one thread processes the file in the whole process of single thread breakpoint. Multi thread breakpoint download plans the number of download threads according to the number of files, that is, several thread processing files. This article describes how multiple threads download files and send back the download progress. For the time being, it does not support saving the download progress, that is, it does not support transmitting data where the file was interrupted last time. You can use the database to save the download progress yourself.

International rules, first rendering:

design sketch

Multithreaded file download process

  1. First obtain the length of the downloaded file, ContentLength (fileSize), and then generate a temporary file equal to the length of the downloaded file
  2. Set the number of file download threads threadNum, and calculate the interval (startPosition~ endPosition) for each thread to download files, that is, the size that each thread needs to download
 int blockSize = ((fileSize % threadNum) == 0) ? (fileSize / threadNum) : (fileSize / threadNum + 1);
  1. Create threadNum threads to download data in different locations of the file, write data in the corresponding location of the local temporary file, and record the current download size of each thread
  2. Summarize the download size of each thread, merge the download size of all threads, calculate the download percentage, download rate and download time, and finally echo the download progress information

Download implementation

1. Download interface DownloadApi:

 public interface DownloadApi {


    /**
     * Download File
     *
     */
    @Streaming
    @GET
    Observable<ResponseBody> downLoad(@Url String url);


    /**
     *Download File
     * @param range Range Represents the request header parameter for breakpoint continuation
     * @param url Download url
     * @return
     */
    @Streaming
    @GET
    Observable<ResponseBody> download(@Header("Range") String range, @Url String url);
}

2. Download method:

   /**
     * Download file method 3 (multi thread file download, update UI with RXJava)
     *
     * @param threadNum       Number of download threads
     * @param url
     * @param destDir
     * @param fileName
     * @param progressHandler
     */
    public static void downloadFile3(final int threadNum, final String url, final String destDir, final String fileName, final DownloadProgressHandler progressHandler) {
        DownloadApi apiService = RetrofitHelper.getInstance().getApiService(DownloadApi.class);
        final DownloadInfo downloadInfo = new DownloadInfo();
        Observable<ResponseBody> getFileSizeObservable = apiService.downLoad(url);
        final FileDownloadObservable[] fileDownloadObservables = new FileDownloadObservable[threadNum];//Set the number of threads
        //Get file size, split file download
        getFileSizeObservable
                .flatMap(new Function<ResponseBody, ObservableSource<DownloadInfo>>() {

                    @Override
                    public ObservableSource<DownloadInfo> apply(final ResponseBody responseBody) throws Exception {

                        return Observable.create(new ObservableOnSubscribe<DownloadInfo>() {
                            @Override
                            public void subscribe(ObservableEmitter<DownloadInfo> emitter) throws Exception {
                                long fileSize = responseBody.contentLength();


                                File dir = new File(destDir);
                                if (!dir.exists()) {
                                    dir.mkdirs();
                                }
                                final File file = new File(destDir, fileName);
                                if (file.exists()){
                                    file.delete();
                                }
                                downloadInfo.setFile(file);
                                downloadInfo.setFileSize(fileSize);
                                RandomAccessFile accessFile = new RandomAccessFile(file, "rwd");
                                //Set the length of the local file to be the same as that of the downloaded file
                                accessFile.setLength(fileSize);
                                accessFile.close();
                                //Download data per thread
                                long blockSize = ((fileSize % threadNum) == 0) ? (fileSize / threadNum) : (fileSize / threadNum + 1);
								//Specify the download interval for each thread
                                for (int i = 0; i < threadNum; i++) {
                                    int curThreadEndPosition = (int) ((i + 1) != threadNum ? ((i + 1) * blockSize - 1) : fileSize);
                                    FileDownloadObservable fileDownloadObservable = new FileDownloadObservable(url, file, (int) (i * blockSize), curThreadEndPosition);
                                    fileDownloadObservable.download();
                                    fileDownloadObservables[i] = fileDownloadObservable;
                                    mDisposable.add(fileDownloadObservable.getDisposable());

                                }
                                boolean finished = false;
                                long startTime = System.currentTimeMillis();
                                int downloadSize;
                                int progress;
                                long usedTime;
                                long curTime;
                                boolean completed;
                                int speed;
								//Calculate the download progress, download rate and download time according to the download size of all threads
                                while (!finished && !emitter.isDisposed()) {
                                    downloadSize = 0;
                                    finished = true;
                                    for (int i = 0; i < fileDownloadObservables.length; i++) {
                                        downloadSize += fileDownloadObservables[i].getDownloadSize();
                                        if (!fileDownloadObservables[i].isFinished()) {
                                            finished = false;
                                        }
                                    }
                                    progress = (int) ((downloadSize * 1.0 / fileSize) * 100);
                                    curTime = System.currentTimeMillis();
                                    usedTime = (curTime - startTime) / 1000;
                                    if (usedTime == 0) usedTime = 1;
                                    speed = (int) (downloadSize / usedTime);
                                    downloadInfo.setSpeed(speed);
                                    downloadInfo.setProgress(progress);
                                    downloadInfo.setCurrentSize(downloadSize);
                                    downloadInfo.setUsedTime(usedTime);
									//Echo download information
                                    if (!emitter.isDisposed()) {
                                        emitter.onNext(downloadInfo);
                                    }
                                    SystemClock.sleep(1000);
                                }
                                completed = true;
                                if (!emitter.isDisposed()) {
                                    if (completed) {
                                        emitter.onComplete();
                                    } else {
                                        emitter.onError(new RuntimeException("Download failed"));
                                    }
                                }

                            }
                        });
                    }
                })
                .subscribeOn(Schedulers.io())
                .observeOn(AndroidSchedulers.mainThread())
                .subscribe(new Observer<DownloadInfo>() {
                    @Override
                    public void onSubscribe(Disposable d) {
                        mDisposable.add(d);
                    }

                    @Override
                    public void onNext(DownloadInfo downloadInfo) {
                        progressHandler.onProgress(downloadInfo);
                    }

                    @Override
                    public void onError(Throwable e) {
                        progressHandler.onError(e);
                    }

                    @Override
                    public void onComplete() {
                        progressHandler.onCompleted(downloadInfo.getFile());
                    }
                });

    }

3. Download Observable, that is, the download task of each thread, FileDownloadObservable:

/**
 * TODO
 *
 * @author Kelly
 * @version 1.0.0
 * @filename FileDownloadObservable
 * @time 2021/9/6 17:37
 * @copyright(C) 2021 song
 */
public class FileDownloadObservable {

    /**
     * Download url
     */
    private String url;
    /**
     * Cached FIle
     */
    private File file;
    /**
     * Start position
     */
    private int startPosition;
    /**
     * End position
     */
    private int endPosition;
    /**
     * current location
     */
    private int curPosition;
    /**
     * complete
     */
    private boolean finished = false;
    /**
     * How many have been downloaded
     */
    private int downloadSize = 0;
    private Disposable disposable;
    private String name;

    public FileDownloadObservable(String url, File file, int startPosition,
                                  int endPosition) {
        this.url = url;
        this.file = file;
        this.startPosition = startPosition;
        this.curPosition = startPosition;
        this.endPosition = endPosition;
        this.name = "";
    }

    public void download() {
        DownloadApi apiService = RetrofitHelper.getInstance().getApiService(DownloadApi.class);
        String range = "bytes=" + (startPosition) + "-" + endPosition;

        apiService.download(range, url)
                .flatMap(new Function<ResponseBody, ObservableSource<Integer>>() {

                    @Override
                    public ObservableSource<Integer> apply(final ResponseBody responseBody) throws Exception {

                        return Observable.create(new ObservableOnSubscribe<Integer>() {
                            @Override
                            public void subscribe(ObservableEmitter<Integer> emitter) throws Exception {
                                InputStream inputStream = null;
                                long contentLength;

                                RandomAccessFile randomAccessFile = null;
                                byte[] buf = new byte[1024 * 8];
                                name = Thread.currentThread().getName();
                                try {
                                    inputStream = responseBody.byteStream();
                                    contentLength = responseBody.contentLength();
                                    System.out.println(name + ",startPosition " + startPosition + ",endPosition " + endPosition);
                                    randomAccessFile = new RandomAccessFile(file, "rwd");
                                    //Set start write location
                                    randomAccessFile.seek(startPosition);
                                    System.out.println(name + "Connection succeeded,Read length:" + FileUtils.formatFileSize(contentLength));
                                    while (curPosition < endPosition) {
                                        //The current location is less than the end location. Continue downloading
                                        int len = inputStream.read(buf);
                                        if (len == -1) {
                                            //Download complete
                                            System.out.println(len);
                                            break;
                                        }
                                        randomAccessFile.write(buf, 0, len);
                                        curPosition = curPosition + len;
                                        if (curPosition > endPosition) {    //If there are too many downloads, subtract the excess
                                            System.out.println(name + "curPosition > endPosition  !!!!");
                                            int extraLen = curPosition - endPosition;
                                            downloadSize += (len - extraLen + 1);
                                        } else {
                                            downloadSize += len;
                                        }
//                                        emitter.onNext(downloadSize);
                                    }
                                    finished = true;  //Download completed at current stage
                                    System.out.println("current" + name + "Download complete");

                                    if (!emitter.isDisposed()) {
                                        emitter.onComplete();
                                    }
                                } catch (Exception e) {
                                    if (!emitter.isDisposed()) {
                                        emitter.onError(e);
                                    }
                                } finally {
                                    //Close flow
                                    if (inputStream != null) {
                                        try {
                                            inputStream.close();
                                        } catch (IOException e) {
                                            e.printStackTrace();
                                        }
                                    }
                                    try {
                                        randomAccessFile.close();
                                    } catch (IOException e) {
                                        System.out.println("AccessFile IOException " + e.getMessage());
                                    }
                                }

                            }
                        });
                    }
                })
                .subscribeOn(Schedulers.newThread())//New thread Download
                .subscribe(new Observer<Integer>() {
                    @Override
                    public void onSubscribe(Disposable d) {
                        disposable = d;
                    }

                    @Override
                    public void onNext(Integer downloadSize) {

                    }

                    @Override
                    public void onError(Throwable e) {
                        System.out.println(name + "download error Exception " + e.getMessage());
                    }

                    @Override
                    public void onComplete() {

                    }
                });
    }


    /**
     * Do you want to finish downloading the current segment
     *
     * @return
     */
    public boolean isFinished() {
        return finished;
    }

    /**
     * How many have been downloaded
     *
     * @return
     */
    public int getDownloadSize() {
        return downloadSize;
    }

    public Disposable getDisposable() {
        return disposable;
    }
}

Download call

public void multiThreadDownloadTest(View view) {

    String url = "https://imtt.dd.qq.com/16891/apk/B168BCBBFBE744DA4404C62FD18FFF6F.apk?fsname=com.tencent.tmgp.sgame_1.61.1.6_61010601.apk&csr=1bbd";
    final NumberFormat numberFormat = NumberFormat.getInstance();
    // The setting is accurate to 2 decimal places
    numberFormat.setMaximumFractionDigits(2);
    FileDownloader.downloadFile3(threadNum, url, FileDownloadActivity.DOWNLOAD_APK_PATH, "multi_test.apk", new DownloadProgressHandler() {

        @Override
        public void onProgress(DownloadInfo downloadInfo) {

            long fileSize = downloadInfo.getFileSize();
            long speed = downloadInfo.getSpeed();
            long usedTime = downloadInfo.getUsedTime();
            long currentSize = downloadInfo.getCurrentSize();
            String percent = numberFormat.format((float) currentSize / (float) fileSize * 100);
            mProgress.setText(percent + "%");
            mFileSize.setText(FileUtils.formatFileSize(fileSize));
            mRate.setText(FileUtils.formatFileSize(speed) + "/s");
            mTime.setText(FileUtils.formatTime(usedTime));
        }

        @Override
        public void onCompleted(File file) {
            showMsg("Download complete:" + file.getAbsolutePath());
        }

        @Override
        public void onError(Throwable e) {
            showMsg("Download File exception:" + e.getMessage());
        }
    });
}

Summary

Theoretically, the larger the number of threads, the faster the download. However, too many threads will cause most of the CPU overhead to be spent on inter process switching, but it will be slower. In addition, the download speed is also related to the bandwidth. Therefore, the number of threads must be appropriate.

Posted on Fri, 10 Sep 2021 03:29:26 -0400 by cetaces