preface
I wrote an article before Implementation of file download based on RxJava2.0+Retrofit2.0 (with progress, non overwriting ResponseBody and interceptor) , it is a file downloaded by one-way single task, not a breakpoint download. Breakpoint download can be single threaded or multi-threaded. However, multi-threaded download requires files to support breakpoint continuation, and the request header parameter Range is required. Whether single thread breakpoint download or multi-threaded breakpoint download, the principle is the same. The file needs to be divided into several parts. Each part adopts one thread to download. The difference is that one thread processes the file in the whole process of single thread breakpoint. Multi thread breakpoint download plans the number of download threads according to the number of files, that is, several thread processing files. This article describes how multiple threads download files and send back the download progress. For the time being, it does not support saving the download progress, that is, it does not support transmitting data where the file was interrupted last time. You can use the database to save the download progress yourself.
International rules, first rendering:
design sketch
Multithreaded file download process
- First obtain the length of the downloaded file, ContentLength (fileSize), and then generate a temporary file equal to the length of the downloaded file
- Set the number of file download threads threadNum, and calculate the interval (startPosition~ endPosition) for each thread to download files, that is, the size that each thread needs to download
int blockSize = ((fileSize % threadNum) == 0) ? (fileSize / threadNum) : (fileSize / threadNum + 1);
- Create threadNum threads to download data in different locations of the file, write data in the corresponding location of the local temporary file, and record the current download size of each thread
- Summarize the download size of each thread, merge the download size of all threads, calculate the download percentage, download rate and download time, and finally echo the download progress information
Download implementation
1. Download interface DownloadApi:
public interface DownloadApi { /** * Download File * */ @Streaming @GET Observable<ResponseBody> downLoad(@Url String url); /** *Download File * @param range Range Represents the request header parameter for breakpoint continuation * @param url Download url * @return */ @Streaming @GET Observable<ResponseBody> download(@Header("Range") String range, @Url String url); }
2. Download method:
/** * Download file method 3 (multi thread file download, update UI with RXJava) * * @param threadNum Number of download threads * @param url * @param destDir * @param fileName * @param progressHandler */ public static void downloadFile3(final int threadNum, final String url, final String destDir, final String fileName, final DownloadProgressHandler progressHandler) { DownloadApi apiService = RetrofitHelper.getInstance().getApiService(DownloadApi.class); final DownloadInfo downloadInfo = new DownloadInfo(); Observable<ResponseBody> getFileSizeObservable = apiService.downLoad(url); final FileDownloadObservable[] fileDownloadObservables = new FileDownloadObservable[threadNum];//Set the number of threads //Get file size, split file download getFileSizeObservable .flatMap(new Function<ResponseBody, ObservableSource<DownloadInfo>>() { @Override public ObservableSource<DownloadInfo> apply(final ResponseBody responseBody) throws Exception { return Observable.create(new ObservableOnSubscribe<DownloadInfo>() { @Override public void subscribe(ObservableEmitter<DownloadInfo> emitter) throws Exception { long fileSize = responseBody.contentLength(); File dir = new File(destDir); if (!dir.exists()) { dir.mkdirs(); } final File file = new File(destDir, fileName); if (file.exists()){ file.delete(); } downloadInfo.setFile(file); downloadInfo.setFileSize(fileSize); RandomAccessFile accessFile = new RandomAccessFile(file, "rwd"); //Set the length of the local file to be the same as that of the downloaded file accessFile.setLength(fileSize); accessFile.close(); //Download data per thread long blockSize = ((fileSize % threadNum) == 0) ? (fileSize / threadNum) : (fileSize / threadNum + 1); //Specify the download interval for each thread for (int i = 0; i < threadNum; i++) { int curThreadEndPosition = (int) ((i + 1) != threadNum ? ((i + 1) * blockSize - 1) : fileSize); FileDownloadObservable fileDownloadObservable = new FileDownloadObservable(url, file, (int) (i * blockSize), curThreadEndPosition); fileDownloadObservable.download(); fileDownloadObservables[i] = fileDownloadObservable; mDisposable.add(fileDownloadObservable.getDisposable()); } boolean finished = false; long startTime = System.currentTimeMillis(); int downloadSize; int progress; long usedTime; long curTime; boolean completed; int speed; //Calculate the download progress, download rate and download time according to the download size of all threads while (!finished && !emitter.isDisposed()) { downloadSize = 0; finished = true; for (int i = 0; i < fileDownloadObservables.length; i++) { downloadSize += fileDownloadObservables[i].getDownloadSize(); if (!fileDownloadObservables[i].isFinished()) { finished = false; } } progress = (int) ((downloadSize * 1.0 / fileSize) * 100); curTime = System.currentTimeMillis(); usedTime = (curTime - startTime) / 1000; if (usedTime == 0) usedTime = 1; speed = (int) (downloadSize / usedTime); downloadInfo.setSpeed(speed); downloadInfo.setProgress(progress); downloadInfo.setCurrentSize(downloadSize); downloadInfo.setUsedTime(usedTime); //Echo download information if (!emitter.isDisposed()) { emitter.onNext(downloadInfo); } SystemClock.sleep(1000); } completed = true; if (!emitter.isDisposed()) { if (completed) { emitter.onComplete(); } else { emitter.onError(new RuntimeException("Download failed")); } } } }); } }) .subscribeOn(Schedulers.io()) .observeOn(AndroidSchedulers.mainThread()) .subscribe(new Observer<DownloadInfo>() { @Override public void onSubscribe(Disposable d) { mDisposable.add(d); } @Override public void onNext(DownloadInfo downloadInfo) { progressHandler.onProgress(downloadInfo); } @Override public void onError(Throwable e) { progressHandler.onError(e); } @Override public void onComplete() { progressHandler.onCompleted(downloadInfo.getFile()); } }); }
3. Download Observable, that is, the download task of each thread, FileDownloadObservable:
/** * TODO * * @author Kelly * @version 1.0.0 * @filename FileDownloadObservable * @time 2021/9/6 17:37 * @copyright(C) 2021 song */ public class FileDownloadObservable { /** * Download url */ private String url; /** * Cached FIle */ private File file; /** * Start position */ private int startPosition; /** * End position */ private int endPosition; /** * current location */ private int curPosition; /** * complete */ private boolean finished = false; /** * How many have been downloaded */ private int downloadSize = 0; private Disposable disposable; private String name; public FileDownloadObservable(String url, File file, int startPosition, int endPosition) { this.url = url; this.file = file; this.startPosition = startPosition; this.curPosition = startPosition; this.endPosition = endPosition; this.name = ""; } public void download() { DownloadApi apiService = RetrofitHelper.getInstance().getApiService(DownloadApi.class); String range = "bytes=" + (startPosition) + "-" + endPosition; apiService.download(range, url) .flatMap(new Function<ResponseBody, ObservableSource<Integer>>() { @Override public ObservableSource<Integer> apply(final ResponseBody responseBody) throws Exception { return Observable.create(new ObservableOnSubscribe<Integer>() { @Override public void subscribe(ObservableEmitter<Integer> emitter) throws Exception { InputStream inputStream = null; long contentLength; RandomAccessFile randomAccessFile = null; byte[] buf = new byte[1024 * 8]; name = Thread.currentThread().getName(); try { inputStream = responseBody.byteStream(); contentLength = responseBody.contentLength(); System.out.println(name + ",startPosition " + startPosition + ",endPosition " + endPosition); randomAccessFile = new RandomAccessFile(file, "rwd"); //Set start write location randomAccessFile.seek(startPosition); System.out.println(name + "Connection succeeded,Read length:" + FileUtils.formatFileSize(contentLength)); while (curPosition < endPosition) { //The current location is less than the end location. Continue downloading int len = inputStream.read(buf); if (len == -1) { //Download complete System.out.println(len); break; } randomAccessFile.write(buf, 0, len); curPosition = curPosition + len; if (curPosition > endPosition) { //If there are too many downloads, subtract the excess System.out.println(name + "curPosition > endPosition !!!!"); int extraLen = curPosition - endPosition; downloadSize += (len - extraLen + 1); } else { downloadSize += len; } // emitter.onNext(downloadSize); } finished = true; //Download completed at current stage System.out.println("current" + name + "Download complete"); if (!emitter.isDisposed()) { emitter.onComplete(); } } catch (Exception e) { if (!emitter.isDisposed()) { emitter.onError(e); } } finally { //Close flow if (inputStream != null) { try { inputStream.close(); } catch (IOException e) { e.printStackTrace(); } } try { randomAccessFile.close(); } catch (IOException e) { System.out.println("AccessFile IOException " + e.getMessage()); } } } }); } }) .subscribeOn(Schedulers.newThread())//New thread Download .subscribe(new Observer<Integer>() { @Override public void onSubscribe(Disposable d) { disposable = d; } @Override public void onNext(Integer downloadSize) { } @Override public void onError(Throwable e) { System.out.println(name + "download error Exception " + e.getMessage()); } @Override public void onComplete() { } }); } /** * Do you want to finish downloading the current segment * * @return */ public boolean isFinished() { return finished; } /** * How many have been downloaded * * @return */ public int getDownloadSize() { return downloadSize; } public Disposable getDisposable() { return disposable; } }
Download call
public void multiThreadDownloadTest(View view) { String url = "https://imtt.dd.qq.com/16891/apk/B168BCBBFBE744DA4404C62FD18FFF6F.apk?fsname=com.tencent.tmgp.sgame_1.61.1.6_61010601.apk&csr=1bbd"; final NumberFormat numberFormat = NumberFormat.getInstance(); // The setting is accurate to 2 decimal places numberFormat.setMaximumFractionDigits(2); FileDownloader.downloadFile3(threadNum, url, FileDownloadActivity.DOWNLOAD_APK_PATH, "multi_test.apk", new DownloadProgressHandler() { @Override public void onProgress(DownloadInfo downloadInfo) { long fileSize = downloadInfo.getFileSize(); long speed = downloadInfo.getSpeed(); long usedTime = downloadInfo.getUsedTime(); long currentSize = downloadInfo.getCurrentSize(); String percent = numberFormat.format((float) currentSize / (float) fileSize * 100); mProgress.setText(percent + "%"); mFileSize.setText(FileUtils.formatFileSize(fileSize)); mRate.setText(FileUtils.formatFileSize(speed) + "/s"); mTime.setText(FileUtils.formatTime(usedTime)); } @Override public void onCompleted(File file) { showMsg("Download complete:" + file.getAbsolutePath()); } @Override public void onError(Throwable e) { showMsg("Download File exception:" + e.getMessage()); } }); }
Summary
Theoretically, the larger the number of threads, the faster the download. However, too many threads will cause most of the CPU overhead to be spent on inter process switching, but it will be slower. In addition, the download speed is also related to the bandwidth. Therefore, the number of threads must be appropriate.