Teach you how to import and export large amount of data with progress bar (Chapter 2)

summary

Next, take the export of 10000000 mobile phone number data as an example to develop the export function with progress bar.

requirement analysis

1. Progress bar analysis

There are two stages of file export: one is to query the database to generate data files; Second, data file download. For the download of data files, all major browsers can display the download progress well without adding a progress bar. Therefore, the progress bar shows the progress of generating data files by querying the database in one stage. The process of generating files with large amount of data will last for a long time. It is very appropriate to use the progress bar here.

1. Analysis of large data export query data

The basic scheme adopted for the query of large amount of data is to query the total amount of data first, calculate the number of queries required according to the number of queries set each time, then page query, dynamically write the query results into the data file, and then output the data file to the browser in the form of stream.

Traditional export development

The traditional method does not use the progress bar to display the export progress, and the export effect is as follows:

1. The page code is as follows:

<form action="download" method="post"> <input type="submit" value="export"> </form>

1. Java backend export processing, the code is as follows:

The exported files are temporarily stored in the / tmp directory

public void download() throws IOException { final String fileName = UUID.randomUUID().toString().replace("-",""); try (final FileWriter fileWriter = new FileWriter(new File("/tmp/" + fileName + ".csv")); final BufferedWriter bufferedWriter = new BufferedWriter(fileWriter);){ final long totalNum = remoteGetUserNum(); final long pageSize = 100L ; long totalPage = (totalNum%pageSize == 0L) ? totalNum/pageSize:(totalNum/pageSize + 1); for (long pageNum = 1L; pageNum <= totalPage; pageNum++) { List<String> userList = remoteGetUser(pageNum,pageSize); for (String s : userList) { bufferedWriter.write(s+"\r\n"); } } bufferedWriter.flush(); } final HttpServletResponse response = getResponse(); final File file = new File("/tmp/" + fileName + ".csv"); response.addHeader("Content-Disposition","attachment;filename=" + new String(fileName.getBytes())+".csv"); response.addHeader("Content-Length",""+file.length()); response.setContentType("application/octet-stream"); try (final FileInputStream in = new FileInputStream(file); final BufferedInputStream bufferedInputStream = new BufferedInputStream(in); final OutputStream outputStream = new BufferedOutputStream(response.getOutputStream());){ byte[] buffer = new byte[1024]; int len; while ((len = bufferedInputStream.read(buffer)) != -1){ outputStream.write(buffer,0,len); } outputStream.flush(); } renderNull(); } private List<String> remoteGetUser(long pageNum, long pageSize) { return userList; } List<String> userList = new ArrayList<>(); { for (long i = 0; i < 100; i++) { userList.add(String.valueOf(18321714630L+i)); } } private long remoteGetUserNum() { return 10000000L; }

To be exported from the progress bar

The effects are as follows:

1. Progress bar interface design

Write the static progress bar code under the export function button to change the progress by controlling the width of the internal div. it is hidden by default.

<form action="download" method="post"> <input type="submit" value="export"> <div id="progress" style="height:20px;width:100%;background: #efefef;border:1px solid #eee;border-radius:10px;display:none;"> <div style="background: green;width:10%;height: 100%;border-radius:10px;line-height:20px;">0%</div> </div> </form>

The static effect is as follows:

1. Front end page download request transformation

On the traditional export function page, click export to submit the form request to obtain the file. After using the progress bar export function, it is necessary to submit the export request and update the change of the progress bar. The request becomes a multi task. You need to use ajax requests to generate data files and update the progress bar display. Wait until the background data file generates a response to the Ajax request, and then initiate a form request through js to download the processed data file.

The code is changed as follows:

function download(){ var bar =$('#progress').show().find('.bar') bar.text('0%') bar.css(); var taskId = '' + new Date().getTime() var timer = setInterval(function(){ $.ajax({ type:'post', url:'getProgress', data:, dataType: "json", }).success(function(data){ if(data.result){ bar.text(data.value + '%') bar.animate(,300) } }).error(function(e){ }) },300); $.ajax({ type:'post', url:'download', data:, dataType: 'json', }).success(function(data){ if(data.result){ jQuery('<form action="downloadFile" method="post">' + // action request path and push method '<input type="text" name="taskId" value="' + taskId + '"/>' + '</form>') .appendTo('body').submit().remove(); } clearInterval(timer) $('#progress').hide().find('.bar').css() }).error(function(e){ clearInterval(timer) $('#progress').hide().find('.bar').css() // do some things }) }

1. Back end java code development

The two stages of file export are separated. The download method processes the contents of one stage and queries and generates data files in batches. This stage takes a long time and needs to add progress control; The downloadFile method handles the content of the second stage and outputs the data file to the browser in the form of stream. The download progress of this stage is well displayed by major browsers.

public void download() throws IOException { JSONObject jsonObject = new JSONObject(); jsonObject.put("result",true); jsonObject.put("msg","success"); final String taskId = getPara("taskId"); try (final FileWriter fileWriter = new FileWriter(new File("/tmp/" + taskId + ".csv")); final BufferedWriter bufferedWriter = new BufferedWriter(fileWriter);){ final long totalNum = remoteGetUserNum(); final long pageSize = 100L ; long totalPage = (totalNum%pageSize == 0L) ? totalNum/pageSize:(totalNum/pageSize + 1); for (long pageNum = 1L; pageNum <= totalPage; pageNum++) { List<String> userList = remoteGetUser(pageNum,pageSize); for (String s : userList) { bufferedWriter.write(s+"\r\n"); } taskProgressMap.put(taskId, Math.toIntExact(pageNum * 100 / totalPage)); } } renderJson(jsonObject.toJSONString()); } public void downloadFile() throws IOException { final String taskId = getPara("taskId"); final File file = new File("/tmp/" + taskId + ".csv"); final HttpServletResponse response = getResponse(); response.addHeader("Content-Disposition","attachment;filename=" + new String(file.getName().getBytes())); response.addHeader("Content-Length",""+file.length()); response.setContentType("application/octet-stream"); try (final FileInputStream in = new FileInputStream(file); final BufferedInputStream bufferedInputStream = new BufferedInputStream(in); final OutputStream outputStream = new BufferedOutputStream(response.getOutputStream());){ byte[] buffer = new byte[1024]; int len; while ((len = bufferedInputStream.read(buffer)) != -1){ outputStream.write(buffer,0,len); } outputStream.flush(); } renderNull(); }

12 November 2021, 10:45 | Views: 6689