catalogue
2, Source code analysis of Arrays.sort
3. TimSort exception case analysis
1, Problem recurrence@Data public class BusinessApplyDto implements Comparable<BusinessApplyDto>, Serializable { private static final long serialVersionUID = 7697030805482359129L; private String period; private Long businessId; @Override public int compareTo(BusinessApplyDto target) { String currentNo = this.period; String targetNo = target.getPeriod(); if (currentNo == null) { return -1; } else if (targetNo == null) { return 1; } int sort = currentNo.compareTo(targetNo); if (sort == 0) { if (this.businessId == null) { return 1; } else if (target.businessId == null) { return -1; } return this.businessId.compareTo(target.businessId); } return -sort; } }
First, take a closer look at the above code. BusinessApplyDto implements a Comparable, and then rewrites the compareTo method. The compareTo method determines the size of the two BusinessApplyDto by comparing the values of the two fields. However, the compareTo method is too complex. At first glance, it is not easy to find bugs in the code. It will not even run for a period of time after going online, because if there is no specific data to trigger a bug, the online can always run normally. However, if you run with the following data, an exception will occur.
public class MainTest { public static void main(String[] args) { List<BusinessApplyDto> list = new ArrayList<>(); BusinessApplyDto businessApplyDto1 = new BusinessApplyDto(); businessApplyDto1.setPeriod("20210927"); businessApplyDto1.setBusinessId(5L); list.add(businessApplyDto1); BusinessApplyDto businessApplyDto2 = new BusinessApplyDto(); businessApplyDto2.setPeriod("20210927"); list.add(businessApplyDto2); BusinessApplyDto businessApplyDto3 = new BusinessApplyDto(); businessApplyDto3.setPeriod("20210927"); list.add(businessApplyDto3); BusinessApplyDto businessApplyDto4 = new BusinessApplyDto(); businessApplyDto4.setPeriod("20210927"); list.add(businessApplyDto4); BusinessApplyDto businessApplyDto5 = new BusinessApplyDto(); businessApplyDto5.setPeriod("20210927"); list.add(businessApplyDto5); BusinessApplyDto businessApplyDto6 = new BusinessApplyDto(); businessApplyDto6.setPeriod("20210927"); list.add(businessApplyDto6); BusinessApplyDto businessApplyDto7 = new BusinessApplyDto(); businessApplyDto7.setPeriod("20210927"); list.add(businessApplyDto7); BusinessApplyDto businessApplyDto8 = new BusinessApplyDto(); businessApplyDto8.setPeriod("20210927"); businessApplyDto8.setBusinessId(2L); list.add(businessApplyDto8); BusinessApplyDto businessApplyDto9 = new BusinessApplyDto(); businessApplyDto9.setPeriod("20210927"); list.add(businessApplyDto9); BusinessApplyDto businessApplyDto10 = new BusinessApplyDto(); businessApplyDto10.setPeriod("20210927"); list.add(businessApplyDto10); BusinessApplyDto businessApplyDto11 = new BusinessApplyDto(); businessApplyDto11.setPeriod("20210927"); list.add(businessApplyDto11); BusinessApplyDto businessApplyDto12 = new BusinessApplyDto(); businessApplyDto12.setPeriod("20210927"); list.add(businessApplyDto12); BusinessApplyDto businessApplyDto13 = new BusinessApplyDto(); businessApplyDto13.setPeriod("20210927"); list.add(businessApplyDto13); BusinessApplyDto businessApplyDto14 = new BusinessApplyDto(); businessApplyDto14.setPeriod("20210927"); list.add(businessApplyDto14); BusinessApplyDto businessApplyDto15 = new BusinessApplyDto(); businessApplyDto15.setPeriod("20210927"); list.add(businessApplyDto15); BusinessApplyDto businessApplyDto16 = new BusinessApplyDto(); businessApplyDto16.setPeriod("20210927"); list.add(businessApplyDto16); BusinessApplyDto businessApplyDto17 = new BusinessApplyDto(); businessApplyDto17.setPeriod("20210927"); list.add(businessApplyDto17); BusinessApplyDto businessApplyDto18 = new BusinessApplyDto(); businessApplyDto18.setPeriod("20210927"); list.add(businessApplyDto18); BusinessApplyDto businessApplyDto19 = new BusinessApplyDto(); businessApplyDto19.setPeriod("20210927"); list.add(businessApplyDto19); BusinessApplyDto businessApplyDto20 = new BusinessApplyDto(); businessApplyDto20.setPeriod("20210927"); list.add(businessApplyDto20); BusinessApplyDto businessApplyDto21 = new BusinessApplyDto(); businessApplyDto21.setPeriod("20210927"); list.add(businessApplyDto21); BusinessApplyDto businessApplyDto22 = new BusinessApplyDto(); businessApplyDto22.setPeriod("20210927"); list.add(businessApplyDto22); BusinessApplyDto businessApplyDto23 = new BusinessApplyDto(); businessApplyDto23.setPeriod("20210927"); list.add(businessApplyDto23); BusinessApplyDto businessApplyDto24 = new BusinessApplyDto(); businessApplyDto24.setPeriod("20210927"); list.add(businessApplyDto24); BusinessApplyDto businessApplyDto25 = new BusinessApplyDto(); businessApplyDto25.setPeriod("20210927"); list.add(businessApplyDto25); BusinessApplyDto businessApplyDto26 = new BusinessApplyDto(); businessApplyDto26.setPeriod("20210927"); list.add(businessApplyDto26); BusinessApplyDto businessApplyDto27 = new BusinessApplyDto(); businessApplyDto27.setPeriod("20210927"); list.add(businessApplyDto27); BusinessApplyDto businessApplyDto28 = new BusinessApplyDto(); businessApplyDto28.setPeriod("20210927"); list.add(businessApplyDto28); BusinessApplyDto businessApplyDto29 = new BusinessApplyDto(); businessApplyDto29.setPeriod("20210927"); list.add(businessApplyDto29); BusinessApplyDto businessApplyDto30 = new BusinessApplyDto(); businessApplyDto30.setPeriod("20210927"); list.add(businessApplyDto30); BusinessApplyDto businessApplyDto31 = new BusinessApplyDto(); businessApplyDto31.setPeriod("20210927"); businessApplyDto31.setBusinessId(6L); list.add(businessApplyDto31); BusinessApplyDto businessApplyDto32 = new BusinessApplyDto(); businessApplyDto32.setPeriod("20210927"); list.add(businessApplyDto32); Collections.sort(list); } }
If the above main method is executed, the following exception will be reported:
Exception in thread "main" java.lang.IllegalArgumentException: Comparison method violates its general contract! at java.util.ComparableTimSort.mergeLo(ComparableTimSort.java:744) at java.util.ComparableTimSort.mergeAt(ComparableTimSort.java:481) at java.util.ComparableTimSort.mergeCollapse(ComparableTimSort.java:406) at java.util.ComparableTimSort.sort(ComparableTimSort.java:213) at java.util.Arrays.sort(Arrays.java:1312) at java.util.Arrays.sort(Arrays.java:1506) at java.util.ArrayList.sort(ArrayList.java:1464) at java.util.Collections.sort(Collections.java:143) at MainTest.main(MainTest.java:148)
In the above MainTest data, if the businessId is null, no error will be reported, and only special data will trigger this error. As for why this error is triggered, the more intuitive reason is that compareTo in BusinessApplyDto violates several principles of sort:
- Reflexivity: the comparison results of x and y are opposite to those of Y and x.
- Transitivity: x > y, Y > Z, then x > Z.
- Symmetry: if x=y, the comparison results of X and z are the same as those of Y and z.
The following goes deep into the sort source code to analyze the root causes of the above errors.
2, Source code analysis of Arrays.sortBy viewing Collections.sort, you can know that the bottom layer calls the Arrays.sort method. The code of Arrays.sort is as follows:
public static <T> void sort(T[] a, Comparator<? super T> c) { if (c == null) { sort(a); } else { if (LegacyMergeSort.userRequested) legacyMergeSort(a, c); else TimSort.sort(a, 0, a.length, c, null, 0, 0); } }
It can be found from the above code that if c is null, the sort(Object[] a) method will be called. Otherwise, the legacyMergeSort method or TimSort.sort method will be called according to whether LegacyMergeSort.userRequested is true. LegacyMergeSort.userRequested can be specified by adding - Djava.util.Arrays.useLegacyMergeSort=true/false to the JVM parameters. The sort(Object[] a) code is as follows:
public static void sort(Object[] a) { if (LegacyMergeSort.userRequested) legacyMergeSort(a); else ComparableTimSort.sort(a, 0, a.length, null, 0, 0); }
It can be seen that the logic of sort(Object[] a) and sort (t [] A, Comparator <? Super T > C) methods is roughly the same. The bottom layer of sort(Object[] a) calls ComparableTimSort.sort, while sort (t [] A, Comparator <? Super t > C) calls TimSort.sort. In fact, both ComparableTimSort and TimSort call the TimSort algorithm, but ComparableTimSort is compared through Comparable, and TimSort is compared through Comparator. / * * To be removed in a future release. * / is annotated on legacyMergeSort method, indicating that it will be removed in future versions. Next, instead of posting detailed code, I sorted out a flow chart of Arrays.sort.
From the above flow chart, we can feel that the underlying implementation of Arrays.sort is relatively complex. It can be found from the flowchart that the underlying logic of legacyMergeSort will judge the array. If the array size is less than 7, insert sort will be used to sort the array, otherwise merge sort will be used. ComparableTimSort and TimSort have the same logic and are implemented using TimSort. The comparison method violations its general contract! Errors are also thrown in TimSort, so the following mainly analyzes TimSort.
3, TimSort1. Introduction to TimSort
TimSort was invented by Tim Peter in 2002. It is an adaptive, hybrid and stable sorting algorithm, which integrates the essence of merging algorithm and binary insertion sorting algorithm. It has a particularly excellent performance in real-world data, so it is used as a built-in sorting algorithm in Java and Python. The reason why this algorithm is fast is that it makes full use of the data to be sorted in the real world. Many substrings have been sorted and do not need to be reordered. Using this feature and adding appropriate consolidation rules can sort the remaining to be sorted more efficiently.
2. TimSort principle
TimSort combines binary insertion sort and merge sort. When the array length is less than a certain threshold, use binary insertion sorting. Otherwise, calculate minRun and then perform binary insertion sorting + merge sorting. This threshold was 64 when TimSort was first implemented, and later changed to 32. It is changed to 32 because this threshold can get better performance.
1) , binary insertion sort
Binary insertion sorting: first, find the position where this element should be inserted through binary search, which can reduce many comparisons. Although the same number of elements still need to be moved, the time consumption of copying the array is less than the one-to-one exchange between elements. For example, for the array [3,7,12,32,50,1], if you want to insert 1 into the front of the array, you need to go through 5 comparisons through direct insertion sorting, but using binary insertion sorting, you only need 2 comparisons to find the insertion position, and then use System.arraycopy to move the array.
2),minRunLength
The main function of the minRunLength method is to confirm the minimum value of run. After determining the minRun value, the array to be sorted will be divided into a block sub array with the size of minRun as the block. The code of minRunLength is as follows:
private static int minRunLength(int n) { assert n >= 0; int r = 0; // Becomes 1 if any 1 bits are shifted off while (n >= MIN_MERGE) { r |= (n & 1); n >>= 1; } return n + r; }
As you can see from the code, the size of minRunLength is between 0 and 32. When n < 100, the corresponding data of N and minRunLength(n) are as follows:
n minRunLength(n) 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 12 12 13 13 14 14 15 15 16 16 17 17 18 18 19 19 20 20 21 21 22 22 23 23 24 24 25 25 26 26 27 27 28 28 29 29 30 30 31 31 32 16 33 17 34 17 35 18 36 18 37 19 38 19 39 20 40 20 41 21 42 21 43 22 44 22 45 23 46 23 47 24 48 24 49 25 50 25 51 26 52 26 53 27 54 27 55 28 56 28 57 29 58 29 59 30 60 30 61 31 62 31 63 32 64 16 65 17 66 17 67 17 68 17 69 18 70 18 71 18 72 18 73 19 74 19 75 19 76 19 77 20 78 20 79 20 80 20 81 21 82 21 83 21 84 21 85 22 86 22 87 22 88 22 89 23 90 23 91 23 92 23 93 24 94 24 95 24 96 24 97 25 98 25 99 25
3) , do while loop
After calculating the minRunLength, enter a do while loop. The main function of this loop is to sort blocks through run and stack. There are several steps in do while:
- Countrunandmakescreening finds a run that must be in order. If it is in descending order, it will be operated in ascending order.
- If the block size determined in step a is less than the minRunLength value, call binarySort to extend the run.
- Call ts.pushRun(lo, runLen) to maintain the current run on the stack to prepare for the mergeCollapse later.
- Call ts.mergeCollapse() to merge each block. mergeCollapse will merge different run blocks. It is assumed that there are three consecutive blocks A, B and C. The following rules will be followed for merging:
- Only A and B or B and C will be merged, and A and C will not be merged.
- If there are only two blocks at present, if A < B, merge A and B.
- If there are currently three blocks, if A < = B + C, combine A and B until A > b + C and b > Z are met.
- Repeat steps a to d until all the arrays to be sorted are sorted.
- Finally, call mergeForceCollapse to merge all runs until there is only one run.
Specific data analysis: For demonstration purposes, the MIN_MERGE Set to 2 (the source code is 32); take minRun Set to 2. Assume that the initial array is[9,6,3,4,5,12,14,15,7,11,1,8,13,16,18,19] 1,Perform steps a，call countRunAndMakeAscending，Calculated runLen Value of. If there is a continuous descending order, the ascending order operation is performed. After the first step, the data becomes[3,6,9][4,5,12,14,15,7,11,1,8,13,16,18,19]. 2,Perform steps c，Will current run Maintained in the stack. The current stack block is[3]. 3,Perform steps d，call ts.mergeCollapse()，But because stackSize=1，Therefore, there is no need to enter mergeCollapse of while Cycle. 4,Continue with step a，call countRunAndMakeAscending. Data becomes[3,6,9][4,5,12,14,15][7,11,1,8,13,16,18,19]. 5,Perform steps c，The data of the stack is[3,5]. 6,Perform steps d，because runLen[0]=3，runLen[1]=5，therefore runLen[0]<=runLen[1]. So enter mergeAt method. 6.1,call gallopRight calculation run1 The first element of the run0 Location in, i.e run1 4 in should be inserted in run0 After 3 in. 6.2,call gallopLeft calculation run0 The last element of the run1 Position in, 9 should be inserted before 12, and then ignored run1 All elements after this position are smaller than run0 The element in is large. 6.3,So the remaining elements to be sorted are[6,9][4,5]. Then call mergeLo. The data after completion is:[3,4,5,6,9,12,14,15][7,11,1,8,13,16,18,19]. 7,Perform steps a，Find continuous ascending or descending data, and the resulting data becomes[3,4,5,6,9,12,14,15][7,11][1,8,13,16,18,19]. 8,Perform steps c，Current stack block is[8,2]. 9,Perform steps d，call ts.mergeCollapse()，because runLen[0]=8，runLen[1]=2，runLen[0]>runLen[1]，Therefore, do not enter while Cycle. 10,Perform steps a，Find continuous ascending or descending data, and the result is[3,4,5,6,9,12,14,15][7,11][1,8,13,16,18][19]. 11,Perform steps c，Current stack block is[8,2,5]. 12,Perform steps d，runLen[0]=8,runLen[1]=2,runLen[2]=5，because runLen[0]>runLen[1]+runLen[2]，however runLen[1]<=runLen[2]，Therefore merge run0 and run2. The result data is[3,4,5,6,9,12,14,15][1,7,8,11,13,16,18][19]. 13,Perform steps c，Current stack block is[8,7]. 14,Perform steps d，call ts.mergeCollapse(). runLen[0]=8，runLen[1]=7，runLen[0]>runLen[1]，Therefore, do not enter while Cycle. 15,Perform steps a,Find continuous ascending or descending data, and the result is[3,4,5,6,9,12,14,15][1,7,8,11,13,16,18][19]. 16,Perform steps c，Current block is[8,7,1]. 17,Perform steps d，call ts.mergeCollapse()，runLen[0]=8，runLen[1]=7，runLen[2]=1，because runLen[0]<=runLen[1]+runLen[2]also runLen[0]>runLen[2]，So first run1 and run2 Merge. 17.1,call gallopRight calculation run2 The first element of the run1 Location in, i.e run2 19 in should be inserted in run1 Behind 18 in. 17.2,call gallopLeft calculation run1 The last element of the run2 In position, 18 should be inserted in front of 19. 17.3,The remaining elements are[19]. Then call mergeHi. The data after completion is:[3,4,5,6,9,12,14,15][1,7,8,11,13,16,18,19]. The currently stacked block is[8,8]. 18,continue while Loop because runLen[0]=8，runLen[1]=8，runLen[0]<=runLen[1]. 18.1,call gallopRight，run1 1 inserted in run0 In front of 3. 18.2,call gallopLeft，run0 15 should be inserted in front of 13. 18.3,The remaining array to be sorted is[3,4,5,6,9,12,14,15][1,7,8,11,13]. Then call mergeHi，The result data is[1,3,4,5,6,7,8,9,11,12,13,14,15,16,18,19]. The current stack block size is[16]. 19,Last call mergeForceCollapse. because stackSize=1，So don't enter while Loop, end the whole sort Operation.
3. TimSort exception case analysis
At the beginning of this article, the period s in the MainTest data are actually equal, so the data can only look at the businessId field. Therefore, the above code can be abstracted into an array: [5, null, null, null, null, null, null, 2, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, 6, null]. After merging the data results once, the array is divided into two groups: [2,5, null, null, null, null, null, null, null, null, null, null, null, null] [6, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]. Where runLen[0]=16, runLen[1]=16. After calling mergeCollapse, because runlen [n] < = runlen [n + 1], it will enter the mergeAt(n) method.
After entering the mergeAt method, the value determined by calling gallopRight is 2, because 6 of run1 is beat 5 of run0, so the index value inserted is 2. The index value determined after calling gallopLeft is 16. Then enter the mergeLo method to merge [null,null,null,null,null,null, null,null,null,null,null,null,null,null, null], and [6, null,null,null,null,null,null, null,null,null,null,null,null,null,null, null, null, null]. The combined result is [2,5,6, null,null,null,null,null,null, null,null,null,null,null,null,null,null, null,null,null,null,null,null, null,null,null,null,null,null,null,null, null, null, null, null, null, null].
When the execution continues, it is found that the value of len1 is finally reduced to 0, so it enters the logic in the figure below, and then the comparison method violations its general contract is thrown! Abnormal.
Through analysis, we can know that the root cause of throwing this exception is the violation of the reflexive principle in the compareTo method of BusinessApplyDto. Null in the above array means that businessId is null, so 1 will be returned in BusinessApplyDto (see the code snippet below), representing ascending order. That is, BusinessApplyDto2 > = BusinessApplyDto3, but BusinessApplyDto2 < BusinessApplyDto3 in the merging process. It violates the reflexive principle, which eventually leads to an exception being thrown.
if (this.businessId == null) { return 1; }
After finding out the cause of the problem, you need to fix the bug. The repaired code is as follows:
@Data public class BusinessApplyDto implements Comparable<BusinessApplyDto>, Serializable { private static final long serialVersionUID = 7697030805482359129L; private String period; private Long businessId; @Override public int compareTo(BusinessApplyDto target) { String currentNo = this.period; String targetNo = target.getPeriod(); if (currentNo == null) { return -1; } else if (targetNo == null) { return 1; } int sort = currentNo.compareTo(targetNo); if (sort == 0) { if (this.businessId == null && target.businessId == null) { return 0; } if (this.businessId == null) { return 1; } else if (target.businessId == null) { return -1; } return this.businessId.compareTo(target.businessId); } return -sort; } }4, Summary
At the beginning, this paper shows the phenomenon of errors, then goes deep into the source code, reveals the implementation principle of TimSort and the hidden comparator criteria: reflexivity, transitivity and symmetry, and finally solves the above bug s. Through this article, we can find the pits easily encountered in TimSort. When implementing Comparable, we must ensure that the returned results include - 1, 0 and 1, otherwise it may violate the TimSort criteria.
5, Reference The fastest sorting algorithm in the world -- timport - fosseshensen - blog Park
Simple analysis of the sort() method in java.util.ComparableTimSort_ Life lies in tossing - CSDN blogTimsort for OpenJDK source code reading_ on_ 1y CSDN blog