A few days ago, I encountered a problem when implementing MyBatis batch insertion. When the amount of data inserted in batch is relatively large, the program execution will report an error, as shown in the following figure:
The reason is that MySQL can only execute a certain length of SQL statements, but when a large amount of data is inserted, a long SQL will be generated, so the program will report an error during execution.
There are two ways to solve this problem: first, set the maximum length that MySQL can execute SQL; Second, divide a large List into N small lists. Since it is impossible to accurately define the maximum SQL length in the program, the best solution is the second, so today's article is available.
brief introduction
The process of dividing a List into several small lists is called fragmentation. Of course, it can also be called "List separation". Just choose a name you like and easy to understand.
In Java, the common implementation methods of sharding are as follows:
- Use Google's Guava framework to realize fragmentation;
- Use the commons framework of Apache to realize fragmentation;
- Use domestic God level framework Hutool to realize fragmentation;
- Use the Stream provided in JDK 8 to realize fragmentation;
- Custom slicing function.
Let's look at it separately.
1.Google Guava
First, add framework support in pom.xml of the project, and add the following configurations:
<!-- google guava Tool class --> <!-- https://mvnrepository.com/artifact/com.google.guava/guava --> <dependency> <groupId>com.google.guava</groupId> <artifactId>guava</artifactId> <version>31.0.1-jre</version> </dependency>
With the Guava framework, you only need to use the Lists.partition method to implement sharding, as shown in the following code:
import com.google.common.collect.Lists; import java.util.Arrays; import java.util.List; /** * Guava Slice */ public class PartitionByGuavaExample { // Original set private static final List<String> OLD_LIST = Arrays.asList( "Tang Monk,name of a fictitious monkey with supernatural powers,Bajie,Monk Sha,Cao Cao,Liu Bei,Sun Quan".split(",")); public static void main(String[] args) { // Set partition List<List<String>> newList = Lists.partition(OLD_LIST, 3); // Print slice set newList.forEach(i -> { System.out.println("Set length:" + i.size()); }); } }
The execution result of the above code is shown in the following figure:
2.apache commons
First, add framework support in pom.xml of the project, and add the following configurations:
<!-- apache Collection tool class --> <!-- https://mvnrepository.com/artifact/org.apache.commons/commons-collections4 --> <dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-collections4</artifactId> <version>4.4</version> </dependency>
With the commons framework, you only need to use the ListUtils.partition method to implement sharding, as shown in the following code:
import org.apache.commons.collections4.ListUtils; import java.util.Arrays; import java.util.List; /** * commons.collections4 Set partition */ public class PartitionExample { // Original set private static final List<String> OLD_LIST = Arrays.asList( "Tang Monk,name of a fictitious monkey with supernatural powers,Bajie,Monk Sha,Cao Cao,Liu Bei,Sun Quan".split(",")); public static void main(String[] args) { // Set partition List<List<String>> newList = ListUtils.partition(OLD_LIST, 3); newList.forEach(i -> { System.out.println("Set length:" + i.size()); }); } }
The execution result of the above code is shown in the following figure:
3.Hutool
First, add framework support in pom.xml of the project, and add the following configurations:
<!-- Tool class hutool --> <!-- https://mvnrepository.com/artifact/cn.hutool/hutool-all --> <dependency> <groupId>cn.hutool</groupId> <artifactId>hutool-all</artifactId> <version>5.7.14</version> </dependency>
After having the Hutool framework, you only need to use the ListUtil.partition method to realize fragmentation, as shown in the following code:
import cn.hutool.core.collection.ListUtil; import java.util.Arrays; import java.util.List; public class PartitionByHutoolExample { // Original set private static final List<String> OLD_LIST = Arrays.asList( "Tang Monk,name of a fictitious monkey with supernatural powers,Bajie,Monk Sha,Cao Cao,Liu Bei,Sun Quan".split(",")); public static void main(String[] args) { // Slice processing List<List<String>> newList = ListUtil.partition(OLD_LIST, 3); newList.forEach(i -> { System.out.println("Set length:" + i.size()); }); } }
The execution result of the above code is shown in the following figure:
4.JDK
Stream
It is unnecessary to add any framework to realize fragmentation through Stream in JDK 8. The specific implementation code is as follows:
import java.util.Arrays; import java.util.List; import java.util.Map; import java.util.stream.Collectors; /** * JDK Stream Partition */ public class PartitionByStreamExample { // Original set private static final List<Integer> OLD_LIST = Arrays.asList( 1, 2, 3, 4, 5, 6); public static void main(String[] args) { // Set fragmentation: divide the data greater than 3 and less than or equal to 3 into two groups respectively Map<Boolean, List<Integer>> newMap = OLD_LIST.stream().collect( Collectors.partitioningBy(i -> i > 3) ); // Print results System.out.println(newMap); } }
The execution result of the above code is shown in the following figure:
The advantage of this method is that there is no need to add any framework, but the disadvantage is that it can only realize simple sharding (dividing a List into two) and have clear sharding conditions. For example, the fragmentation condition set in this case is whether the array is greater than 3. If it is greater than 3, it will be divided into one group, otherwise it will be divided into another group.
5. Custom segmentation
If you don't want to introduce a third-party framework and using Stream can't meet your needs, you can consider writing your own code to realize the fragmentation function. Because this method is not commonly used, we only give the key methods here.
The key implementation method of user-defined slicing function is the subList method provided by JDK, as shown in the following figure:
Use examples are as follows:
import java.util.Arrays; import java.util.List; public class App { private static final List<String> _OLD_LIST = Arrays.asList( "Tang Monk,name of a fictitious monkey with supernatural powers,Bajie,Monk Sha,Cao Cao,Liu Bei,Sun Quan".split(",")); public static void main(String[] args) { // Collection separation List<String> list = _OLD_LIST.subList(0, 3); // Prints the elements in the collection list.forEach(i -> { System.out.println(i); }); } }
The execution result of the above code is shown in the following figure:
summary
This paper introduces five implementation methods of List fragmentation. The most convenient implementation method is to introduce a third-party framework, such as Google's Guava, Apache's Commons or domestic open source Hutool. Of course, if your project already includes any of the above, you can use it directly. If it is a simple slicing, you can consider using the Stream of JDK or the subList method built in List to realize the slicing function.
Pay attention to the Chinese community of official account "Java Chinese community" to see more Java summary articles.