SpringBoot 2 Batch Service

Opening Word

This guide will guide you through creating basic batch-driven solutions.
 

Applications you will create

We will build a service that imports data from a CSV spreadsheet, converts it using custom code, and stores the final results in a database.
 

Tools you will need

How to complete this guide

Like most Starter's Guide to Spring Similarly, you can start from scratch and complete each step, or you can bypass the basic setup steps you are already familiar with.Either way, you end up with code that works.

  • To start over, move to Start with Spring Initializr
  • To skip the basics, do the following:
    • Download and unzip what the guide will use source code Or clone it with Git: git clone https://github.com/spring-guides/gs-batch-processing.git
    • Switch to the gs-batch-processing/initial directory;
    • Jump to the Guide's Create Business Class.

When you're ready, you can check the code in the gs-batch-processing/completion directory.
 

Business Data

Typically, our customers or business analysts provide spreadsheets.For this simple example, we can find some test data in src/main/resources/sample-data.csv:

Jill,Doe
Joe,Doe
Justin,Doe
Jane,Doe
John,Doe

The spreadsheet contains a first and last name on each line, separated by commas.This is a common pattern Spring can handle without customization.

Next, we need to write an SQL script to create a table to store the data.We can find the script in src/main/resources/schema-all.sql:

DROP TABLE people IF EXISTS;

CREATE TABLE people  (
    person_id BIGINT IDENTITY NOT NULL PRIMARY KEY,
    first_name VARCHAR(20),
    last_name VARCHAR(20)
);

Spring Boot automatically runs schema-@@platform@@.sql during startup.-all is the default for all platforms.

 

Start with Spring Initializr

For all Spring apps, you should start with Spring Initializr Start.Initializr provides a quick way to extract the dependencies your application needs and complete many settings for you.This example requires Spring Batch and HyperSQL database dependencies.The following figure shows the Initializr settings for this sample project:

The figure above shows Initializr, which chooses Maven as the build tool.You can also use Gradle.It also displays the values of com.example and batch-processing as Group and Artifact, respectively.These values will be used in the rest of this example.

The following list shows the pom.xml file that was created when Maven was selected:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>2.2.2.RELEASE</version>
		<relativePath/> <!-- lookup parent from repository -->
	</parent>
	<groupId>com.example</groupId>
	<artifactId>batch-processing</artifactId>
	<version>0.0.1-SNAPSHOT</version>
	<name>batch-processing</name>
	<description>Demo project for Spring Boot</description>

	<properties>
		<java.version>1.8</java.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-batch</artifactId>
		</dependency>

		<dependency>
			<groupId>org.hsqldb</groupId>
			<artifactId>hsqldb</artifactId>
			<scope>runtime</scope>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
			<exclusions>
				<exclusion>
					<groupId>org.junit.vintage</groupId>
					<artifactId>junit-vintage-engine</artifactId>
				</exclusion>
			</exclusions>
		</dependency>
		<dependency>
			<groupId>org.springframework.batch</groupId>
			<artifactId>spring-batch-test</artifactId>
			<scope>test</scope>
		</dependency>
	</dependencies>

	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
			</plugin>
		</plugins>
	</build>

</project>

The following list shows the build.gradle file that was created when you selected Gradle:

plugins {
	id 'org.springframework.boot' version '2.2.2.RELEASE'
	id 'io.spring.dependency-management' version '1.0.8.RELEASE'
	id 'java'
}

group = 'com.example'
version = '0.0.1-SNAPSHOT'
sourceCompatibility = '1.8'

repositories {
	mavenCentral()
}

dependencies {
	implementation 'org.springframework.boot:spring-boot-starter-batch'
	runtimeOnly 'org.hsqldb:hsqldb'
	testImplementation('org.springframework.boot:spring-boot-starter-test') {
		exclude group: 'org.junit.vintage', module: 'junit-vintage-engine'
	}
	testImplementation 'org.springframework.batch:spring-batch-test'
}

test {
	useJUnitPlatform()
}

 

Create Business Class

Now you can see the format of the data input and output, and you can write code to represent a row of data, as shown in the following example (from src/main/java/com/example/batchprocessing/Person.java):

package com.example.batchprocessing;

public class Person {

  private String lastName;
  private String firstName;

  public Person() {
  }

  public Person(String firstName, String lastName) {
    this.firstName = firstName;
    this.lastName = lastName;
  }

  public void setFirstName(String firstName) {
    this.firstName = firstName;
  }

  public String getFirstName() {
    return firstName;
  }

  public String getLastName() {
    return lastName;
  }

  public void setLastName(String lastName) {
    this.lastName = lastName;
  }

  @Override
  public String toString() {
    return "firstName: " + firstName + ", lastName: " + lastName;
  }

}

We can instantiate the Person class with a name and name either through a constructor or by setting properties.
 

Create Intermediate Processor

A common example in batch processing is to ingest data, convert it, and pipe it elsewhere.Here, we need to write a simple converter that converts names to uppercase.The following list (from src/main/java/com/example/batchprocessing/PersonItemProcessor.java) shows how to do this:

package com.example.batchprocessing;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import org.springframework.batch.item.ItemProcessor;

public class PersonItemProcessor implements ItemProcessor<Person, Person> {

  private static final Logger log = LoggerFactory.getLogger(PersonItemProcessor.class);

  @Override
  public Person process(final Person person) throws Exception {
    final String firstName = person.getFirstName().toUpperCase();
    final String lastName = person.getLastName().toUpperCase();

    final Person transformedPerson = new Person(firstName, lastName);

    log.info("Converting (" + person + ") into (" + transformedPerson + ")");

    return transformedPerson;
  }

}

PersonItemProcessor implements Spring Batch's ItemProcessor interface.This makes it easy to connect code to batch jobs, which we will define later in this guide.Depending on the interface, we receive an incoming Person object and convert it to uppercase Person.

The input and output types do not have to be the same.In fact, sometimes an applied data stream requires another data type after reading a data source.
 

Summarize Batch Jobs

Now we need to organize the actual batch jobs.Spring Batch provides a number of utility classes that reduce the need to write custom code.Instead, we can focus on business logic.

To configure our jobs, we must first create a Spring @Configuration class in src/main/java/com/exampe/batchprocessing/BatchConfiguration.java, as shown in the following example:

@Configuration
@EnableBatchProcessing
public class BatchConfiguration {

  @Autowired
  public JobBuilderFactory jobBuilderFactory;

  @Autowired
  public StepBuilderFactory stepBuilderFactory;

    ...

}

For beginners, the @EnableBatchProcessing annotation adds a number of key beans that support jobs and save us a lot of work.This example uses a memory-based database (provided by @EnableBatchProcessing), which means the data will disappear after completion.It will also automatically identify the few factories needed below.Now add the following beans to our BatchConfiguration class to define readers, processors, and writers:

@Bean
public FlatFileItemReader<Person> reader() {
  return new FlatFileItemReaderBuilder<Person>()
    .name("personItemReader")
    .resource(new ClassPathResource("sample-data.csv"))
    .delimited()
    .names(new String[]{"firstName", "lastName"})
    .fieldSetMapper(new BeanWrapperFieldSetMapper<Person>() {{
      setTargetType(Person.class);
    }})
    .build();
}

@Bean
public PersonItemProcessor processor() {
  return new PersonItemProcessor();
}

@Bean
public JdbcBatchItemWriter<Person> writer(DataSource dataSource) {
  return new JdbcBatchItemWriterBuilder<Person>()
    .itemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>())
    .sql("INSERT INTO people (first_name, last_name) VALUES (:firstName, :lastName)")
    .dataSource(dataSource)
    .build();
}

The first part of the code defines the inputs, processors, and outputs:

  • Reader() creates an ItemReader.It looks for a file named sample-data.csv, analyzes each item, and provides enough information to convert it to Person;
  • processor() creates a PersonItemProcessor instance that we previously defined to convert data to uppercase;
  • write(DataSource) creates an ItemWriter.This is for the JDBC target and automatically gets a copy of the dataSource created by @EnableBatchProcessing.It contains the SQL statements required to insert a single Person driven by Java bean properties.

The last block (from src/main/java/com/example/batchprocessing/BatchConfiguration.java) shows the actual job configuration:

@Bean
public Job importUserJob(JobCompletionNotificationListener listener, Step step1) {
  return jobBuilderFactory.get("importUserJob")
    .incrementer(new RunIdIncrementer())
    .listener(listener)
    .flow(step1)
    .end()
    .build();
}

@Bean
public Step step1(JdbcBatchItemWriter<Person> writer) {
  return stepBuilderFactory.get("step1")
    .<Person, Person> chunk(10)
    .reader(reader())
    .processor(processor())
    .writer(writer)
    .build();
}

The first method defines the job, and the second method defines a step.Jobs are built step by step, with each step involving a reader, processor, and writer.

In this job definition, we need an incrementer because jobs use a database to maintain execution status.Then we list each step (although the job has only one step).The job ends, and the Java API produces a well-configured job.

In the step definition, we define how much data to write at a time.In this case, it writes up to 10 records at a time.Next, use the previously injected bits to match the reader, processor, and writer.

chunk() is prefixed with <Person, Person> because it is a canonical method.This indicates the input and output types processed by each "block" and aligns with ItemReader <Person>and ItemWriter <Person>

The last bit of the batch configuration is the way to get notifications when the job is finished.The following example (from src/main/java/com/example/batchprocessing/JobCompletionNotificationListener.java) shows this class:

package com.example.batchprocessing;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.batch.core.BatchStatus;
import org.springframework.batch.core.JobExecution;
import org.springframework.batch.core.listener.JobExecutionListenerSupport;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Component;

@Component
public class JobCompletionNotificationListener extends JobExecutionListenerSupport {

  private static final Logger log = LoggerFactory.getLogger(JobCompletionNotificationListener.class);

  private final JdbcTemplate jdbcTemplate;

  @Autowired
  public JobCompletionNotificationListener(JdbcTemplate jdbcTemplate) {
    this.jdbcTemplate = jdbcTemplate;
  }

  @Override
  public void afterJob(JobExecution jobExecution) {
    if(jobExecution.getStatus() == BatchStatus.COMPLETED) {
      log.info("!!! JOB FINISHED! Time to verify the results");

      jdbcTemplate.query("SELECT first_name, last_name FROM people",
        (rs, row) -> new Person(
          rs.getString(1),
          rs.getString(2))
      ).forEach(person -> log.info("Found <" + person + "> in the database."));
    }
  }
}

JobCompletionNotificationListener listens for jobs in BatchStatus.COMPLETED and uses JdbcTemplate to check the results.
 

Make the application executable

Although batch processing can be embedded in Web applications and WAR files, the approach shown below creates a stand-alone application in a simpler way.We packaged everything in an executable JAR file, driven by an old and good Java main() method.

Spring Initializr creates an application class.For this simple example, it works without further modifications.The following list (from src/main/java/com/example/batchprocessing/BatchProcessingApplication.java) shows the application classes:

package com.example.batchprocessing;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class BatchProcessingApplication {

  public static void main(String[] args) throws Exception {
    SpringApplication.run(BatchProcessingApplication.class, args);
  }
}

@SpringBootApplication is a convenient comment that adds all of the following:

  • @Configuration: Mark the class as the source of the application context Bean definition;
  • @EnableAutoConfiguration: Tell Spring Boot to add beans based on class path configuration, other beans, and configuration of various properties.
  • @ComponentScan: Tell Spring to find his components, configurations, and services in the com/example package.

The main() method uses Spring Boot's Spring Application.run () method to start the application.

For demonstration purposes, some code creates a JdbcTemplate, queries the database, and prints the name of the person inserted by the batch job.
 

Build Executable JAR

We can combine Gradle or Maven to run the application from the command line.We can also build and run an executable JAR file that contains all the required dependencies, classes, and resources.Building an executable JAR makes it easy to publish, version, and deploy services as applications throughout the development life cycle, across environments, and so on.

If you use Gradle, you can run the application with the help of. /gradlew bootRun.Or you can build a JAR file with the help of. /gradlew build and run the JAR file as follows:

java -jar build/libs/gs-batch-processing-0.1.0.jar

The above command executed by the official website is different from my local one and I need it to run: java-jar build/libs/batch-processing-0.0.1-SNAPSHOT.jar.

If you use Maven, you can run it with the help of. /mvnw spring-boot:run.Or you can build a JAR file with. /mvnw clean package and run the JAR file as follows:

java -jar target/gs-batch-processing-0.1.0.jar

The above command from the official website executes differently than my local counterpart, and I need it to run: java-jar target/batch-processing-0.0.1-SNAPSHOT.jar.

We can still Build a classic WAR file.

This job prints a line for each person to change.After the job runs, we can also see the output of the database query.It should look like the following output:

Converting (firstName: Jill, lastName: Doe) into (firstName: JILL, lastName: DOE)
Converting (firstName: Joe, lastName: Doe) into (firstName: JOE, lastName: DOE)
Converting (firstName: Justin, lastName: Doe) into (firstName: JUSTIN, lastName: DOE)
Converting (firstName: Jane, lastName: Doe) into (firstName: JANE, lastName: DOE)
Converting (firstName: John, lastName: Doe) into (firstName: JOHN, lastName: DOE)
Found <firstName: JILL, lastName: DOE> in the database.
Found <firstName: JOE, lastName: DOE> in the database.
Found <firstName: JUSTIN, lastName: DOE> in the database.
Found <firstName: JANE, lastName: DOE> in the database.
Found <firstName: JOHN, lastName: DOE> in the database.

 

Summary

Congratulations!We build a batch job, change the job to extract data from the spreadsheet, process it, and write it to the database.
 

See

The following guidelines may also be helpful:

Want to see the rest of the guide?Visit the Guide's own column: ' Spring Official Guide>

110 original articles published, 6 praised, 6371 visits
Private letter follow

Tags: Spring Java Database Maven

Posted on Sat, 01 Feb 2020 21:52:48 -0500 by ale8oneboy