After the upgrade of Spring Boot, the initialization SQL does not run?

Spring Boot unit tests automatically execute test scripts

cause

Recently, we are upgrading our microservices from SpringBoot 1.5.x to 2.x. During this period, some configuration changes were made, and the configuration changes of SpringBoot itself made it impossible to run the unit test after the upgrade.

Every code merge of our company needs to pass the unit test.

The reason is that the data does not meet the test. The data is used once. Each time unit test is conducted, the data script will delete the last new data and create a new one.

1.5.x

In version 1.5.x, the execution of data scripts depends on the automatic configuration of Spring Boot, DataSourceAutoConfiguration and DataSourceInitializer. The actual script execution depends on the DataSourceInitializer, but the creation of the DataSourceInitializer depends on DataSourceAutoConfiguration. So we need to analyze these two classes together.

Of course, if you are familiar with Spring container management Bean lifecycle, in fact, DataSourceAutoConfiguration can be ignored.

DataSourceAutoConfiguration

DataSourceAutoConfiguration implements the automatic Configuration of DataSource (it is actually a Configuration class with multiple @ Conditional annotations). However, due to the single type of DataSource it supports, it is generally impossible to rely on its own DataSource initialization.

Our microservices have implemented their own DataSource automatic configuration, using DruidDataSource.

@Configuration
@ConditionalOnClass({ DataSource.class, EmbeddedDatabaseType.class })
@EnableConfigurationProperties(DataSourceProperties.class)
@Import({ Registrar.class, DataSourcePoolMetadataProvidersConfiguration.class })
public class DataSourceAutoConfiguration {

     // When initializing, inject DataSourceProperties and context
	@Bean
	@ConditionalOnMissingBean
	public DataSourceInitializer dataSourceInitializer(DataSourceProperties properties,
			ApplicationContext applicationContext) {
		return new DataSourceInitializer(properties, applicationContext);
	}
	/* Omit irrelevant code*/
}

The main function of DataSourceAutoConfiguration is to declare the instantiation of DataSourceInitializer and initialize the Registrar through the @Import dependency.

@EnableConfigurationProperties( DataSourceProperties.class )The datasourceproperties class is also initialized.

If you rely on the automatic configuration of the data source, the configuration of the data source depends on the configuration stored in the DataSourceProperties.
However, since the data source configuration of our company is self configured and initialized, the meaning of data source properties is to automatically execute the configuration of test scripts

The key configurations are as follows (1.5.x is the same as 2.x):

# DML scripts that define automation scripts
spring.datasource.schema=
# DDL scripts that define automation scripts
spring.datasource.data=
# The data source configuration for cannot be used because the data source has already been initialized
# Used to initialize the data source when executing a script
spring.datasource.schemaUsername=
spring.datasource.schemaPassword=
spring.datasource.dataUsername=
spring.datasource.dataPassword=

The corresponding DataSourceProperties properties are as follows:

/**
 * Schema (DDL) script resource references.
 */
private List<String> schema;

/**
 * User of the database to execute DDL scripts (if different).
 */
private String schemaUsername;

/**
 * Password of the database to execute DDL scripts (if different).
 */
private String schemaPassword;

/**
 * Data (DML) script resource references.
 */
private List<String> data;

/**
 * User of the database to execute DML scripts.
 */
private String dataUsername;

/**
 * Password of the database to execute DML scripts.
 */
private String dataPassword;

So what is the role of Registrar?
Its purpose is to inject a PostProcessor, and the function of PostProcessor is to initialize DataSourceInitializer as soon as DataSource is initialized.

class DataSourceInitializerPostProcessor implements BeanPostProcessor, Ordered {

	private int order = Ordered.HIGHEST_PRECEDENCE;

	@Override
	// Highest priority initialization, execution
	public int getOrder() {
		return this.order;
	}

	@Autowired
	private BeanFactory beanFactory;

	@Override
	public Object postProcessBeforeInitialization(Object bean, String beanName)
			throws BeansException {
		return bean;
	}

	@Override
	public Object postProcessAfterInitialization(Object bean, String beanName)
			throws BeansException {
		if (bean instanceof DataSource) {
			// force initialization of this bean as soon as we see a DataSource
			// Force the initialization of the DataSourceInitializer immediately after the DataSource is initialized
			this.beanFactory.getBean(DataSourceInitializer.class);
		}
		return bean;
	}

	/**
	 * {@link ImportBeanDefinitionRegistrar} to register the
	 * {@link DataSourceInitializerPostProcessor} without causing early bean instantiation
	 * issues.
	 */
	static class Registrar implements ImportBeanDefinitionRegistrar {

		private static final String BEAN_NAME = "dataSourceInitializerPostProcessor";

		@Override
		public void registerBeanDefinitions(AnnotationMetadata importingClassMetadata,
				BeanDefinitionRegistry registry) {
				// To inject BeanDefinition of DataSourceInitializerPostProcessor
				// Ensure that the container initializes it
			if (!registry.containsBeanDefinition(BEAN_NAME)) {
				GenericBeanDefinition beanDefinition = new GenericBeanDefinition();
				beanDefinition.setBeanClass(DataSourceInitializerPostProcessor.class);
				beanDefinition.setRole(BeanDefinition.ROLE_INFRASTRUCTURE);
				// We don't need this one to be post processed otherwise it can cause a
				// cascade of bean instantiation that we would rather avoid.
				beanDefinition.setSynthetic(true);
				registry.registerBeanDefinition(BEAN_NAME, beanDefinition);
			}
		}
	}
}
DataSourceInitializer

Next, take a look at the main DataSourceInitializer. It is responsible for the automatic execution of scripts. There are two entries: trigger after initialization and event trigger.

class DataSourceInitializer implements ApplicationListener<DataSourceInitializedEvent> {
    /* Required properties, data sources, context*/
	private final DataSourceProperties properties;

	private final ApplicationContext applicationContext;

	private DataSource dataSource;
    /* Identifies whether the script has been initialized, which is executed only once globally */
	private boolean initialized = false;

	@PostConstruct
	public void init() {
	    // to configure spring.datasource.initialize
	    // The default value is true, that is, 1.5.x automatically executes initialization scripts by default
		if (!this.properties.isInitialize()) {
			logger.debug("Initialization disabled (not running DDL scripts)");
			return;
		}
		// Get DataSource
		if (this.applicationContext.getBeanNamesForType(DataSource.class, false,
				false).length > 0) {
			this.dataSource = this.applicationContext.getBean(DataSource.class);
		}
		if (this.dataSource == null) {
			logger.debug("No DataSource found so not initializing");
			return;
		}
		// Execute DML script
		runSchemaScripts();
	}

	@Override
	public void onApplicationEvent(DataSourceInitializedEvent event) {
	    // Respond to the datasource initialized event (datasource source) event
	    // Configuration judgment
		if (!this.properties.isInitialize()) {
			logger.debug("Initialization disabled (not running data scripts)");
			return;
		}
		// NOTE the event can happen more than once and
		// the event datasource is not used here
		// Idempotent judgment
		if (!this.initialized) {
		    // Execute DDL script
			runDataScripts();
			this.initialized = true;
		}
	}
}	

DataSourceInitializedEvent(DataSource source) is the event that triggers automatic DDL execution. You can notice that its event source is DataSource. But note: executing scripts does not depend on the event's DataSource. It's no use.
There are two trigger times: 1. After executing DML script in runSchemaScripts, events will be pushed; 2. After JPA data source initialization, events will be triggered. The former will be mentioned later, and the latter interested students can have a look at it by themselves.

runSchemaScripts & runDataScripts
    private void runSchemaScripts() {
        // 1. Get the DML script location, this.properties.getXxx  It's user-defined
        List<Resource> scripts = getScripts("spring.datasource.schema",
                this.properties.getSchema(), "schema");
        if (!scripts.isEmpty()) {
            // Script execution exists
            // 2. Corresponding user name and password
            String username = this.properties.getSchemaUsername();
            String password = this.properties.getSchemaPassword();
            // 3. Execute the script
            runScripts(scripts, username, password);
            try {
                // Push DataSourceInitializedEvent
                this.applicationContext
                        .publishEvent(new DataSourceInitializedEvent(this.dataSource));
                // Note: at this time, the event listener may not be registered in the container, so you can't rely on the event driver to ensure the execution of DDL scripts
                // The listener might not be registered yet, so don't rely on it.
                if (!this.initialized) {
                    // Execute DDL script
                    runDataScripts();
                    this.initialized = true;
                }
            }
            catch (IllegalStateException ex) {
                logger.warn("Could not send event to complete DataSource initialization ("
                        + ex.getMessage() + ")");
            }
        }
    }

    private void runDataScripts() {
        // 1. Get the DDL script location, this.properties.getXxx  It's user-defined
		List<Resource> scripts = getScripts("spring.datasource.data",
				this.properties.getData(), "data");
        // 2. Corresponding user name and password
		String username = this.properties.getDataUsername();
		String password = this.properties.getDataPassword();
		// 3. Execute the script
		runScripts(scripts, username, password);
	}

	private List<Resource> getScripts(String propertyName, List<String> resources,
			String fallback) {
        // If it is a user-defined location, the resource is searched directly according to the user-defined path
        // The last parameter is whether to throw an exception when the resource does not exist. For user-defined resources, an exception is thrown, and the container fails to start quickly
		if (resources != null) {
			return getResources(propertyName, resources, true);
		}
		// spring.datasource.platform  =Test, used to splice the script name
		String platform = this.properties.getPlatform();
		List<String> fallbackResources = new ArrayList<String>();
		// fallback is the passed in 'data' or 'schema'
		// Example: classpath*:data-test.sql
		fallbackResources.add("classpath*:" + fallback + "-" + platform + ".sql");
		// For example: classpath*:data.sql
		fallbackResources.add("classpath*:" + fallback + ".sql");
		// According to the default path to obtain resources. The default is not thrown, indicating that it will not be executed
		return getResources(propertyName, fallbackResources, false);
	}

	private List<Resource> getResources(String propertyName, List<String> locations,
			boolean validate) {
		List<Resource> resources = new ArrayList<Resource>();
		for (String location : locations) {
			for (Resource resource : doGetResources(location)) {
				if (resource.exists()) {
					resources.add(resource);
				}
				else if (validate) {
					throw new ResourceNotFoundException(propertyName, resource);
				}
			}
		}
		return resources;
	}

	private Resource[] doGetResources(String location) {
		try {
			SortedResourcesFactoryBean factory = new SortedResourcesFactoryBean(
					this.applicationContext, Collections.singletonList(location));
			factory.afterPropertiesSet();
			return factory.getObject();
		}
		catch (Exception ex) {
			throw new IllegalStateException("Unable to load resources from " + location,
					ex);
		}
	}
runScripts
    private void runScripts(List<Resource> resources, String username, String password) {
		if (resources.isEmpty()) {
			return;
		}
		// Script executor, built-in some script configuration, such as encoding, sql statement separator character, comment character, whether to continue error, whether to continue to delete table failure, etc
		ResourceDatabasePopulator populator = new ResourceDatabasePopulator();
		// spring.datasource.continueOnError  The default is false. If the execution fails, do you want to continue
		populator.setContinueOnError(this.properties.isContinueOnError());
		// spring.datasource.separator  Default ';', sql statement separator character
		populator.setSeparator(this.properties.getSeparator());
		// Script character encoding
		if (this.properties.getSqlScriptEncoding() != null) {
			populator.setSqlScriptEncoding(this.properties.getSqlScriptEncoding().name());
		}
		for (Resource resource : resources) {
			populator.addScript(resource);
		}
		DataSource dataSource = this.dataSource;
		if (StringUtils.hasText(username) && StringUtils.hasText(password)) {
		// The username here is the previous data username or schema username, and the password is the same
		// If not null, the DataSource is initialized according to the configuration
			dataSource = DataSourceBuilder.create(this.properties.getClassLoader())
					.driverClassName(this.properties.determineDriverClassName())
					.url(this.properties.determineUrl()).username(username)
					.password(password).build();
		}
		// Since the DataSource is initialized by the project itself, the configuration is also the same as the spring.datasource . isolation
		// So by default, it is executed according to the DataSource created by ourselves
		DatabasePopulatorUtils.execute(populator, dataSource);
	}

Due to our specification, the initialization script is placed in classpath*:/db/xxx.sql , so you need to add it manually in the configuration.
Although the specification requires the table structure to be defined in advance, there must be schema.sql Can be executed data.sql .
So:

spring.datasource.schema=classpath*:db/schema.sql
spring.datasource.data=classpath*:db/data.sql

However, after upgrading to Spring Boot 2.x, there was some cleanup
First, the above configuration was cleaned up by mistake. However, after the investigation was added, it still couldn't work.
Second, 2.x is abandoned spring.datasource.initialized The initialization script is executed only when the data source is embedded by default.
What's the new configuration? Let's take a look at the logical changes in this area.

Version 2.x

Structural changes

In version 2.x, the responsibilities of related classes are more clear, and each of them is his own. Avoid too many responsibilities for the class.

  • Independent DataSourceInitializationConfiguration to avoid confusion of responsibilities for DataSourceAutoConfiguration.
  • Move the register of registered PostProcessor to DataSourceInitializationConfiguration to ensure the single responsibility of PostProcessor.
  • Split the DataSourceInitializer into DataSourceInitializerInvoker. Only DataSourceInitializerInvoker is exposed to the public. This class is actually configured and registered.
    DataSourceInitializerInvoker is responsible for initializing DataSourceInitializer, pre initialization script configuration, data source, context preparation, event processing, etc. The DataSourceInitializer is only responsible for executing DML and DDL scripts.
DataSourceInitializerInvoker
class DataSourceInitializerInvoker implements ApplicationListener<DataSourceSchemaCreatedEvent>, InitializingBean {

	private final ObjectProvider<DataSource> dataSource;

	private final DataSourceProperties properties;

	private final ApplicationContext applicationContext;

	private DataSourceInitializer dataSourceInitializer;

	private boolean initialized;

	DataSourceInitializerInvoker(ObjectProvider<DataSource> dataSource, DataSourceProperties properties,
			ApplicationContext applicationContext) {
		this.dataSource = dataSource;
		this.properties = properties;
		this.applicationContext = applicationContext;
	}

	@Override
	public void afterPropertiesSet() {
		DataSourceInitializer initializer = getDataSourceInitializer();
		if (initializer != null) {
		    // Execute DML script
			boolean schemaCreated = this.dataSourceInitializer.createSchema();
			if (schemaCreated) {
				initialize(initializer);
			}
		}
	}

	private void initialize(DataSourceInitializer initializer) {
		try {
		    // Push events
			this.applicationContext.publishEvent(new DataSourceSchemaCreatedEvent(initializer.getDataSource()));
			// The listener might not be registered yet, so don't rely on it.
			if (!this.initialized) {
			    // Execute DDL script
				this.dataSourceInitializer.initSchema();
				this.initialized = true;
			}
		}
		catch (IllegalStateException ex) {
			logger.warn("Could not send event to complete DataSource initialization (" + ex.getMessage() + ")");
		}
	}

	@Override
	public void onApplicationEvent(DataSourceSchemaCreatedEvent event) {
		// NOTE the event can happen more than once and
		// the event datasource is not used here
		DataSourceInitializer initializer = getDataSourceInitializer();
		if (!this.initialized && initializer != null) {
		    // Execute DDL script
			initializer.initSchema();
			this.initialized = true;
		}
	}

	private DataSourceInitializer getDataSourceInitializer() {
	    // Delay initialization of dataSourceInitializer
		if (this.dataSourceInitializer == null) {
		    // Ensure that only one or more datasources have Primary set
			DataSource ds = this.dataSource.getIfUnique();
			if (ds != null) {
				this.dataSourceInitializer = new DataSourceInitializer(ds, this.properties, this.applicationContext);
			}
		}
		return this.dataSourceInitializer;
	}

}

As you can see, the general process is the same. It's just that the logic in some places can be extracted to make it clearer. But there are still some small changes:

  • DataSourceInitializer was originally initialized by a container, but now it is initialized by DataSourceInitializerInvoker.
    The initialization process guarantees the existence of a unique or primary DataSource. Otherwise, an error will be reported
    Originally, as long as there is a DataSource, any one will be selected. This is a change.
  • Originally, in PostConstruct, first determine whether initialization is allowed( spring.datasource.initialized). Now it's up to the next step.
  • DataSourceInitializedEvent -> DataSourceSchemaCreatedEvent. Only the name has changed, the behavior and trigger time have not changed.

The above mainly depends on two methods of DataSourceInitializer: createSchema and initSchema.

DataSourceInitializer
    /**
	 * Create the schema if necessary.
	 * @return {@code true} if the schema was created
	 * @see DataSourceProperties#getSchema()
	 */
	public boolean createSchema() {
		List<Resource> scripts = getScripts("spring.datasource.schema", this.properties.getSchema(), "schema");
		if (!scripts.isEmpty()) {
		    // Judge whether execution is allowed
			if (!isEnabled()) {
				logger.debug("Initialization disabled (not running DDL scripts)");
				return false;
			}
			String username = this.properties.getSchemaUsername();
			String password = this.properties.getSchemaPassword();
			runScripts(scripts, username, password);
		}
		// The schema created returned above only cares whether the schema script is empty, not whether it is actually executed
		return !scripts.isEmpty();
	}

	/**
	 * Initialize the schema if necessary.
	 * @see DataSourceProperties#getData()
	 */
	public void initSchema() {
		List<Resource> scripts = getScripts("spring.datasource.data", this.properties.getData(), "data");
		if (!scripts.isEmpty()) {
		     // Judge whether execution is allowed
			if (!isEnabled()) {
				logger.debug("Initialization disabled (not running data scripts)");
				return;
			}
			String username = this.properties.getDataUsername();
			String password = this.properties.getDataPassword();
			runScripts(scripts, username, password);
		}
	}

	private boolean isEnabled() {
	    // spring.datasource.initialization-mode = never || embedded || always
	    // The default is embedded
		DataSourceInitializationMode mode = this.properties.getInitializationMode();
		//never
		if (mode == DataSourceInitializationMode.NEVER) {
			return false;
		}
		// Embedded. If the data source is embedded, it is allowed to execute
		if (mode == DataSourceInitializationMode.EMBEDDED && !isEmbedded()) {
			return false;
		}
		// always allows execution
		return true;
	}

	private boolean isEmbedded() {
		try {
			return EmbeddedDatabaseConnection.isEmbedded(this.dataSource);
		}
		catch (Exception ex) {
			logger.debug("Could not determine if datasource is embedded", ex);
			return false;
		}
	}

Another change of 2.x DataSourceInitializer is the addition of a property resourceLoader: the loader used to load script resources.

In 1.5.x, use ApplicationContext to load. Now you can specify it yourself, otherwise create a default resource loader

DataSourceInitializer(DataSource dataSource, DataSourceProperties properties, ResourceLoader resourceLoader) {
    this.dataSource = dataSource;
    this.properties = properties;
    this.resourceLoader = (resourceLoader != null) ? resourceLoader : new DefaultResourceLoader();
}

private Resource[] doGetResources(String location) {
    try {
        SortedResourcesFactoryBean factory = new SortedResourcesFactoryBean(this.resourceLoader,
                Collections.singletonList(location));
        factory.afterPropertiesSet();
        return factory.getObject();
    }
    catch (Exception ex) {
        throw new IllegalStateException("Unable to load resources from " + location, ex);
    }
}

summary

Because it is not a critical situation, if the execution fails, you can run the script manually and then run the test again.
So the problem was put on for a long time, and it was only filled in when a colleague found a hole when he was upgrading recently.

Note: Although our data table structure is initialized in advance, it is generally required to have a DML before DDL can be executed. So schema.sql There must be, and there must be statements.

Therefore, the sentence show tables; is usually added;. Otherwise, it will be judged whether the script content is empty or empty string during actual execution.

Assert.hasText(script, "'script' must not be null or empty");

Tags: Spring SQL Database SpringBoot

Posted on Mon, 29 Jun 2020 03:22:12 -0400 by Hurklefish