After the upgrade of Spring Boot, the initialization SQL does not run?

Spring Boot unit tests automatically execute test scripts

Every code merge of our company needs to pass the unit test.

The reason is that the data does not meet the test. The data is used once. Each time unit test is conducted, the data script will delete the last new data and create a new one.

In version 1.5.x, the execution of data scripts depends on the automatic configuration of Spring Boot, DataSourceAutoConfiguration and DataSourceInitializer. The actual script execution depends on the DataSourceInitializer, but the creation of the DataSourceInitializer depends on DataSourceAutoConfiguration. So we need to analyze these two classes together.

Of course, if you are familiar with Spring container management Bean lifecycle, in fact, DataSourceAutoConfiguration can be ignored.

DataSourceAutoConfiguration implements the automatic Configuration of DataSource (it is actually a Configuration class with multiple @ Conditional annotations). However, due to the single type of DataSource it supports, it is generally impossible to rely on its own DataSource initialization.

Our microservices have implemented their own DataSource automatic configuration, using DruidDataSource.

@Configuration @ConditionalOnClass({ DataSource.class, EmbeddedDatabaseType.class }) @EnableConfigurationProperties(DataSourceProperties.class) @Import({ Registrar.class, DataSourcePoolMetadataProvidersConfiguration.class }) public class DataSourceAutoConfiguration { // When initializing, inject DataSourceProperties and context @Bean @ConditionalOnMissingBean public DataSourceInitializer dataSourceInitializer(DataSourceProperties properties, ApplicationContext applicationContext) { return new DataSourceInitializer(properties, applicationContext); } /* Omit irrelevant code*/ }

The main function of DataSourceAutoConfiguration is to declare the instantiation of DataSourceInitializer and initialize the Registrar through the @Import dependency.

@EnableConfigurationProperties( DataSourceProperties.class )The datasourceproperties class is also initialized.

If you rely on the automatic configuration of the data source, the configuration of the data source depends on the configuration stored in the DataSourceProperties.
However, since the data source configuration of our company is self configured and initialized, the meaning of data source properties is to automatically execute the configuration of test scripts

The key configurations are as follows (1.5.x is the same as 2.x):

# DML scripts that define automation scripts spring.datasource.schema= # DDL scripts that define automation scripts spring.datasource.data= # The data source configuration for cannot be used because the data source has already been initialized # Used to initialize the data source when executing a script spring.datasource.schemaUsername= spring.datasource.schemaPassword= spring.datasource.dataUsername= spring.datasource.dataPassword=

The corresponding DataSourceProperties properties are as follows:

/** * Schema (DDL) script resource references. */ private List<String> schema; /** * User of the database to execute DDL scripts (if different). */ private String schemaUsername; /** * Password of the database to execute DDL scripts (if different). */ private String schemaPassword; /** * Data (DML) script resource references. */ private List<String> data; /** * User of the database to execute DML scripts. */ private String dataUsername; /** * Password of the database to execute DML scripts. */ private String dataPassword;

So what is the role of Registrar?
Its purpose is to inject a PostProcessor, and the function of PostProcessor is to initialize DataSourceInitializer as soon as DataSource is initialized.

class DataSourceInitializerPostProcessor implements BeanPostProcessor, Ordered { private int order = Ordered.HIGHEST_PRECEDENCE; @Override // Highest priority initialization, execution public int getOrder() { return this.order; } @Autowired private BeanFactory beanFactory; @Override public Object postProcessBeforeInitialization(Object bean, String beanName) throws BeansException { return bean; } @Override public Object postProcessAfterInitialization(Object bean, String beanName) throws BeansException { if (bean instanceof DataSource) { // force initialization of this bean as soon as we see a DataSource // Force the initialization of the DataSourceInitializer immediately after the DataSource is initialized this.beanFactory.getBean(DataSourceInitializer.class); } return bean; } /** * {@link ImportBeanDefinitionRegistrar} to register the * {@link DataSourceInitializerPostProcessor} without causing early bean instantiation * issues. */ static class Registrar implements ImportBeanDefinitionRegistrar { private static final String BEAN_NAME = "dataSourceInitializerPostProcessor"; @Override public void registerBeanDefinitions(AnnotationMetadata importingClassMetadata, BeanDefinitionRegistry registry) { // To inject BeanDefinition of DataSourceInitializerPostProcessor // Ensure that the container initializes it if (!registry.containsBeanDefinition(BEAN_NAME)) { GenericBeanDefinition beanDefinition = new GenericBeanDefinition(); beanDefinition.setBeanClass(DataSourceInitializerPostProcessor.class); beanDefinition.setRole(BeanDefinition.ROLE_INFRASTRUCTURE); // We don't need this one to be post processed otherwise it can cause a // cascade of bean instantiation that we would rather avoid. beanDefinition.setSynthetic(true); registry.registerBeanDefinition(BEAN_NAME, beanDefinition); } } } }

Next, take a look at the main DataSourceInitializer. It is responsible for the automatic execution of scripts. There are two entries: trigger after initialization and event trigger.

class DataSourceInitializer implements ApplicationListener<DataSourceInitializedEvent> { /* Required properties, data sources, context*/ private final DataSourceProperties properties; private final ApplicationContext applicationContext; private DataSource dataSource; /* Identifies whether the script has been initialized, which is executed only once globally */ private boolean initialized = false; @PostConstruct public void init() { // to configure spring.datasource.initialize // The default value is true, that is, 1.5.x automatically executes initialization scripts by default if (!this.properties.isInitialize()) { logger.debug("Initialization disabled (not running DDL scripts)"); return; } // Get DataSource if (this.applicationContext.getBeanNamesForType(DataSource.class, false, false).length > 0) { this.dataSource = this.applicationContext.getBean(DataSource.class); } if (this.dataSource == null) { logger.debug("No DataSource found so not initializing"); return; } // Execute DML script runSchemaScripts(); } @Override public void onApplicationEvent(DataSourceInitializedEvent event) { // Respond to the datasource initialized event (datasource source) event // Configuration judgment if (!this.properties.isInitialize()) { logger.debug("Initialization disabled (not running data scripts)"); return; } // NOTE the event can happen more than once and // the event datasource is not used here // Idempotent judgment if (!this.initialized) { // Execute DDL script runDataScripts(); this.initialized = true; } } }

DataSourceInitializedEvent(DataSource source) is the event that triggers automatic DDL execution. You can notice that its event source is DataSource. But note: executing scripts does not depend on the event's DataSource. It's no use.
There are two trigger times: 1. After executing DML script in runSchemaScripts, events will be pushed; 2. After JPA data source initialization, events will be triggered. The former will be mentioned later, and the latter interested students can have a look at it by themselves.

private void runSchemaScripts() { // 1. Get the DML script location, this.properties.getXxx It's user-defined List<Resource> scripts = getScripts("spring.datasource.schema", this.properties.getSchema(), "schema"); if (!scripts.isEmpty()) { // Script execution exists // 2. Corresponding user name and password String username = this.properties.getSchemaUsername(); String password = this.properties.getSchemaPassword(); // 3. Execute the script runScripts(scripts, username, password); try { // Push DataSourceInitializedEvent this.applicationContext .publishEvent(new DataSourceInitializedEvent(this.dataSource)); // Note: at this time, the event listener may not be registered in the container, so you can't rely on the event driver to ensure the execution of DDL scripts // The listener might not be registered yet, so don't rely on it. if (!this.initialized) { // Execute DDL script runDataScripts(); this.initialized = true; } } catch (IllegalStateException ex) { logger.warn("Could not send event to complete DataSource initialization (" + ex.getMessage() + ")"); } } } private void runDataScripts() { // 1. Get the DDL script location, this.properties.getXxx It's user-defined List<Resource> scripts = getScripts("spring.datasource.data", this.properties.getData(), "data"); // 2. Corresponding user name and password String username = this.properties.getDataUsername(); String password = this.properties.getDataPassword(); // 3. Execute the script runScripts(scripts, username, password); } private List<Resource> getScripts(String propertyName, List<String> resources, String fallback) { // If it is a user-defined location, the resource is searched directly according to the user-defined path // The last parameter is whether to throw an exception when the resource does not exist. For user-defined resources, an exception is thrown, and the container fails to start quickly if (resources != null) { return getResources(propertyName, resources, true); } // spring.datasource.platform =Test, used to splice the script name String platform = this.properties.getPlatform(); List<String> fallbackResources = new ArrayList<String>(); // fallback is the passed in 'data' or 'schema' // Example: classpath*:data-test.sql fallbackResources.add("classpath*:" + fallback + "-" + platform + ".sql"); // For example: classpath*:data.sql fallbackResources.add("classpath*:" + fallback + ".sql"); // According to the default path to obtain resources. The default is not thrown, indicating that it will not be executed return getResources(propertyName, fallbackResources, false); } private List<Resource> getResources(String propertyName, List<String> locations, boolean validate) { List<Resource> resources = new ArrayList<Resource>(); for (String location : locations) { for (Resource resource : doGetResources(location)) { if (resource.exists()) { resources.add(resource); } else if (validate) { throw new ResourceNotFoundException(propertyName, resource); } } } return resources; } private Resource[] doGetResources(String location) { try { SortedResourcesFactoryBean factory = new SortedResourcesFactoryBean( this.applicationContext, Collections.singletonList(location)); factory.afterPropertiesSet(); return factory.getObject(); } catch (Exception ex) { throw new IllegalStateException("Unable to load resources from " + location, ex); } }

private void runScripts(List<Resource> resources, String username, String password) { if (resources.isEmpty()) { return; } // Script executor, built-in some script configuration, such as encoding, sql statement separator character, comment character, whether to continue error, whether to continue to delete table failure, etc ResourceDatabasePopulator populator = new ResourceDatabasePopulator(); // spring.datasource.continueOnError The default is false. If the execution fails, do you want to continue populator.setContinueOnError(this.properties.isContinueOnError()); // spring.datasource.separator Default ';', sql statement separator character populator.setSeparator(this.properties.getSeparator()); // Script character encoding if (this.properties.getSqlScriptEncoding() != null) { populator.setSqlScriptEncoding(this.properties.getSqlScriptEncoding().name()); } for (Resource resource : resources) { populator.addScript(resource); } DataSource dataSource = this.dataSource; if (StringUtils.hasText(username) && StringUtils.hasText(password)) { // The username here is the previous data username or schema username, and the password is the same // If not null, the DataSource is initialized according to the configuration dataSource = DataSourceBuilder.create(this.properties.getClassLoader()) .driverClassName(this.properties.determineDriverClassName()) .url(this.properties.determineUrl()).username(username) .password(password).build(); } // Since the DataSource is initialized by the project itself, the configuration is also the same as the spring.datasource . isolation // So by default, it is executed according to the DataSource created by ourselves DatabasePopulatorUtils.execute(populator, dataSource); }

Due to our specification, the initialization script is placed in classpath*:/db/xxx.sql , so you need to add it manually in the configuration.
Although the specification requires the table structure to be defined in advance, there must be schema.sql Can be executed data.sql .
So:

spring.datasource.schema=classpath*:db/schema.sql spring.datasource.data=classpath*:db/data.sql

However, after upgrading to Spring Boot 2.x, there was some cleanup
First, the above configuration was cleaned up by mistake. However, after the investigation was added, it still couldn't work.
Second, 2.x is abandoned spring.datasource.initialized The initialization script is executed only when the data source is embedded by default.
What's the new configuration? Let's take a look at the logical changes in this area.

Structural changes

In version 2.x, the responsibilities of related classes are more clear, and each of them is his own. Avoid too many responsibilities for the class.

• Independent DataSourceInitializationConfiguration to avoid confusion of responsibilities for DataSourceAutoConfiguration.
• Move the register of registered PostProcessor to DataSourceInitializationConfiguration to ensure the single responsibility of PostProcessor.
• Split the DataSourceInitializer into DataSourceInitializerInvoker. Only DataSourceInitializerInvoker is exposed to the public. This class is actually configured and registered.
  DataSourceInitializerInvoker is responsible for initializing DataSourceInitializer, pre initialization script configuration, data source, context preparation, event processing, etc. The DataSourceInitializer is only responsible for executing DML and DDL scripts.

class DataSourceInitializerInvoker implements ApplicationListener<DataSourceSchemaCreatedEvent>, InitializingBean { private final ObjectProvider<DataSource> dataSource; private final DataSourceProperties properties; private final ApplicationContext applicationContext; private DataSourceInitializer dataSourceInitializer; private boolean initialized; DataSourceInitializerInvoker(ObjectProvider<DataSource> dataSource, DataSourceProperties properties, ApplicationContext applicationContext) { this.dataSource = dataSource; this.properties = properties; this.applicationContext = applicationContext; } @Override public void afterPropertiesSet() { DataSourceInitializer initializer = getDataSourceInitializer(); if (initializer != null) { // Execute DML script boolean schemaCreated = this.dataSourceInitializer.createSchema(); if (schemaCreated) { initialize(initializer); } } } private void initialize(DataSourceInitializer initializer) { try { // Push events this.applicationContext.publishEvent(new DataSourceSchemaCreatedEvent(initializer.getDataSource())); // The listener might not be registered yet, so don't rely on it. if (!this.initialized) { // Execute DDL script this.dataSourceInitializer.initSchema(); this.initialized = true; } } catch (IllegalStateException ex) { logger.warn("Could not send event to complete DataSource initialization (" + ex.getMessage() + ")"); } } @Override public void onApplicationEvent(DataSourceSchemaCreatedEvent event) { // NOTE the event can happen more than once and // the event datasource is not used here DataSourceInitializer initializer = getDataSourceInitializer(); if (!this.initialized && initializer != null) { // Execute DDL script initializer.initSchema(); this.initialized = true; } } private DataSourceInitializer getDataSourceInitializer() { // Delay initialization of dataSourceInitializer if (this.dataSourceInitializer == null) { // Ensure that only one or more datasources have Primary set DataSource ds = this.dataSource.getIfUnique(); if (ds != null) { this.dataSourceInitializer = new DataSourceInitializer(ds, this.properties, this.applicationContext); } } return this.dataSourceInitializer; } }

As you can see, the general process is the same. It's just that the logic in some places can be extracted to make it clearer. But there are still some small changes:

• DataSourceInitializer was originally initialized by a container, but now it is initialized by DataSourceInitializerInvoker.
  The initialization process guarantees the existence of a unique or primary DataSource. Otherwise, an error will be reported
  Originally, as long as there is a DataSource, any one will be selected. This is a change.
• Originally, in PostConstruct, first determine whether initialization is allowed( spring.datasource.initialized). Now it's up to the next step.
• DataSourceInitializedEvent -> DataSourceSchemaCreatedEvent. Only the name has changed, the behavior and trigger time have not changed.

The above mainly depends on two methods of DataSourceInitializer: createSchema and initSchema.

/** * Create the schema if necessary. * @return {@code true} if the schema was created * @see DataSourceProperties#getSchema() */ public boolean createSchema() { List<Resource> scripts = getScripts("spring.datasource.schema", this.properties.getSchema(), "schema"); if (!scripts.isEmpty()) { // Judge whether execution is allowed if (!isEnabled()) { logger.debug("Initialization disabled (not running DDL scripts)"); return false; } String username = this.properties.getSchemaUsername(); String password = this.properties.getSchemaPassword(); runScripts(scripts, username, password); } // The schema created returned above only cares whether the schema script is empty, not whether it is actually executed return !scripts.isEmpty(); } /** * Initialize the schema if necessary. * @see DataSourceProperties#getData() */ public void initSchema() { List<Resource> scripts = getScripts("spring.datasource.data", this.properties.getData(), "data"); if (!scripts.isEmpty()) { // Judge whether execution is allowed if (!isEnabled()) { logger.debug("Initialization disabled (not running data scripts)"); return; } String username = this.properties.getDataUsername(); String password = this.properties.getDataPassword(); runScripts(scripts, username, password); } } private boolean isEnabled() { // spring.datasource.initialization-mode = never || embedded || always // The default is embedded DataSourceInitializationMode mode = this.properties.getInitializationMode(); //never if (mode == DataSourceInitializationMode.NEVER) { return false; } // Embedded. If the data source is embedded, it is allowed to execute if (mode == DataSourceInitializationMode.EMBEDDED && !isEmbedded()) { return false; } // always allows execution return true; } private boolean isEmbedded() { try { return EmbeddedDatabaseConnection.isEmbedded(this.dataSource); } catch (Exception ex) { logger.debug("Could not determine if datasource is embedded", ex); return false; } }

Another change of 2.x DataSourceInitializer is the addition of a property resourceLoader: the loader used to load script resources.

In 1.5.x, use ApplicationContext to load. Now you can specify it yourself, otherwise create a default resource loader

DataSourceInitializer(DataSource dataSource, DataSourceProperties properties, ResourceLoader resourceLoader) { this.dataSource = dataSource; this.properties = properties; this.resourceLoader = (resourceLoader != null) ? resourceLoader : new DefaultResourceLoader(); } private Resource[] doGetResources(String location) { try { SortedResourcesFactoryBean factory = new SortedResourcesFactoryBean(this.resourceLoader, Collections.singletonList(location)); factory.afterPropertiesSet(); return factory.getObject(); } catch (Exception ex) { throw new IllegalStateException("Unable to load resources from " + location, ex); } }

Because it is not a critical situation, if the execution fails, you can run the script manually and then run the test again.
So the problem was put on for a long time, and it was only filled in when a colleague found a hole when he was upgrading recently.

Note: Although our data table structure is initialized in advance, it is generally required to have a DML before DDL can be executed. So schema.sql There must be, and there must be statements.

Therefore, the sentence show tables; is usually added;. Otherwise, it will be judged whether the script content is empty or empty string during actual execution.

Assert.hasText(script, "'script' must not be null or empty");

29 June 2020, 03:22 | Views: 1575