SpringBoot recognizes numbers with deep learning model: development details

Overview of this article

  1. What is the model file minist-model.zip mentioned earlier? How did you get here?
  2. After SpringBoot gets the model file, how to use it? What does it have to do with recognition?
  • Today, let's write code together to develop the application mentioned above, which will also answer your questions. The whole article consists of the following contents:
  1. Overview of the whole process
  2. Training practice
  3. In the design phase of SpringBoot application, the first key point is highlighted
  4. The second key point is emphasized in the design stage of SpringBoot application
  5. In the design stage of SpringBoot application, sort out the complete process
  6. SpringBoot application coding
  7. Make the application into a docker image

Overview of the whole process

  • In short, if you want to solve practical problems through in-depth learning (taking image recognition as an example), you need to do the following two things:
  1. Take the existing data for training, and save the training results in the model file
  2. Use model files in business code to identify images
  • Let's start with training:
  1. Prepare some handwritten digital pictures in advance. The corresponding numbers of these pictures have been determined. As shown in the figure below, the handwritten characters of 9 are all under directory 9
  1. Write the code of the training model and set various parameters of the neural network, such as normalization, activation function, loss function, convolution layers and so on
  2. Execute the code of the training model and take the above pictures for training
  3. Save the training results to a file, which is the model file. In the previous article, it is minist-model.zip
  4. This completes the training
  • The next step is to use this model file to solve practical problems
  1. Develop a business application
  2. Load the model file generated by the previous training in the application
  3. After receiving the data submitted by the user, submit it to the model for processing
  4. Return the processing result to the user
  • The above is the general process of applying in-depth learning to business scenarios. Next, let's start with training and start the actual combat process

Source download

  • The complete source code in this actual combat can be downloaded from GitHub. The address and link information are shown in the table below( https://github.com/zq2599/blog_demos):

name

link

remarks

Project Home

https://github.com/zq2599/blog_demos

The project is on the GitHub home page

git warehouse address (https)

https://github.com/zq2599/blog_demos.git

The warehouse address of the source code of the project, https protocol

git warehouse address (ssh)

git@github.com:zq2599/blog_demos.git

The project source code warehouse address, ssh protocol

  • There are multiple folders in this git project. The source code of DL4J practical combat series is in the DL4J tutorials folder, as shown in the red box below:
  • There are several sub projects in the dl4j tutorials folder. The actual combat code is in the dlfj tutorials directory, as shown in the red box below:

Training practice

  • The detailed process of the whole training, that is, the generation process of minist-model.zip, has been completed in DL4J practice 3: Classic convolution example (LeNet-5) It is explained in detail in the article, and the complete code is also given. You only need to operate once according to the article, but there is a problem to pay special attention to before operation:
  • DL4J practice 3: Classic convolution example (LeNet-5) The version of deeplearning 4J framework used in this article is 1.0.0-beta6. Please change it to 1.0.0-beta7. The specific change method is to open the pom.xml file of the simple Revolution project (note that it is the simple Revolution project, not its parent project dlfj tutorials), and modify the content in red box 2 in the figure below:
  • After modification, you can run the program to generate the model file, and then enter the stage of using the model
  • You may ask, why DL4J practice 3: Classic convolution example (LeNet-5) Will you use version 1.0.0-beta 6 in this article? In fact, it is to use GPU to accelerate the training process. At that time, 1.0.0-beta7 does not support CUDA-9.2. GPU acceleration will not be used in this article, so it is recommended to use version 1.0.0-beta7
  • Next, start developing the SpringBoot application, which uses the model to identify images

SpringBoot application design (the first key point)

  • There are two key points to pay attention to in advance in the design stage. The first one is related to the picture size: when training the model, the pictures used are 28 * 28 pixels, so our application needs to scale the pictures submitted by the user to 28 * 28 pixels

SpringBoot application design (the second key point)

  • Take another look at the pictures used for training, as shown below. All the pictures are white on black background:
  • So the problem comes: the model is trained according to the white characters on a black background. It can't recognize the black characters on a white background. How to deal with the pictures with black characters on a white background?
  • Therefore, if the user inputs a picture with black characters on a white background, our program will reverse the color, change it into white characters on a black background, and then do recognition

SpringBoot application design (process design)

  • Now let's sort out the whole process. If the user inputs a picture with black characters on a white background, the processing flow of the whole application is as follows:
  • If the user inputs a picture with white characters on a black background, you only need to remove the reverse color processing in the above process
  • Provide special interface predict with white background for black word pictures on white background
  • Provide special interface predict with black background for white pictures on black background
  • Now that the design work has been completed, you can start coding

Use model (coding)

  • To facilitate the management of demo code and the versions of dependent libraries, DL4J practice one: preparation A maven project named dl4j tutorials is created in this article. The application we are building today is also a sub project of dl4j tutorials, named predict number image. Its own pom.xml content is as follows:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>dlfj-tutorials</artifactId>
        <groupId>com.bolingcavalry</groupId>
        <version>1.0-SNAPSHOT</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <artifactId>predict-number-image</artifactId>

    <packaging>jar</packaging>

    <!--no need spring-boot-starter-parent As parent Configuration at-->
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-dependencies</artifactId>
                <version>${springboot.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>

        <dependency>
            <groupId>com.bolingcavalry</groupId>
            <artifactId>commons</artifactId>
            <version>${project.version}</version>
        </dependency>

        <dependency>
            <groupId>org.nd4j</groupId>
            <!--Be careful to use nd4j-native-platform,Otherwise, an error occurs when the container starts: no jnind4jcpu in java.library.path-->
            <!--<artifactId>${nd4j.backend}</artifactId>-->
            <artifactId>nd4j-native-platform</artifactId>
        </dependency>

        <dependency>
            <groupId>ch.qos.logback</groupId>
            <artifactId>logback-classic</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

    </dependencies>

    <build>
        <plugins>
            <!-- If the parent project is not springboot,You need to use the plug-in in the following ways to generate a normal plug-in jar -->
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <mainClass>com.bolingcavalry.predictnumber.PredictNumberApplication</mainClass>
                </configuration>
                <executions>
                    <execution>
                        <goals>
                            <goal>repackage</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>
  • It can be seen that the predict number image project has little to do with its parent project dlfj tutorials. It only uses the version numbers of several libraries defined by the parent project. You can also create a project without parent-child relationship independently;
  • Create a new configuration file application.properties, which contains the configuration related to the picture:
# Maximum total uploaded files
spring.servlet.multipart.max-request-size=1024MB

# Maximum value for a single file
spring.servlet.multipart.max-file-size=10MB

# Directory for processing picture files
predict.imagefilepath=/app/images/

# Model location
predict.modelpath=/app/model/minist-model.zip
  • The static methods required for processing images are concentrated in the ImageFileUtil.java file, mainly including save (save to disk), resize (zoom), colorRevert (reverse color), clear (clean), getGrayImageFeatures (feature extraction, the operation is the same as that during training):
package com.bolingcavalry.commons.utils;

import lombok.extern.slf4j.Slf4j;
import org.datavec.api.split.FileSplit;
import org.datavec.image.loader.NativeImageLoader;
import org.datavec.image.recordreader.ImageRecordReader;
import org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator;
import org.nd4j.linalg.api.ndarray.INDArray;
import org.nd4j.linalg.dataset.api.iterator.DataSetIterator;
import org.nd4j.linalg.dataset.api.preprocessor.ImagePreProcessingScaler;
import org.springframework.web.multipart.MultipartFile;
import javax.imageio.ImageIO;
import java.awt.*;
import java.awt.image.BufferedImage;
import java.io.File;
import java.io.IOException;
import java.util.UUID;

@Slf4j
public class ImageFileUtil {

    /**
     * Adjusted file width
     */
    public static final int RESIZE_WIDTH = 28;

    /**
     * Adjusted file height
     */
    public static final int RESIZE_HEIGHT = 28;

    /**
     * Save the uploaded file on the server
     * @param base The directory of the file to process
     * @param file Files to process
     * @return
     */
    public static String save(String base, MultipartFile file) {

        // Check whether it is empty
        if (file.isEmpty()) {
            log.error("invalid file");
            return null;
        }

        // The file name comes from the original file
        String fileName = file.getOriginalFilename();

        // Location to save
        File dest = new File(base + fileName);

        // Start saving
        try {
            file.transferTo(dest);
        } catch (IOException e) {
            log.error("upload fail", e);
            return null;
        }

        return fileName;
    }

    /**
     * Convert picture to 28 * 28 pixels
     * @param base     Directory to process files
     * @param fileName File name to be adjusted
     * @return
     */
    public static String resize(String base, String fileName) {

        // The new file name is the original file name with a random number suffix, and the extension is fixed to png
        String resizeFileName = fileName.substring(0, fileName.lastIndexOf(".")) + "-" + UUID.randomUUID() + ".png";

        log.info("start resize, from [{}] to [{}]", fileName, resizeFileName);

        try {
            // Read original file
            BufferedImage bufferedImage = ImageIO.read(new File(base + fileName));

            // Scaled instance
            Image image = bufferedImage.getScaledInstance(RESIZE_WIDTH, RESIZE_HEIGHT, Image.SCALE_SMOOTH);

            BufferedImage resizeBufferedImage = new BufferedImage(28, 28, BufferedImage.TYPE_INT_RGB);
            Graphics graphics = resizeBufferedImage.getGraphics();

            // mapping
            graphics.drawImage(image, 0, 0, null);
            graphics.dispose();

            // The converted picture is written into a file
            ImageIO.write(resizeBufferedImage, "png", new File(base + resizeFileName));

        } catch (Exception exception) {
            log.info("resize error from [{}] to [{}], {}", fileName, resizeFileName, exception);
            resizeFileName = null;
        }

        log.info("finish resize, from [{}] to [{}]", fileName, resizeFileName);

        return resizeFileName;
    }

    /**
     * Convert RGB to int number
     * @param alpha
     * @param red
     * @param green
     * @param blue
     * @return
     */
    private static int colorToRGB(int alpha, int red, int green, int blue) {
        int pixel = 0;

        pixel += alpha;
        pixel = pixel << 8;

        pixel += red;
        pixel = pixel << 8;

        pixel += green;
        pixel = pixel << 8;

        pixel += blue;

        return pixel;
    }

    /**
     * Reverse color treatment
     * @param base Directory to process files
     * @param src Source file for processing
     * @return New file after reverse color processing
     * @throws IOException
     */
    public static String colorRevert(String base, String src) throws IOException {
        int color, r, g, b, pixel;

        // Read original file
        BufferedImage srcImage = ImageIO.read(new File(base + src));

        // Modified file
        BufferedImage destImage = new BufferedImage(srcImage.getWidth(), srcImage.getHeight(), srcImage.getType());

        for (int i=0; i<srcImage.getWidth(); i++) {

            for (int j=0; j<srcImage.getHeight(); j++) {
                color = srcImage.getRGB(i, j);
                r = (color >> 16) & 0xff;
                g = (color >> 8) & 0xff;
                b = color & 0xff;
                pixel = colorToRGB(255, 0xff - r, 0xff - g, 0xff - b);
                destImage.setRGB(i, j, pixel);
            }
        }

        // The name of the reflection file
        String revertFileName =  src.substring(0, src.lastIndexOf(".")) + "-revert.png";

        // The converted picture is written into a file
        ImageIO.write(destImage, "png", new File(base + revertFileName));

        return revertFileName;
    }

    /**
     * Take the features of black-and-white pictures
     * @param base
     * @param fileName
     * @return
     * @throws Exception
     */
    public static INDArray getGrayImageFeatures(String base, String fileName) throws Exception {
        log.info("start getImageFeatures [{}]", base + fileName);

        // The same settings as when training the model
        ImageRecordReader imageRecordReader = new ImageRecordReader(RESIZE_HEIGHT, RESIZE_WIDTH, 1);

        FileSplit fileSplit = new FileSplit(new File(base + fileName),
                NativeImageLoader.ALLOWED_FORMATS);

        imageRecordReader.initialize(fileSplit);

        DataSetIterator dataSetIterator = new RecordReaderDataSetIterator(imageRecordReader, 1);
        dataSetIterator.setPreProcessor(new ImagePreProcessingScaler(0, 1));

        // Feature extraction
        return dataSetIterator.next().getFeatures();
    }

    /**
     * Batch cleanup files
     * @param base      Directory to process files
     * @param fileNames Collection of files to be cleaned
     */
    public static void clear(String base, String...fileNames) {
        for (String fileName : fileNames) {

            if (null==fileName) {
                continue;
            }

            File file = new File(base + fileName);

            if (file.exists()) {
                file.delete();
            }
        }
    }
}
  • There is only one method for defining the service layer. You can decide whether to reverse color processing through input parameters:
package com.bolingcavalry.predictnumber.service;

import org.springframework.web.multipart.MultipartFile;

public interface PredictService {

    /**
     * Get the uploaded picture, convert it and recognize it as a number
     * @param file Uploaded files
     * @param isNeedRevert Do you want to reverse color
     * @return
     */
    int predict(MultipartFile file, boolean isNeedRevert) throws Exception ;
}
  • The implementation of the sevice layer is also the core of this article. There are several points to note, which will be mentioned later:
package com.bolingcavalry.predictnumber.service.impl;

import com.bolingcavalry.commons.utils.ImageFileUtil;
import com.bolingcavalry.predictnumber.service.PredictService;
import lombok.extern.slf4j.Slf4j;
import org.deeplearning4j.nn.multilayer.MultiLayerNetwork;
import org.deeplearning4j.util.ModelSerializer;
import org.nd4j.linalg.api.ndarray.INDArray;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import org.springframework.web.multipart.MultipartFile;
import javax.annotation.PostConstruct;
import java.io.File;

@Service
@Slf4j
public class PredictServiceImpl implements PredictService {

    /**
     * -1 Indicates recognition failure
     */
    private static final int RLT_INVALID = -1;

    /**
     * Location of model files
     */
    @Value("${predict.modelpath}")
    private String modelPath;

    /**
     * Directory for processing picture files
     */
    @Value("${predict.imagefilepath}")
    private String imageFilePath;

    /**
     * neural network
     */
    private MultiLayerNetwork net;

    /**
     * bean Load the model when the instantiation is successful
     */
    @PostConstruct
    private void loadModel() {
        log.info("load model from [{}]", modelPath);

        // Loading model
        try {
            net = ModelSerializer.restoreMultiLayerNetwork(new File(modelPath));
            log.info("module summary\n{}", net.summary());
        } catch (Exception exception) {
            log.error("loadModel error", exception);
        }
    }

    @Override
    public int predict(MultipartFile file, boolean isNeedRevert) throws Exception {
        log.info("start predict, file [{}], isNeedRevert [{}]", file.getOriginalFilename(), isNeedRevert);

        // Pre stored file
        String rawFileName = ImageFileUtil.save(imageFilePath, file);

        if (null==rawFileName) {
            return RLT_INVALID;
        }

        // File name after reverse color processing
        String revertFileName = null;

        // Resized file name
        String resizeFileName;

        // Whether reverse color processing is required
        if (isNeedRevert) {
            // Reverse the original file, and the return result is the new file after reverse color processing
            revertFileName = ImageFileUtil.colorRevert(imageFilePath, rawFileName);

            // Adjust the color to 28*28 size file.
            resizeFileName = ImageFileUtil.resize(imageFilePath, revertFileName);
        } else {
            // Directly adjust the original file to a file of 28 * 28 size
            resizeFileName = ImageFileUtil.resize(imageFilePath, rawFileName);
        }

        // Now you've got the result, the file after anti color and resizing,
        // Then the original file and the anti color processed file can be deleted
        ImageFileUtil.clear(imageFilePath, rawFileName, revertFileName);

        // Take out the features of the black-and-white picture
        INDArray features = ImageFileUtil.getGrayImageFeatures(imageFilePath, resizeFileName);

        // Pass the features to the model for recognition
        return net.predict(features)[0];
    }
}
  • In the above code, there are two points to note:
  1. The loadModel method is executed during bean initialization, and the model file is loaded through ModelSerializer.restoreMultiLayerNetwork
  2. The real recognition operation is actually the MultiLayerNetwork.predict method. It's only one step. How simple
  • Then there is the web interface layer, which provides two interfaces:
package com.bolingcavalry.predictnumber.controller;

import com.bolingcavalry.predictnumber.service.PredictService;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.ResponseBody;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.multipart.MultipartFile;

@RestController
public class PredictController {

    final PredictService predictService;

    public PredictController(PredictService predictService) {
        this.predictService = predictService;
    }

    @PostMapping("/predict-with-black-background")
    @ResponseBody
    public int predictWithBlackBackground(@RequestParam("file") MultipartFile file) throws Exception {
        // When training the model, the numbers are white and black,
        // Therefore, if you upload a picture with white characters and black background, you can take it directly for recognition without reverse color processing
        return predictService.predict(file, false);
    }

    @PostMapping("/predict-with-white-background")
    @ResponseBody
    public int predictWithWhiteBackground(@RequestParam("file") MultipartFile file) throws Exception {
        // When training the model, the numbers are white and black,
        // Therefore, if you upload a picture with black characters and white background, you need to reverse the color,
        // After the color reversal, there are white characters and black background, which can be used for identification
        return predictService.predict(file, true);
    }
}
  • Finally, the startup class:
package com.bolingcavalry.predictnumber;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
@SpringBootApplication
public class PredictNumberApplication {
    public static void main(String[] args) {
        SpringApplication.run(PredictNumberApplication.class, args);
    }
}

Make docker image

  • There are many ways to make SpringBoot applications into docker images. Here, we use the methods officially recommended by SpringBoot:
  • First write the Dockerfile file and put it in the predict number image directory. It can be seen that it is only a simple file copy operation. Then specify the startup command:
# When the 8-jdk-alpine version starts, it will crash
FROM openjdk:8u292-jdk

# Create directory
RUN mkdir -p /app/images && mkdir -p /app/model

# Specifies the source location of the mirrored content
ARG DEPENDENCY=target/dependency

# Copy content to mirror
COPY ${DEPENDENCY}/BOOT-INF/lib /app/lib
COPY ${DEPENDENCY}/META-INF /app/META-INF
COPY ${DEPENDENCY}/BOOT-INF/classes /app

# Specify start command
ENTRYPOINT ["java","-cp","app:app/lib/*","com.bolingcavalry.predictnumber.PredictNumberApplication"]
  • Next, prepare the files required in the Dockerfile and execute mvn clean package -U in the parent project directory. This is a pure make operation and has nothing to do with docker
  • Enter the predict number image directory and execute the following command to extract class, configuration file, dependency library and other contents from the jar file to the target/dependency Directory:
mkdir -p target/dependency && (cd target/dependency; jar -xf ../*.jar)
  • Finally, execute the command docker build - t bolingcavalry / dl4j model app: 0.0.3 in the directory where the Dockerfile file is located (there is a point at the end of the command, don't miss it), and then the image production can be completed
  • If you have an account of hub.docker.com, you can also push the image to the central warehouse through the docker push command, so that more people can use it:
  • Finally, let's review Three minute experience: SpringBoot recognizes numbers with deep learning model The command to start the docker container in the article is as follows. Through the two - v parameters, the directory of the host computer is mapped to the container. Therefore, the / app/images and / app/model in the container can remain unchanged as long as the directory mapping of the host computer can be ensured to be correct:
docker run \
--rm \
-p 18080:8080 \
-v /home/will/temp/202106/29/images:/app/images \
-v /home/will/temp/202106/29/model:/app/model \
bolingcavalry/dl4j-model-app:0.0.3
  • For more information about docker image production officially recommended by SpringBoot, please refer to Making Docker image with SpringBoot(2.4) application (official scheme of Gradle version)
  • So far, the development practice of SpringBoot using deep learning model to identify numbers has been completed. If you are a java programmer interested in deep learning, I believe this article can bring you some references. For more in-depth learning practice, please pay attention to the original series of DL4J practice by Xinchen;

Posted on Mon, 06 Dec 2021 22:40:10 -0500 by Thikho