Spring boot logs, configuration files, interface data desensitization

Core privacy data is particularly important for both enterprises and users. Therefore, in order to find ways to eliminate the leakage of various privacy data, it is necessary to pay attention to in daily development:
  1. Profile data desensitization
  2. Interface return data desensitization
  3. Log file data desensitization  

How is the profile desensitized?

We often encounter such a situation: there are always some sensitive information in the project configuration file, such as the url, user name and password of the data source. Once these information is exposed, the whole database will be leaked. So how to hide these configurations? In the past, the encrypted configuration was written into the configuration file manually and decrypted manually when extracted. Of course, this is an idea and can also solve the problem, but don't you think it's troublesome to encrypt and decrypt manually every time?
 
Today, we introduce a scheme that enables you to encrypt and decrypt configuration files without perception. Use an open source plug-in: jasypt spring boot.

1. Add dependency

<dependency>
        <groupId>com.github.ulisesbocchio</groupId>
        <artifactId>jasypt-spring-boot-starter</artifactId>
        <version>3.0.3</version>
</dependency>

2. Configure the secret key

Add an encrypted secret key (any) to the configuration file, as follows:
jasypt:
  encryptor:
    password: Y6M9fAJQdU7jNp5MW
Of course, it is not safe to put the secret key directly in the configuration file. We can configure the secret key when the project is started. The command is as follows:
java -jar xxx.jar  -Djasypt.encryptor.password=Y6M9fAJQdU7jNp5MW

3. Generate encrypted data

This step is to encrypt the configuration plaintext. The code is as follows:
@SpringBootTest
@RunWith(SpringRunner.class)
public class SpringbootJasyptApplicationTests {

    /**
     * Injection encryption method
     */
    @Autowired
    private StringEncryptor encryptor;

    /**
     * The ciphertext is generated manually. The url, user and password are demonstrated here
     */
    @Test
    public void encrypt() {
        String url = encryptor.encrypt("jdbc\\:mysql\\:test?useUnicode\\");
        String name = encryptor.encrypt("root");
        String password = encryptor.encrypt("123456");
        System.out.println("database url: " + url);
        System.out.println("database name: " + name);
        System.out.println("database password: " + password);
        Assert.assertTrue(url.length() > 0);
        Assert.assertTrue(name.length() > 0);
        Assert.assertTrue(password.length() > 0);
    }
}
The above code encrypts the url, user and password of the data source in plaintext, and the output results are as follows:
database url: szkFDG56WcAOzG2utv0m2aoAvNFH5g3DXz0o6joZjT26Y5WNA+1Z+pQFpyhFBokqOp2jsFtB+P9b3gB601rfas3dSfvS8Bgo3MyP1nojJgVp6gCVi+B/XUs0keXPn+iYDjEi5zDbZtwxD3hA=

database name: L8I2RqYPptEtQNL4x8VhRVakSUdlsTGzEND/3TOnVTYPWe0ZnWsW0/5JdUsw9ulm

database password: EJYCSbBL8Pmf2HubIH7dHhpfDZcLyJCEGMR9jAV3apJtvFtx9TVdhUPsAxjQ2pnJ

4. Write the encrypted ciphertext to the configuration

jasypt uses ENC() package by default. At this time, the data source configuration is as follows:
spring:
  datasource:
    #   Basic configuration of data source
    username: ENC(L8I2RqYPptEtQNL4x8VhRVakSUdlsTGzEND/3TOnVTYPWe0ZnWsW0/5JdUsw9ulm)
    password: ENC(EJYCSbBL8Pmf2HubIH7dHhpfDZcLyJCEGMR9jAV3apJtvFtx9TVdhUPsAxjQ2pnJ)
    driver-class-name: com.mysql.jdbc.Driver
    url: ENC(szkFDG56WcAOzG2utv0m2aoAojJgVp6gCVi3mIAYj51Cr0vM8iYDjEi5zDbZtwxD3hA=)
    type: com.alibaba.druid.pool.DruidDataSource
The above configuration uses the default prefix=ENC(, suffix =), of course, we can change it according to our own requirements, just change it in the configuration file, as follows:
jasypt:
  encryptor:
    ## Specify prefix and suffix
    property:
      prefix: 'PASS('
      suffix: ')'
Then the configuration at this time must use PASS() to decrypt the package, as follows:
spring:
  datasource:
    #   Basic configuration of data source
    username: PASS(L8I2RqYPptEtQNL4x8VhRVakSUdlsTGzEND/3TOnVTYPWe0ZnWsW0/5JdUsw9ulm)
    password: PASS(EJYCSbBL8Pmf2HubIH7dHhpfDZcLyJCEGMR9jAV3apJtvFtx9TVdhUPsAxjQ2pnJ)
    driver-class-name: com.mysql.jdbc.Driver
    url: PASS(szkFDG56WcAOzG2utv0mUs0keXPn+pVz/uXQJ1spLUAMuXCKKrXM/6dSRnWyTtdFRost5cChEU9uRjw5M+8HU3BLemtcK0vM8iYDjEi5zDbZtwxD3hA=)
    type: com.alibaba.druid.pool.DruidDataSource

5. Summary

jasypt also has many advanced uses. For example, you can configure your own encryption algorithm. For specific operations, please refer to the documentation on Github.  

How to desensitize the data returned by the interface?

Usually, some sensitive data in the interface return value are desensitized, such as the ID number, mobile phone number, address, etc. the usual method is to hide some data by *, and of course, can also customize according to their needs. Back to business, how to achieve it gracefully? There are two implementation schemes, as follows:
  • Integrate Mybatis plug-in to desensitize specific fields during query
  • Integrate Jackson and desensitize specific fields in the serialization phase
  • Data desensitization based on Sharding Sphere
There are many ways to implement the first scheme online. The second one is to integrate Jackson.

1. Customize a Jackson annotation

You need to customize a desensitization annotation. Once an attribute is marked, desensitize it accordingly, as follows:
/**
 * Custom jackson annotation, marked on the attribute
 */
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.FIELD)
@JacksonAnnotationsInside
@JsonSerialize(using = SensitiveJsonSerializer.class)
public @interface Sensitive {
    //Desensitization strategy
    SensitiveStrategy strategy();
}

2. Customize desensitization strategy

Customize desensitization rules for different fields according to project requirements. For example, the middle digits of mobile phone number are replaced with * as follows:
/**
 * Desensitization strategy, enumeration class, and customize specific strategies for different data
 */
public enum SensitiveStrategy {
    /**
     * user name
     */
    USERNAME(s -> s.replaceAll("(\\S)\\S(\\S*)", "$1*$2")),
    /**
     * ID
     */
    ID_CARD(s -> s.replaceAll("(\\d{4})\\d{10}(\\w{4})", "$1****$2")),
    /**
     * cell-phone number
     */
    PHONE(s -> s.replaceAll("(\\d{3})\\d{4}(\\d{4})", "$1****$2")),
    /**
     * address
     */
    ADDRESS(s -> s.replaceAll("(\\S{3})\\S{2}(\\S*)\\S{2}", "$1****$2****"));


    private final Function<String, String> desensitizer;

    SensitiveStrategy(Function<String, String> desensitizer) {
        this.desensitizer = desensitizer;
    }

    public Function<String, String> desensitizer() {
        return desensitizer;
    }
}
The above only provides some information, which can be configured according to your project requirements.

3. Customize JSON serialization implementation

The following is an important implementation. Desensitize the fields marked with @ Sensitive. The implementation is as follows:
/**
 * Serialization annotation custom implementation
 * JsonSerializer<String>: Specify the String type, and the serialize() method is used to load the modified data
 */
public class SensitiveJsonSerializer extends JsonSerializer<String> implements ContextualSerializer {
    private SensitiveStrategy strategy;

    @Override
    public void serialize(String value, JsonGenerator gen, SerializerProvider serializers) throws IOException {
        gen.writeString(strategy.desensitizer().apply(value));
    }

    /**
     * Gets the annotation property on the property
     */
    @Override
    public JsonSerializer<?> createContextual(SerializerProvider prov, BeanProperty property) throws JsonMappingException {

        Sensitive annotation = property.getAnnotation(Sensitive.class);
        if (Objects.nonNull(annotation)&&Objects.equals(String.class, property.getType().getRawClass())) {
            this.strategy = annotation.strategy();
            return this;
        }
        return prov.findValueSerializer(property.getType(), property);

    }
}

4. Define the Person class and desensitize its data

The @ Sensitive annotation is used for data desensitization, and the code is as follows:
@Data
public class Person {
    /**
     * Real name
     */
    @Sensitive(strategy = SensitiveStrategy.USERNAME)
    private String realName;
    /**
     * address
     */
    @Sensitive(strategy = SensitiveStrategy.ADDRESS)
    private String address;
    /**
     * Telephone number
     */
    @Sensitive(strategy = SensitiveStrategy.PHONE)
    private String phoneNumber;
    /**
     * ID card No.
     */
    @Sensitive(strategy = SensitiveStrategy.ID_CARD)
    private String idCard;
}

5. Analog interface test

The Jackson annotation of data desensitization is completed in the above four steps, and a controller is written below for testing. The code is as follows:
@RestController
public class TestController {
    @GetMapping("/test")
    public Person test(){
        Person user = new Person();
        user.setRealName("No, Chen");
        user.setPhoneNumber("19796328206");
        user.setAddress("Wenzhou City, Hangzhou City, Zhejiang Province....");
        user.setIdCard("4333333333334334333");
        return user;
    }
}
Call the interface to check whether the data is desensitized normally. The results are as follows:
{
    "realName": "no*Chen Mou",
    "address": "Zhejiang Province****Wenzhou City..****",
    "phoneNumber": "197****8206",
    "idCard": "4333****34333"
}

6. Summary

There are many ways to implement data desensitization. The key is which is more suitable and which is more elegant  

How do log files desensitize data?

The data desensitization of the configuration file and the return value of the interface has been described above. Now it's time to desensitize the log. The print log is always unavoidable in the project. It will definitely involve some sensitive data being printed in clear text. At that time, these sensitive data (ID card, number, user name...) need to be filtered out.
Let's take log4j2 as an example to explain how to desensitize the log. The idea of other log frameworks is roughly the same.
1. Add log4j2 log dependency
The default logging framework of Spring Boot is logback, but we can switch to log4j2. The dependencies are as follows:
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
    <!-- Remove springboot Default configuration -->
    <exclusions>
        <exclusion>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-logging</artifactId>
        </exclusion>
    </exclusions>
</dependency>
<!--use log4j2 replace LogBack-->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-log4j2</artifactId>
</dependency>

2. Create a new log4j2.xml configuration in the / resource directory

The log configuration of log4j2 is very simple. You only need to create a new log4j2.xml configuration file in the / resource folder, as shown in the following figure:  
 
The above configuration does not realize data desensitization. It is a normal configuration and uses PatternLayout

3. Customize PatternLayout to realize data desensitization

The configuration in step 2 uses the format of the PatternLayout log, so we can also customize a PatternLayout to filter and desensitize the log. The inheritance relationship of the class diagram of PatternLayout is as follows:  
It can be clearly seen from the above figure that PatternLayout inherits an abstract class AbstractStringLayout. Therefore, if you want to customize, you only need to inherit this abstract class.
1. Create CustomPatternLayout and inherit the abstract class AbstractStringLayout
The code is as follows:
/**
 * log4j2 Desensitization plug-in
 * Inherit AbstractStringLayout
 **/
@Plugin(name = "CustomPatternLayout", category = Node.CATEGORY, elementType = Layout.ELEMENT_TYPE, printObject = true)
public class CustomPatternLayout extends AbstractStringLayout {


    public final static Logger logger = LoggerFactory.getLogger(CustomPatternLayout.class);
    private PatternLayout patternLayout;


    protected CustomPatternLayout(Charset charset, String pattern) {
        super(charset);
        patternLayout = PatternLayout.newBuilder().withPattern(pattern).build();
        initRule();
    }

    /**
     * Regular expression map to match
     */
    private static Map<String, Pattern> REG_PATTERN_MAP = new HashMap<>();
    private static Map<String, String> KEY_REG_MAP = new HashMap<>();


    private void initRule() {
        try {
            if (MapUtils.isEmpty(Log4j2Rule.regularMap)) {
                return;
            }
            Log4j2Rule.regularMap.forEach((a, b) -> {
                if (StringUtils.isNotBlank(a)) {
                    Map<String, String> collect = Arrays.stream(a.split(",")).collect(Collectors.toMap(c -> c, w -> b, (key1, key2) -> key1));
                    KEY_REG_MAP.putAll(collect);
                }
                Pattern compile = Pattern.compile(b);
                REG_PATTERN_MAP.put(b, compile);
            });

        } catch (Exception e) {
            logger.info(">>>>>> Failed to initialize log desensitization rule ERROR: {}", e);
        }

    }

    /**
     * Process log information for desensitization
     * 1.Determine whether the desensitization field has been configured in the configuration file
     * 2.Determine whether the content has sensitive information that needs desensitization
     * 2.1 No desensitization information needs to be returned directly
     * 2.2 Processing: ID card, name, mobile phone number, sensitive information
     */
    public String hideMarkLog(String logStr) {
        try {
            //1.Determine whether the desensitization field has been configured in the configuration file
            if (StringUtils.isBlank(logStr) || MapUtils.isEmpty(KEY_REG_MAP) || MapUtils.isEmpty(REG_PATTERN_MAP)) {
                return logStr;
            }
            //2.Determine whether the content has sensitive information that needs desensitization
            Set<String> charKeys = KEY_REG_MAP.keySet();
            for (String key : charKeys) {
                if (logStr.contains(key)) {
                    String regExp = KEY_REG_MAP.get(key);
                    logStr = matchingAndEncrypt(logStr, regExp, key);
                }
            }
            return logStr;
        } catch (Exception e) {
            logger.info(">>>>>>>>> Abnormal desensitization processing ERROR:{}", e);
            //If an exception is thrown, the original information is returned directly in order not to affect the process
            return logStr;
        }
    }

    /**
     * Regular matching the corresponding object.
     *
     * @param msg
     * @param regExp
     * @return
     */
    private static String matchingAndEncrypt(String msg, String regExp, String key) {
        Pattern pattern = REG_PATTERN_MAP.get(regExp);
        if (pattern == null) {
            logger.info(">>> logger No matching regular expression ");
            return msg;
        }
        Matcher matcher = pattern.matcher(msg);
        int length = key.length() + 5;
        boolean contains = Log4j2Rule.USER_NAME_STR.contains(key);
        String hiddenStr = "";
        while (matcher.find()) {
            String originStr = matcher.group();
            if (contains) {
                // The distance between the calculated keywords and the words requiring desensitization is less than 5.
                int i = msg.indexOf(originStr);
                if (i < 0) {
                    continue;
                }
                int span = i - length;
                int startIndex = span >= 0 ? span : 0;
                String substring = msg.substring(startIndex, i);
                if (StringUtils.isBlank(substring) ||  !substring.contains(key)) {
                    continue;
                }
                hiddenStr = hideMarkStr(originStr);
                msg = msg.replace(originStr, hiddenStr);
            } else {
                hiddenStr = hideMarkStr(originStr);
                msg = msg.replace(originStr, hiddenStr);
            }

        }
        return msg;
    }

    /**
     * Mark sensitive text rules
     *
     * @param needHideMark
     * @return
     */
    private static String hideMarkStr(String needHideMark) {
        if (StringUtils.isBlank(needHideMark)) {
            return "";
        }
        int startSize = 0, endSize = 0, mark = 0, length = needHideMark.length();

        StringBuffer hideRegBuffer = new StringBuffer("(\\S{");
        StringBuffer replaceSb = new StringBuffer("$1");

        if (length > 4) {
            int i = length / 3;
            startSize = i;
            endSize = i;
        } else {
            startSize = 1;
            endSize = 0;
        }

        mark = length - startSize - endSize;
        for (int i = 0; i < mark; i++) {
            replaceSb.append("*");
        }
        hideRegBuffer.append(startSize).append("})\\S*(\\S{").append(endSize).append("})");
        replaceSb.append("$2");
        needHideMark = needHideMark.replaceAll(hideRegBuffer.toString(), replaceSb.toString());
        return needHideMark;
    }


    /**
     * Create plug-in
     */
    @PluginFactory
    public static Layout createLayout(@PluginAttribute(value = "pattern") final String pattern,
                                      @PluginAttribute(value = "charset") final Charset charset) {
        return new CustomPatternLayout(charset, pattern);
    }


    @Override
    public String toSerializable(LogEvent event) {
        return hideMarkLog(patternLayout.toSerializable(event));
    }

}
For some details, such as @ Plugin and @ PluginFactory, what do they mean? How log4j2 implements a custom plug-in will not be described in detail here. It is not the focus of this article. If you are interested, you can check the official documentation of log4j2.

2. Customize your own desensitization rules

Log4j2Rule in the above code is a static class of desensitization rules, which I have configured directly in the static class. It can be set in the configuration file in the actual project. The code is as follows:
/**
 * There are three types of logs to intercept encryption:
 * 1,ID
 * 2,full name
 * 3,ID number
 * The encryption rules can be optimized in the configuration file later
 **/
public class Log4j2Rule {

    /**
     * Regular matching keyword category
     */
    public static Map<String, String> regularMap = new HashMap<>();
    /**
     * TODO  Configurable
     * This item can be later placed in the configuration item
     */
    public static final String USER_NAME_STR = "Name,name,contacts,full name";
    public static final String USER_IDCARD_STR = "empCard,idCard,ID,Certificate No";
    public static final String USER_PHONE_STR = "mobile,Phone,phone,Telephone,mobile phone";

    /**
     * Regular matching, customized according to business requirements
     */
    private static String IDCARD_REGEXP = "(\\d{17}[0-9Xx]|\\d{14}[0-9Xx])";
    private static String USERNAME_REGEXP = "[\\u4e00-\\u9fa5]{2,4}";
    private static String PHONE_REGEXP = "(?<!\\d)(?:(?:1[3456789]\\d{9})|(?:861[356789]\\d{9}))(?!\\d)";

    static {
        regularMap.put(USER_NAME_STR, USERNAME_REGEXP);
        regularMap.put(USER_IDCARD_STR, IDCARD_REGEXP);
        regularMap.put(USER_PHONE_STR, PHONE_REGEXP);
    }

}
After the above two steps, the customized PatternLayout has been completed. The following is to rewrite the log4j2.xml configuration file.

4. Modify log4j2.xml configuration file

In fact, the modification here is very simple. The original configuration file directly uses PatternLayout to format the log, so you only need to replace the default node with, as shown in the following figure:  
You can directly replace it globally. At this point, the configuration file has been modified.

5. Demonstration effect

In step 3, we desensitized the desensitization rule static class Log4j2Rule, which defines the three desensitization rules, including name, ID card and number.  
The following is a demonstration of whether the three rules can be desensitized correctly and printed directly using the log. The code is as follows:
@Test
public void test3(){
    log.debug("ID{},full name:{},Telephone:{}","320829112334566767","No, Chen","19896327106");
}
The logs printed on the console are as follows:
ID card: 320829******566767,Name: no***,Tel: 198*****106

Posted on Wed, 24 Nov 2021 22:09:23 -0500 by luked23