In the last iteration, I participated in the development of the company's data application platform. One of the functions I was responsible for completed the coding work early, and I am about to enter the testing phase. Because I have time to think about and summarize the difficulties encountered in coding, I want to refactor the code: One of the optimized functions is to collect sensitive fields about the data platform.
Functional description: Collection of data platform sensitive fields:
Provides a service method to query whether a scanned table is required for trigger collection, specifies a specific instance and a library table, and randomly takes N rows (1~max(id));a. Regular matching of the values of each field in each row (taking values that are not null or empty)
b. See if the key of the sensitive field is included in the value of each field in each row
c. Match each field name for each row; if it matches, add it to secret_column to determine if the field is sensitive or suspected sensitive
Beginning version:
/** * Collection of sensitive fields * * @param instance * @param schema */ public void collectSecretColumn(String instance, String schema, String table) { //Query if the table has been scanned CollectedTable collectedTable = collectedTableService.getCollectedTable(instance, schema, table); if (collectedTable != null) { return; } //Randomly get n rows of records JdbcResult query = getPartQueryResult(instance, schema, table); if (query == null || (query != null && StringUtils.isNotBlank(query.getQueryErrorMsg()))) { throw new CjjServerException(500, "System busy,please try again later"); } //key is a set of column value s Map<String, List<String>> groupMap = convertListToMap(query.getResult()); Set<Map.Entry<String, List<String>>> entries = groupMap.entrySet(); List<SecretContainedKey> secretContainedKeys = secretContainedKeyMapper.listSecretContainedKeys(); List<SecretValueRegex> secretValueRegexs = secretValueRegexService.getSecretValueRegexes(); for (Map.Entry<String, List<String>> entry : entries) { //Get column String column = entry.getKey(); List<String> values = entry.getValue(); //Determine if the field already exists in the sensitive fields table boolean secretColumnExist = isSecretColumnExist(instance, schema, table, column); if (secretColumnExist) { continue; } //c:Match field names boolean isValueContaninsKey = secretContainedKeyService.columnKeyIsContainsKey(instance, schema, table, secretContainedKeys, column); if (isValueContaninsKey) { continue; } //b: Does the value of the field contain key s for sensitive fields boolean isContainsKey = secretContainedKeyService.columnValueIsContainsKey(instance, schema, table, secretContainedKeys, column, values); if (isContainsKey) { continue; } //a:By regularly matching field values secretValueRegexService.regexMatch(instance, schema, table, column, values, secretValueRegexs); } CollectedTable collected = CollectedTable .builder() .instanceName(instance) .schemaName(schema) .tableName(table) .build(); collectedTableMapper.save(collected); }
You can see that logic is scattered in the for loop
Through the chain of responsibility model: descendant codes:
/** * Collection of sensitive fields * * @param instance * @param schema */ public void collectSecretColumn(String instance, String schema, String table) { //Query if the table has been scanned CollectedTable collectedTable = collectedTableService.getCollectedTable(instance, schema, table); if (collectedTable != null) { return; } //Randomly get n rows of records JdbcResult query = getPartQueryResult(instance, schema, table); if (query == null || (query != null && StringUtils.isNotBlank(query.getQueryErrorMsg()))) { throw new CjjServerException(500, "System busy,please try again later"); } //key is a set of column value s Map<String, List<String>> groupMap = convertListToMap(query.getResult()); Set<Map.Entry<String, List<String>>> entries = groupMap.entrySet(); secretValueRegexHandler.setSuccessor(secretValueContainedKeyHandler); secretValueContainedKeyHandler.setSuccessor(secretColumnContainedKeyHandler); for (Map.Entry<String, List<String>> entry : entries) { //Get column String column = entry.getKey(); List<String> values = entry.getValue(); //Determine if the field already exists in the sensitive fields table boolean secretColumnExist = isSecretColumnExist(instance, schema, table, column); if (secretColumnExist) { continue; } secretValueRegexHandler.handleCollect(instance, schema, table, column, values); } CollectedTable collected = CollectedTable .builder() .instanceName(instance) .schemaName(schema) .tableName(table) .build(); collectedTableMapper.save(collected); }
You can see that there's less code on this side and the structure looks clearer
For ease of understanding: I'll list some codes for your reference
package cn.caijiajia.firekylin.service.secret; import java.util.List; /** * Responsibility Chain Design Mode * * @author chenlang * date 2018/7/13 */ public abstract class CollectSecretColumnHandler { protected CollectSecretColumnHandler successor; public abstract void handleCollect(String instance, String schema, String table, String column, List<String> values); /** * Obtain the object of responsibility */ public CollectSecretColumnHandler getSuccessor() { return successor; } /** * Set up successive objects of responsibility */ public void setSuccessor(CollectSecretColumnHandler successor) { this.successor = successor; } }
package cn.caijiajia.firekylin.service.secret; import cn.caijiajia.firekylin.domain.SecretContainedKey; import cn.caijiajia.firekylin.mapper.SecretContainedKeyMapper; import cn.caijiajia.firekylin.service.SecretColumnService; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Component; import java.util.List; import java.util.Map; import java.util.Set; import java.util.stream.Collectors; /** * @author chenlang * date 2018/7/13 */ @Component public class SecretColumnContainedKeyHandler extends CollectSecretColumnHandler { @Autowired private SecretColumnService secretColumnService; @Autowired private SecretContainedKeyMapper secretContainedKeyMapper; @Override public void handleCollect(String instance, String schema, String table, String column, List<String> values) { List<SecretContainedKey> secretContainedKeys = secretContainedKeyMapper.listSecretContainedKeys(); boolean columnKeyIsContainsKey = columnKeyIsContainsKey(instance, schema, table, secretContainedKeys, column); if (!columnKeyIsContainsKey) { } } public boolean columnKeyIsContainsKey(String instance, String schema, String table, List<SecretContainedKey> secretContainedKeys, String column) { SecretContainedKey secretContainedKeyByColumn = getSecretContainedKeyByColumn(column, secretContainedKeys); if (secretContainedKeyByColumn != null) { secretColumnService.saveSecretColumn(instance, schema, table, column, secretContainedKeyByColumn.getSecretType(), secretContainedKeyByColumn.getColumnType()); return true; } return false; } /** * Does the field name contain a sensitive key * * @param column * @param secretContainedKeys * @return */ public SecretContainedKey getSecretContainedKeyByColumn(String column, List<SecretContainedKey> secretContainedKeys) { Map<String, SecretContainedKey> keysMap = secretContainedKeys.stream().collect(Collectors.toMap(SecretContainedKey::getContainedKey, a -> a)); Set<Map.Entry<String, SecretContainedKey>> entries = keysMap.entrySet(); for (Map.Entry<String, SecretContainedKey> entry : entries) { String key = entry.getKey(); boolean contains = column.toLowerCase().contains(key); if (contains) { return keysMap.get(key); } } return null; } }
package cn.caijiajia.firekylin.service.secret; import cn.caijiajia.firekylin.domain.SecretContainedKey; import cn.caijiajia.firekylin.mapper.SecretContainedKeyMapper; import cn.caijiajia.firekylin.service.SecretColumnService; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Component; import java.util.List; /** * @author chenlang * date 2018/7/13 */ @Component public class SecretValueContainedKeyHandler extends CollectSecretColumnHandler { @Autowired private SecretColumnService secretColumnService; @Autowired private SecretContainedKeyMapper secretContainedKeyMapper; @Override public void handleCollect(String instance, String schema, String table, String column, List<String> values) { List<SecretContainedKey> secretContainedKeys = secretContainedKeyMapper.listSecretContainedKeys(); boolean columnValueIsContainsKey = columnValueIsContainsKey(instance, schema, table, secretContainedKeys, column, values); if (!columnValueIsContainsKey) { getSuccessor().handleCollect(instance, schema, table, column, values); } } public boolean columnValueIsContainsKey(String instance, String schema, String table, List<SecretContainedKey> secretContainedKeys, String column, List<String> values) { for (SecretContainedKey secretContainedKey : secretContainedKeys) { boolean isSecretColumnContainsKey = isSecretColumnContainsKey(values, secretContainedKey); if (isSecretColumnContainsKey) { secretColumnService.saveSecretColumn(instance, schema, table, column, secretContainedKey.getSecretType(), secretContainedKey.getColumnType()); return true; } } return false; } /** * Does the field value contain key s for sensitive fields * * @param columnValues * @param secretContainedKey * @return */ public boolean isSecretColumnContainsKey(List<String> columnValues, SecretContainedKey secretContainedKey) { for (String columnValue : columnValues) { if (columnValue.toLowerCase().contains(secretContainedKey.getContainedKey())) { return true; } } return false; } }
package cn.caijiajia.firekylin.service.secret; import cn.caijiajia.firekylin.constant.SecretType; import cn.caijiajia.firekylin.domain.SecretValueRegex; import cn.caijiajia.firekylin.service.SecretColumnService; import cn.caijiajia.firekylin.service.SecretValueRegexService; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Component; import org.springframework.util.CollectionUtils; import java.util.List; import java.util.regex.Pattern; /** * Regular Matching Collects Sensitive Fields * * @author chenlang * date 2018/7/13 */ @Component public class SecretValueRegexHandler extends CollectSecretColumnHandler { @Autowired private SecretColumnService secretColumnService; @Autowired private SecretValueRegexService secretValueRegexService; @Override public void handleCollect(String instance, String schema, String table, String column, List<String> values) { List<SecretValueRegex> secretValueRegexs = secretValueRegexService.getSecretValueRegexes(); boolean regexMatch = regexMatch(instance, schema, table, column, values, secretValueRegexs); if (!regexMatch) { if (getSuccessor() != null) { getSuccessor().handleCollect(instance, schema, table, column, values); } } } public boolean regexMatch(String instance, String schema, String table, String column, List<String> values, List<SecretValueRegex> secretValueRegexs) { for (SecretValueRegex secretValueRegex : secretValueRegexs) { boolean secretByRegex = isSecretByRegex(values, secretValueRegex.getPattern()); if (secretByRegex) { secretColumnService.saveSecretColumn(instance, schema, table, column, SecretType.SECRECT, secretValueRegex.getCode()); return true; } } return false; } /** * Does the field value match the regular expression * * @param columnValues * @return */ public boolean isSecretByRegex(List<String> columnValues, Pattern compile) { if (CollectionUtils.isEmpty(columnValues)) { return false; } for (String columnValue : columnValues) { boolean isSecret = compile.matcher(columnValue).matches(); if (!isSecret) { return false; } } return true; } }
Now each case corresponds to a handler and inherits from
CollectSecretColumnHandler