This article has been included GitHub There are also Internet factory interview questions, interview strategies, efficient learning materials and so on

In this article, I will try to explain how to reduce data storage by more than 50%.

Our 2.5GB Redis ElastiCache is almost full, and if it reaches its limit in some way, our system will start to malfunction.Redis may become a bottleneck.

Basic settings:

With the latest version of Spring Boot, there are two main dependencies - Spring Boot Web and Spring Data Reactive Redis, which will be used to connect to and use Redis's internal applications.Essentially, Redis dependencies use the Lettuce Redis client by default and are supported by the latest version of Spring Boot.

The complete code can be found on my Github redis-util.

Reducing memory usage using MessagePack requires a "conversion" mechanism.What if translation is an expert and translates your English into as few words as possible?MessagePack works the same way!Will need to bePom.xmlMore than two dependencies were added to the file.

<dependency>
        <groupId>org.msgpack</groupId>
        <artifactId>msgpack-core</artifactId>
        <version>0.8.20</version>
    </dependency>
    <dependency>
        <groupId>org.msgpack</groupId>
        <artifactId>jackson-dataformat-msgpack</artifactId>
        <version>0.8.20</version>
    </dependency>
//We created a controller named MsgPackController:

class MsgPackRedisSerializer<T> implements RedisSerializer<T> {
    public static final Charset DEFAULT_CHARSET;
    private final JavaType javaType;
    private ObjectMapper objectMapper = new ObjectMapper(new MessagePackFactory())
            .registerModules(new Jdk8Module(), new JavaTimeModule())
            .configure(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS, true)
            .configure(SerializationFeature.FAIL_ON_EMPTY_BEANS, false)
            .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
            .setSerializationInclusion(JsonInclude.Include.NON_NULL);

    public MsgPackRedisSerializer(Class<T> type) {
        this.javaType = JavaTypeHandler.getJavaType(type);
    }

    public T deserialize(@Nullable byte[] bytes) throws SerializationException {
        if (bytes == null || bytes.length == 0) {
            return null;
        } else {
            try {
                return this.objectMapper.readValue(bytes, 0, bytes.length, this.javaType);
            } catch (Exception ex) {
                throw new SerializationException("Could not read MsgPack JSON: " + ex.getMessage(), ex);
            }
        }
    }

    public byte[] serialize(@Nullable Object value) throws SerializationException {
        if (value == null) {
            return new byte[0];
        } else {
            try {
                return this.objectMapper.writeValueAsBytes(value);
            } catch (Exception ex) {
                throw new SerializationException("Could not write MsgPack JSON: " + ex.getMessage(), ex);
            }
        }
    }

    static {
        DEFAULT_CHARSET = StandardCharsets.UTF_8;
    }

}

Instance MessagePackFactory is passed to ObjectMapper.This serves as a bridge between the binary and string formats of the data between Redis and our Spring Boot application.

Test: ComparisonDataset.bytesCurrent memory versus previously recorded memory.15976 bytes to 32840 bytes, which has been reduced by almost 50%!

But wait, we can further reduce it.Compress!Snappy!

The first problem that follows is that compression and decompression take time.Is this harmful to production?Snappy also has an answer.

It is not intended to maximize compression or be compatible with any other compression library.Instead, it aims at very high speed and reasonable compression.

Using Snappy is as easy as adding dependencies to itPom.xmlAnd several lines of code change.Just Snappy.compress In serialization and Snappy.decompress Add when deserializing.

<dependency>
        <groupId>org.xerial.snappy</groupId>
        <artifactId>snappy-java</artifactId>
        <version>1.1.7.3</version>
    </dependency>

By cleaning and reorganizing the data, and using these techniques, we reduced our memory usage from 2GB to less than 500MB.