json's past life and present life

JSON, full name: JavaScript Object Notation, as a common lightweight data exchange format, should be frequently touched in a programmer's development career. The simple and clear hierarchy makes JSON an ideal data exchange language. It is easy for people to read and write, but also easy for machines to parse and generate, and effectively improve the efficiency of network transmission.

Java is an object-oriented language, so we deal with business in the form of objects more often in projects, but we need to convert objects into JSON format for transmission, and JSON format can generally be parsed into most of the object formats, regardless of the programming language.

In our work, we inevitably use json strings. json has become the most commonly used data format for our resultful interface. I'm sure you're not new to it. We've used it more or less in our work. Today, I'm going to talk about some of the pits we used in json data processing, so as to avoid ourselves or you falling into this trap in the future, so I'll summarize it, Share with you, hope to help you

OK, let's get to the point. When you are dealing with json strings, what are the most commonly used packages and methods?

1. fastjson

1. What is fastjson

The official definition of Alibaba is that fastjson is Alibaba's open-source JSON parsing library, which can parse strings in JSON format, support serializing Java beans into JSON strings, and can also deserialize from JSON strings to JavaBean s.

2. Advantages of fastjson

  • Fast
    fastjson is faster than other JSON libraries. Since the release of 1.1.x in 2011, its performance has never been surpassed by other Java implemented JSON libraries.
  • Widely used
    fastjson is widely used in Alibaba and deployed on tens of thousands of servers. fastjson is widely accepted in the industry. In 2012, it was selected as one of the most popular domestic open source software by open source China.
  • Test completion
    fastjson has a large number of testcases. In version 1.2.11, there are more than 3321 testcases. Regression test will be carried out for each release to ensure stable quality.
  • Easy to use
    fastjson's API is very simple.
  • Complete function
    It supports generics, stream processing, enumeration, serialization and deserialization.

3. Get fastjson

You can download fastjson from:

The latest version of fastjson will be released to maven's central repository, which you can rely on directly.

dependency>
     <groupId>com.alibaba</groupId>
     <artifactId>fastjson</artifactId>
     <version>x.x.x</version>
</dependency>

Where x.x.x is the version number, use the specific version as required, and the latest version is recommended.

fastjson vulnerability

Vulnerability description

FastJson, a common JSON component, has a remote code execution vulnerability. The * * can execute arbitrary commands on the target server through the carefully constructed JSON message, so as to obtain the server permission. This vulnerability is bypassed by autoType in the previous vulnerability.

Scope of influence

FastJson < 1.2.48

Vulnerability description

Using the 0day vulnerability, a malicious person can construct a blacklist policy that requests to bypass FastJSON. For example, the * * person remotely asks the server to execute the specified command through a carefully constructed request (the calculator program runs successfully in the following example).

4. Main API of fastjson

The Fastjson entry class is com.alibaba.fastjson.JSON, and the main API s are JSON.toJSONString and parseObject.

package com.alibaba.fastjson;
public abstract class JSON {
      // Convert Java objects to JSON strings
      public static final String toJSONString(Object object);
      //JSON string to Java object
      public static final <T> T parseObject(String text, Class<T> clazz, Feature... features);
}

Serialization:

String jsonString = JSON.toJSONString(obj);

De serialization

VO vo = JSON.parseObject("...", VO.class);

5. Fastjson performance

fastjson is the fastest json Library in the java language at present, faster than the fastest jackson claimed. See the independent test results of the third party here: https://github.com/eishay/jvm-serializers/wiki.

When you do performance test by yourself, you need to turn off the function of circular reference detection.

In addition, Fastjson is about six times faster than Gson

So in the work, my colleagues or many students are using fastjson

2. @JsonProperty

OK, the above operations are enough for you to use fastjson, but if we encounter a more complex example, how should we parse it?

For example, if I give a string, let's see how to parse it?

{
  "properties": {
    "sonar.analysis.buildNumber": "1555"
  }
}

If you come across this example, what should we do? Because sonar.analysis.buildNumber doesn't conform to the naming method of our java properties, so we can't directly use properties to assign values to it, what should we do?
We found many answers on the Internet. We can use @ JsonProperty to assign and set annotation properties

import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.annotation.JsonProperty;
import io.swagger.annotations.ApiModel;
import io.swagger.annotations.ApiModelProperty;
import lombok.Data;

import java.util.ArrayList;

@Data
@JsonIgnoreProperties(ignoreUnknown = true)
@ApiModel(value = "sonarQube webhook Submission conditions of, only applicable to the current version")
public class ProviderSonarWebhookCondition {

    @Data
    @JsonIgnoreProperties(ignoreUnknown = true)
    public static class SonarProperties {
        @ApiModelProperty(value = "jenkins Launch sonar Scan, need to accept build_id")
        @JsonProperty(value = "sonar.analysis.buildNumber")
        private String buildNumber;
    }

    @ApiModelProperty(value = "attribute")
    private SonarProperties properties;

}

Let's take a look at the function of these notes:

  • @The JsonIgnore annotation is used to ignore some fields. It can be used in variables or Getter methods. When it is used in Setter methods, it has the same effect as variables. This annotation is generally used for the fields we want to ignore.
  • @JsonIgnoreProperties(ignoreUnknown = true). After this annotation is written on the class, fields that do not exist in the class will be ignored. This annotation can also specify the fields to be ignored, such as @ JsonIgnoreProperties({"password", "secretKey"}), password and secretKey are our java member variables

Well, the above notes help me to solve this problem smoothly, and automatically convert sonar.analysis.buildNumber to buildNumber for me. Well, it solves my problem very well

3. Here comes the question

When I was writing a one-sided instance, I suddenly found a problem and found that the buildNumber conversion was not successful

In fact, we found: @ JsonIgnoreProperties is a jackson annotation used, but when we use string conversion, we use Fastjson, so it is not compatible, especially in unit testing,

maven dependence of jackson

<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.5.3</version>
</dependency>

@JsonProperty this annotation is used to serialize the name of the property to another name, such as the trueName property to name, @ JsonProperty(value="name").

import com.fasterxml.jackson.annotation.JsonProperty;

public class Student {

    @JsonProperty(value = "real_name")
    private String realName;

    public String getRealName() {
        return realName;
    }

    public void setRealName(String realName) {
        this.realName = realName;
    }

    @Override
    public String toString() {
        return "Student{" +
                "realName='" + realName + '\'' +
                '}';
    }
}

test

import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;

public class Main {
    public static void main(String[] args) throws JsonProcessingException {
        Student student = new Student();
        student.setRealName("zhangsan");
        System.out.println(new ObjectMapper().writeValueAsString(student));
    }
}

Result:

{"real_name":"zhangsan"}

*Note that the method used to convert objects to json strings is provided by fasterxml.jackson!! *

What if you use fastjson?

import com.alibaba.fastjson.JSON;

public class Main {
    public static void main(String[] args) {
        Student student = new Student();
        student.setRealName("zhangsan");
        System.out.println(JSON.toJSONString(student));
    }
}

Result:

{"realName":"zhangsan"}

As you can see, @ jsonproperty (value = "real name") has not taken effect. Why?

*Because fastjson doesn't know the @ JsonProperty annotation! So use jackson's own serialization tools! *

@JsonProperty is not only useful when serializing, but also useful when deserializing. For example, some interfaces return json strings, and naming is not the standard hump form. When mapping to objects, add @ JsonProperty annotation to the class properties, and write the corresponding name of the returned json string

import com.fasterxml.jackson.databind.ObjectMapper;

import java.io.IOException;

public class Main {
    public static void main(String[] args) throws IOException {
        String jsonStr = "{\"real_name\":\"zhangsan\"}";
        Student student = new ObjectMapper().readValue(jsonStr.getBytes(), Student.class);
        System.out.println(student);
    }
}

Result:

Student{realName='zhangsan'}

4. Thinking is learning

Since fastjson is known as the most powerful json parsing tool, it's impossible not to consider the problem of this special string. If fastjson is developed by ourselves, we should also consider these situations. Big guys can't drive worse than big ones. So maybe we haven't mastered the use method of fastjson better. Let's learn about fastjson in promotion, As expected, this operation is still supported. Come to discount our code

6. fastjson custom serialization

1. introduction

fastjson supports multiple ways of custom serialization

  • Customize serialization through @ JSONField
  • Customize serialization through @ JSONType
  • Customize serialization through @ SerializeFilter
  • Customizing serialization through ParseProcess

2. Use @ JSONField configuration

1. Introduction to jsonfield annotation

package com.alibaba.fastjson.annotation;

public @interface JSONField {
    // Configure the sequence of serialization and deserialization, which is only supported after version 1.1.42
    int ordinal() default 0;

     // Specify the name of the field
    String name() default "";

    // Specifies the format of the field, useful for date formats
    String format() default "";

    // Serialization or not
    boolean serialize() default true;

    // Deserialization or not
    boolean deserialize() default true;
}

2. JSONField configuration mode

You can configure @ JSONField on fields or getter/setter methods, for example:

Configure on field

public class VO {
     @JSONField(name="ID")
     private int id;

     @JSONField(name="birthday",format="yyyy-MM-dd")
     public Date date;
}

Configured on Getter/Setter**

public class VO {
    private int id;

    @JSONField(name="ID")
    public int getId() { return id;}

    @JSONField(name="ID")
    public void setId(int id) {this.id = id;}
}

3. Use format to configure date format

You can customize the format of each date field

public class A {
      // Configure date serialization and deserialization to use yyyyMMdd date format
      @JSONField(format="yyyyMMdd")
      public Date date;
 }

4. Use serialize/deserialize to specify that the field is not serialized

public class A {
      @JSONField(serialize=false)
      public Date date;
 }

 public class A {
      @JSONField(deserialize=false)
      public Date date;
 }

5. Use ordinal to specify the order of fields

By default, Fastjson serializes a java bean according to the alphabetical order of fieldName. You can specify the order of fields through coordinate. This feature requires version 1.1.42 or higher.

public static class VO {
    @JSONField(ordinal = 3)
    private int f0;

    @JSONField(ordinal = 2)
    private int f1;

    @JSONField(ordinal = 1)
    private int f2;
}

6. Use serializeUsing to specify the serialization class of the property

At fa

After stjson version 1.2.16, JSONField supports the new customized configuration of serializeUsing, which can customize and serialize a property of a class separately, such as:

public static class Model {
    @JSONField(serializeUsing = ModelValueSerializer.class)
    public int value;
}

public static class ModelValueSerializer implements ObjectSerializer {
    @Override
    public void write(JSONSerializer serializer, Object object, Object fieldName, Type fieldType,
                      int features) throws IOException {
        Integer value = (Integer) object;
        String text = value + "element";
        serializer.write(text);
    }
}

Test code

Model model = new Model();
model.value = 100;
String json = JSON.toJSONString(model);
Assert.assertEquals("{\"value\":\"100 element\"}", json);

3. Use @ JSONType configuration

Similar to JSONField, but JSONType is configured on a class rather than a field or getter/setter method.

4. Customize serialization through SerializeFilter

1. introduction

SerializeFilter is customized serialization through programming extension. fastjson supports 6 kinds of serializefilters, which are used for customized serialization without scenarios

  • PropertyPreFilter determines whether to serialize according to PropertyName
  • PropertyFilter determines whether to serialize according to PropertyName and PropertyValue
  • NameFilter modifies the Key. If the Key needs to be modified, the process return value can
  • ValueFilter modify Value
  • BeforeFilter add content first when serializing
  • Add content at the end of AfterFilter serialization

2. PropertyFilter determines whether to serialize according to PropertyName and PropertyValue

public interface PropertyFilter extends SerializeFilter {
    boolean apply(Object object, String propertyName, Object propertyValue);
 }

You can judge whether to serialize according to the object or property name or property value through extension implementation. For example:

PropertyFilter filter = new PropertyFilter() {

    public boolean apply(Object source, String name, Object value) {
        if ("id".equals(name)) {
            int id = ((Integer) value).intValue();
            return id >= 100;
        }
        return false;
    }
};

JSON.toJSONString(obj, filter); // Pass in filter when serializing

3. PropertyPreFilter judges whether to serialize according to PropertyName

Unlike PropertyFilter, it only judges based on object and name. Before calling getter, this avoids possible exceptions in getter call.

public interface PropertyPreFilter extends SerializeFilter {
      boolean apply(JSONSerializer serializer, Object object, String name);
  }

4. Modify Key during namefilter serialization

If the key needs to be modified, the process return value can

public interface NameFilter extends SerializeFilter {
    String process(Object object, String propertyName, Object propertyValue);
}

fastjson has a built-in Pascal name filter to output the Pascal style that capitalizes the first character. For example:

import com.alibaba.fastjson.serializer.PascalNameFilter;

Object obj = ...;
String jsonStr = JSON.toJSONString(obj, new PascalNameFilter());

5. Modify Value when serializing valuefilter

public interface ValueFilter extends SerializeFilter {
  Object process(Object object, String propertyName, Object propertyValue);
}

6. Adding content at the top of beforefilter serialization

Do something before serializing all properties of an object, such as calling writeKeyValue to add content

public abstract class BeforeFilter implements SerializeFilter {
   protected final void writeKeyValue(String key, Object value) { ... }
    // An abstract method that needs to be implemented, calling writeKeyValue to add content in the implementation.
    public abstract void writeBefore(Object object);
}

7. Add content at the end of afterfilter serialization

Do something after serializing all the properties of an object, such as calling writeKeyValue to add content

public abstract class AfterFilter implements SerializeFilter {
  protected final void writeKeyValue(String key, Object value) { ... }
    // An abstract method that needs to be implemented, calling writeKeyValue to add content in the implementation.
    public abstract void writeAfter(Object object);
}

5. Custom deserialization through ParseProcess

1. introduction

ParseProcess is an interface for programming extension custom deserialization. fastjson supports the following ParseProcess:

  • ExtraProcessor is used to handle redundant fields
  • ExtraTypeProvider provides type information when used to process redundant fields

2. Use ExtraProcessor to process redundant fields

public static class VO {
    private int id;
    private Map<String, Object> attributes = new HashMap<String, Object>();
    public int getId() { return id; }
    public void setId(int id) { this.id = id;}
    public Map<String, Object> getAttributes() { return attributes;}
}

ExtraProcessor processor = new ExtraProcessor() {
    public void processExtra(Object object, String key, Object value) {
        VO vo = (VO) object;
        vo.getAttributes().put(key, value);
    }
};

VO vo = JSON.parseObject("{\"id\":123,\"name\":\"abc\"}", VO.class, processor);
Assert.assertEquals(123, vo.getId());
Assert.assertEquals("abc", vo.getAttributes().get("name"));

3. Use ExtraTypeProvider to provide types for redundant fields

public static class VO {
    private int id;
    private Map<String, Object> attributes = new HashMap<String, Object>();
    public int getId() { return id; }
    public void setId(int id) { this.id = id;}
    public Map<String, Object> getAttributes() { return attributes;}
}

class MyExtraProcessor implements ExtraProcessor, ExtraTypeProvider {
    public void processExtra(Object object, String key, Object value) {
        VO vo = (VO) object;
        vo.getAttributes().put(key, value);
    }

    public Type getExtraType(Object object, String key) {
        if ("value".equals(key)) {
            return int.class;
        }
        return null;
    }
};
ExtraProcessor processor = new MyExtraProcessor();

VO vo = JSON.parseObject("{\"id\":123,\"value\":\"123456\"}", VO.class, processor);
Assert.assertEquals(123, vo.getId());
Assert.assertEquals(123456, vo.getAttributes().get("value")); // value should have been of string type. It has been changed to Integer type through the processing of getExtraType.

I believe the above examples are enough for you to use in your work!

Author: Story Ling
Introduction: the life of the sub section hand, currently working in a real estate company to do DevOPS related work, has done in the large Internet Co as an advanced operation and maintenance engineer, familiar with Linux operation and maintenance, Python operation and maintenance development, Java development, DevOPS common development components, personal official account number: stromling, welcome to lift me!

Tags: Programming JSON Java Maven

Posted on Wed, 25 Mar 2020 22:27:29 -0400 by lonerunner