Golang practice record: use the gin framework to realize the forwarding function: upload files and transfer them

Recently, we need to implement a gadget to forward post requests to the specified back-end service. Because we always want to learn the gin framework, we use this framework to try. It is expected to produce several articles. This paper first studies how to realize the upload and forward function in gin framework.

Question raised

A background web service has many historical versions. It runs without problems, but the requirements change later. It is necessary to run different historical versions separately and specify different ports. External is equivalent to many services. There is a date and time in the request post. You need to forward the request to the back-end services of different ports according to the date and time. Note that the post request is directly in the form of a file and has specific requirements for the file name.

thinking

nginx can forward according to the port, but this article forwards according to the requested content, so we need to implement a forwarding tool. That is, first read the file content of the external request, parse the time, and then forward it to different port services according to the time. Problems to be solved:
How to resolve the post request and send the request to the back-end service intact? How to return the data returned by the back-end service to the requester intact?
How to manage back-end services? If additional scripts are used, the steps of O & M deployment are added, which is a little troublesome. Therefore, it is considered to be implemented in the forwarding tool.

realization

  • Use the tool to make a post request and specify a file name. You can use postman or curl, which is used in this article.

Forwarding function:

  • Use ctx.Request.FormFile to get the file name and file content. At this time, you can use the SaveUploadedFile function provided by gin to save the file or call io.Copy to save it. The former is easy.
  • Call the forward function again.
  • Convert the return value of the re forwarding function into json form and return to the post request tool.

Forwarding function again:

  • Create a file using multipart package and copy the file obtained in the previous step.
  • Then send the request to the http library. Note that formatting is required.
  • Finally, read the return value of the request and return it. Note that the content is in byte form.

code

Main interface code

func RunWebServer(args []string) {
    runWebOnlyPost()
}

func runWebOnlyPost() {
	router := gin.New()
	router.Use(gin.Logger())
	router.Use(gin.Recovery())

	testRouter(router)

	klog.Println("Server started at ", conf.Port)
	router.Run(":" + conf.Port)
}


func testRouter(r *gin.Engine) {
	fmt.Println("test post...")
    
    r.POST("/foobar/test", foobar_test)
    r.POST("/foobar/test_back", foobar_test_back) 
}

Implementation code

/*
 curl http://127.0.0.1:84/foobar/test -X POST -F  "file=@sample.json"
 
 Temporary:
 file After reading once, there is no content after reading again. The number of bytes is 0
*/
func foobar_test(ctx *gin.Context) {
    
    // Both methods are available, but ctx.Request.FormFile can get the file handle and can be copied directly
    //file, err := ctx.FormFile("file")
    file, header, err := ctx.Request.FormFile("file")
    if err != nil {
        ctx.JSON(http.StatusBadRequest, gin.H{
        "error": err,
        })
        return
    } 
    
    //fmt.Printf("Request: %+v\n", ctx.Request);
    //fmt.Printf("Formfile: %+v | %+v |||  %v %v\n", file, header, err, reflect.TypeOf(file));
    
    // Get the file and length, and use it later
    var jsonfilename string = header.Filename
    mysize := header.Size
    fmt.Printf("filename: %s size: %d\n", jsonfilename, mysize);
    
    if err != nil {
        ctx.JSON(http.StatusBadRequest, gin.H{
        "error": err,
        })
        return
    }

    // Processing json files 
    // Note: if the file is read and forwarded, there will be no content, so it is not read here
    
    // Service URL to the back end
    url := "http://127.0.0.1:85/foobar/test_back"
    resp := post_data_gin(url, jsonfilename, file);

    // Parse the returned character slice, get the map, treat it as json, and assign it to gin
    var data1 map[string]interface{}
    err = json.Unmarshal(resp, &data1)
    //fmt.Println("muti unmarshal: ", err, data1)
    
    ctx.JSON(http.StatusOK, data1)
    
    return
}

/*
 Simulate the background json that only gets the file field and does not do other processing
 curl http://127.0.0.1:84/foobar/test_back -X POST -F  "file=@sample.json"
*/
func foobar_test_back(ctx *gin.Context) {
    
    // Both methods are available, but ctx.Request.FormFile can get the file handle and can be copied directly
    //file, err := ctx.FormFile("file")
    _, header, err := ctx.Request.FormFile("file")
    if err != nil {
        ctx.JSON(
            http.StatusBadRequest,
            gin.H{
                "code": -1,
                "msg":  "failed",
                "data": gin.H{
                    "result": "failed in back end server",
                },
            },
        )

        return
    }

    // Get the file and length, and use it later
    var myfile string = header.Filename
    mysize := header.Size
    fmt.Printf("filename: %s size: %d\n", myfile, mysize);
    
    if mysize <= 0 {
        ctx.JSON(
            http.StatusBadRequest,
            gin.H{
                "code": -1,
                "msg":  "failed",
                "data": gin.H{
                    "result": "failed in back end server, json size 0",
                },
            },
        )

        return
    }
    
    // Here you can save the file

    //After saving successfully, the correct Json data will be returned
    ctx.JSON(
		http.StatusOK,
		gin.H{
			"code": 0,
			"msg":  "ok",
			"data": gin.H{
				"result": "ok in back end server",
			},
		},
	)

    return
}

For the convenience of testing, the gin framework program implemented in this paper can specify the port at run time. Therefore, the response function of two URLs is implemented in the code.

test

This article uses the sample.json file to test, as follows:

{
	"enID": "ID250",
	"exID": "ID251",
    "exTime": "2020-09-17T20:00:27",
	"type": 1,
	"money": 250.44,
	"distance": 274050
}

First run the 84 port service (called 84 service), which is an external service. Then run 85 port service (called 85 service), which is a service simulating the back end.
Start a terminal and execute the test command:

curl http://127.0.0.1:84/foobar/ -X POST -F  "file=@sample.json"

84 service printing:

[2021-08-25 23:51:19.424 busy.go:79] Server started at  84
[GIN-debug] Listening and serving HTTP on :84
filename: sample.json size: 95
io copy: 95 <nil>
[GIN] 2021/08/25 - 23:53:59 | 200 |      3.0002ms |       127.0.0.1 | POST     "/foobar/test"

85 service printing:

[GIN-debug] Listening and serving HTTP on :85
filename: sample.json size: 95
[GIN] 2021/08/25 - 23:53:59 | 200 |            0s |       127.0.0.1 | POST     "/foobar/test_back"

Test command return:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   361  100    63  100   298  31500   145k --:--:-- --:--:-- --:--:--  352k{"code":0,"data":{"result":"ok in back end server"},"msg":"ok"}

You can also request directly from the back-end service:

$ curl http://127.0.0.1:85/fee/test_back -X POST -F  "file=@sample.json"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   361  100    63  100   298  63000   291k --:--:-- --:--:-- --:--:--  352k{"code":0,"data":{"result":"ok in back end server"},"msg":"ok"}

Night on September 17, 2021

Tags: Go Operation & Maintenance

Posted on Mon, 20 Sep 2021 13:52:24 -0400 by lucie