golang API request queue

• outline
• Realization ideas
• usage method
  • Start queue service
  • Use queue service

In order to cope with this speed limit, a simple request queue is used to control the access speed. After that, HTTP 429 has not been encountered

First, each request is wrapped into a struct of RequestParam, which contains the address, type, parameter and channel of response

When sending a request, just put the RequestParam in the request queue. After the request is completed, put the response in the corresponding channel

The whole code implementation is simple:

1 package util 2 3 import ( 4 "fmt" 5 6 apiclient "gitee.com/wangyubin/gutils/api_client" 7 "gitee.com/wangyubin/gutils/logger" 8 ) 9 10 // What is included in the request 11 type RequestParam struct { 12 Api string 13 Method string 14 JsonReq interface{} 15 Resp chan []byte 16 } 17 18 // Request queue is essentially a channel 19 type RequestQueue struct { 20 Queue chan RequestParam 21 } 22 23 var queue *RequestQueue 24 25 // Get queue 26 func GetQueue() *RequestQueue { 27 return queue 28 } 29 30 // Initialize queue 31 func InitRequestQueue(size int) { 32 queue = &RequestQueue{ 33 Queue: make(chan RequestParam, size), 34 } 35 } 36 37 // Put the request on the queue 38 func (rq *RequestQueue) Enqueue(p RequestParam) { 39 rq.Queue <- p 40 } 41 42 // Request queue service, waiting to accept and process requests 43 func (rq *RequestQueue) Run() { 44 lg := logger.GetLogger() 45 for p := range rq.Queue { 46 var resp []byte 47 var err error 48 switch p.Method { 49 case "GET": 50 resp, err = apiclient.GetJson(p.Api, p.JsonReq) 51 case "POST": 52 resp, err = apiclient.PostJson(p.Api, p.JsonReq) 53 default: 54 err = fmt.Errorf("Wrong type of METHOD(%s)\n", p.Method) 55 } 56 57 if err != nil { 58 lg.Err(err).Msg("access api error: " + p.Api) 59 continue 60 } 61 if p.Resp != nil { 62 p.Resp <- resp 63 close(p.Resp) 64 } 65 } 66 67 lg.Info().Msg("request queue finished!") 68 }

The request here is an apiclient encapsulated by myself, which can be replaced according to the actual situation

In my application scenario, as long as the api is accessed in sequence, HTTP 429 will not appear. If this is too fast, you can try to add some time intervals to the Run() function

1 func (rq *RequestQueue) Run() { 2 lg := logger.GetLogger() 3 for p := range rq.Queue { 4 time.Sleep(1 * time.Second) 5 // ... omitted code 6 } 7 8 lg.Info().Msg("request queue finished!") 9 }

It's easy to use. Start it first, and then put RequestParam in the queue at each call place and wait for response

Start queue service

1 func main() { 2 // init request queue and start queue service 3 util.InitRequestQueue(100) 4 queue := util.GetQueue() 5 defer close(queue.Queue) 6 go queue.Run() 7 8 // Other startup codes 9 }

Use queue service

1 func Request(param1 string, param2 int) error { 2 api := "http://xxxx.com" 3 api = fmt.Sprintf("%s?period=%s&size=%d", api, param1, param2) 4 5 queue := util.GetQueue() 6 param := util.RequestParam{ 7 Api: api, 8 Method: "GET", 9 Resp: make(chan []byte, 1), 10 } 11 queue.Enqueue(param) 12 13 var respData struct { 14 Status string `json:"status"` 15 Data []model.Data `json:"data"` 16 } 17 var err error 18 for resp := range param.Resp { 19 err = json.Unmarshal(resp, &respData) 20 if err != nil { 21 lg.Err(err).Msg("unmarshal json error") 22 return err 23 } 24 } 25 26 fmt.Println(respData) 27 return err 28 }