Goang Reverse Proxy Reverseeproxy Source Analysis

1 Reverse proxy example based on reverse proxy implementation

package main

import (
    "log"
    "net/http"
    "net/http/httputil"
    "net/url"
)

func main() {
    // Address Rewrite Instance
    // http://127.0.0.1:8888/test?id=1  => http://127.0.0.1:8081/reverse/test?id=1

    rs1 := "http://127.0.0.1:8081/reverse"
    targetUrl , err := url.Parse(rs1)
    if err != nil {
        log.Fatal("err")
    }
    proxy := httputil.NewSingleHostReverseProxy(targetUrl)
    log.Println("Reverse proxy server serve at : 127.0.0.1:8888")
    if err := http.ListenAndServe(":8888",proxy);err != nil{
        log.Fatal("Start server failed,err:",err)
    }
}
$ curl http://127.0.0.1:8888/hello?id=123 -s
http://127.0.0.1:8081/reverse/hello?id=123

2 reverse proxy source analysis

reverseproxy

// Process incoming requests, send them to another server to implement reverse proxy, and pass them back to the client
type ReverseProxy struct {
    // The request can be modified through the transport, and the responder returns unchanged
    Director func(*http.Request)

    // Connection Pool Reuse Connection, used to execute requests, used by default for nil http.DefaultTransport
    Transport http.RoundTripper

    // Refresh interval to client
    // This parameter is ignored for streaming requests and all reverse proxy requests are refreshed immediately
    FlushInterval time.Duration

    // Default toStd.errCan be used to customize logger
    ErrorLog *log.Logger

    // For executionIo.CopyBufferCopy the response body and store it in byte slices
    BufferPool BufferPool

    // Used to modify response results and HTTP status codes, ErrorHandler is called when the return result error is not empty
    ModifyResponse func(*http.Response) error

    // Used to process error information returned by the endpoint and ModifyResponse, which by default returns the passed error information and returns HTTP 502
    ErrorHandler func(http.ResponseWriter, *http.Request, error)
}

Main methods

// Instantiate ReverseProxy
// Assuming the target URI(target path) is / base and the requested URI (target request) is / dir, the request will be proxied to the reverseHttp://x.x.x.x. /base/dir
// ReverseProxy does not rewrite Host header and needs to override Host, which can be customized in the Director function

func NewSingleHostReverseProxy(target *url.URL) *ReverseProxy {
  // Get the request parameters, such as/dir?id=123, then rawQuery:id=123
    targetQuery := target.RawQuery

    // Instantiate director
    director := func(req *http.Request) {
        req.URL.Scheme = target.Scheme   // http or https
        req.URL.Host = target.Host      // Host name (ip+port or domain name+port)
        req.URL.Path = singleJoiningSlash(target.Path, req.URL.Path) // Request URL Splicing

        // Use the'&'symbol to stitch request parameters
        if targetQuery == "" || req.URL.RawQuery == "" {
            req.URL.RawQuery = targetQuery + req.URL.RawQuery
        } else {
            req.URL.RawQuery = targetQuery + "&" + req.URL.RawQuery
        }

        // Empty if the header "User-Agent" does not exist
        if _, ok := req.Header["User-Agent"]; !ok {
            // explicitly disable User-Agent so it's not set to default value
            req.Header.Set("User-Agent", "")
        }
    }
    return &ReverseProxy{Director: director}
}

url stitching method

func singleJoiningSlash(a, b string) string {
    aslash := strings.HasSuffix(a, "/")
    bslash := strings.HasPrefix(b, "/")
    switch {
    case aslash && bslash:      // If both a and B exist, the first character of the latter is removed, i.e.'/'followed by stitching
        return a + b[1:]
    case !aslash && !bslash:  // If neither a nor B exists, add'/'between them
        return a + "/" + b
    }
    return a + b  // Otherwise, they are stitched together directly
}

From the example above, we already know that the basic step is to instantiate a reverseproxy object and pass it in to theHttp.ListenAndServeMethod

proxy := NewSingleHostReverseProxy(targetUrl)
http.ListenAndServe(":8888",proxy)

amongHttp.ListenAndServeThe method receives an address and handler, and the function signature is as follows:

func ListenAndServe(addr string, handler Handler) error {...}

Here handler is an interface implemented by ServeHTTP

type Handler interface {
    ServeHTTP(ResponseWriter, *Request)
}

Therefore, we can be sure that the instantiated reverseproxy object also implements the ServeHTTP method
The main steps are:
1. Copy Header to Downstream Requests of Upstream Requests
2. Modification requests (e.g. protocols, parameters, url s, etc.)
3. Determine whether Upgrade is required
4. Delete hop-by-hop headers from upstream requests, that is, headers that do not need to be propagated downstream
5. Set X-Forward-For Header to append current node IP
6. Initiate requests downstream using connection pools
7. Processing protocol upgrade (httpcode 101)
8. Delete step-by-step headers that do not need to be returned upstream
9. Modify the content of the response body (if necessary)
10. Copy downstream response header to upstream response request
11. Return HTTP status code
12. Refresh content to response regularly












Let's analyze the following core method serverHttp

func (p *ReverseProxy) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
    transport := p.Transport
    if transport == nil {
        transport = http.DefaultTransport
    }

    // Check if the request is terminated
    // Get the context of the request, get a CloseNotify instance from responseWriter, start a goroutine to listen for notifyChan, and call the context cancel() method after receiving the end of the request notification
    // Close browsers, network interrupts, forced termination requests, or normal termination requests all receive request termination notifications
    ctx := req.Context()
    if cn, ok := rw.(http.CloseNotifier); ok {
        var cancel context.CancelFunc
        ctx, cancel = context.WithCancel(ctx)
        defer cancel()
        notifyChan := cn.CloseNotify()
        go func() {
            select {
            case <-notifyChan:
                cancel()
            case <-ctx.Done():
            }
        }()
    }

    // Set context, referring to the request you want downstream
    outreq := req.WithContext(ctx) // includes shallow copies of maps, but okay
    if req.ContentLength == 0 {
        outreq.Body = nil // Issue 16036: nil Body for http.Transport retries
    }

    // A deep copy of the Header, that is, the upstream Header is copied to the downstream request Header
    outreq.Header = cloneHeader(req.Header)

    // Set up Director, modify request
    p.Director(outreq)
    outreq.Close = false

    // Upgrade http protocol, HTTP Upgrade
    // Determine if there is an Upgrade in header Connection
    reqUpType := upgradeType(outreq.Header)
    removeConnectionHeaders(outreq.Header)

    // Remove hop-by-hop headers to the backend. Especially
    // important is "Connection" because we want a persistent
    // connection, regardless of what the client sent to us.
    // Delete hop-by-hop header s, mainly those that require no downstream delivery
    for _, h := range hopHeaders {
        hv := outreq.Header.Get(h)
        if hv == "" {
            continue
        }
        // Te and trailers Header s are not deleted
        if h == "Te" && hv == "trailers" {
            // Issue 21096: tell backend applications that
            // care about trailer support that we support
            // trailers. (We do, but we don't go out of
            // our way to advertise that unless the
            // incoming client request thought it was
            // worth mentioning)
            continue
        }
        outreq.Header.Del(h)
    }

    // After stripping all the hop-by-hop connection headers above, add back any
    // necessary for protocol upgrades, such as for websockets.
    // If the reqUpType is not empty, set the Connection, Upgrade values to Upgrade, for example, for websocket scenarios
    if reqUpType != "" {
        outreq.Header.Set("Connection", "Upgrade")
        outreq.Header.Set("Upgrade", reqUpType)
    }

    // Set X-Forwarded-For, append node IP
    if clientIP, _, err := net.SplitHostPort(req.RemoteAddr); err == nil {
        // If we aren't the first proxy retain prior
        // X-Forwarded-For information as a comma+space
        // separated list and fold multiple headers into one.
        if prior, ok := outreq.Header["X-Forwarded-For"]; ok {
            clientIP = strings.Join(prior, ", ") + ", " + clientIP
        }
        outreq.Header.Set("X-Forwarded-For", clientIP)
    }

    // Initiate a request downstream
    res, err := transport.RoundTrip(outreq)
    if err != nil {
        p.getErrorHandler()(rw, outreq, err)
        return
    }

    // Deal with 101 Switching Protocols responses: (WebSocket, h2c, etc)
    // Processing Upgrade Agreement Request
    if res.StatusCode == http.StatusSwitchingProtocols {
        if !p.modifyResponse(rw, res, outreq) {
            return
        }
        p.handleUpgradeResponse(rw, outreq, res)
        return
    }
    // Delete step-by-step header s that respond to requests
    removeConnectionHeaders(res.Header)

    for _, h := range hopHeaders {
        res.Header.Del(h)
    }

    // Modify response content
    if !p.modifyResponse(rw, res, outreq) {
        return
    }

    // Copy Response Header Upstream
    copyHeader(rw.Header(), res.Header)

    // The "Trailer" header isn't included in the Transport's response,
    // at least for *http.Transport. Build it up from Trailer.
    announcedTrailers := len(res.Trailer)
    if announcedTrailers > 0 {
        trailerKeys := make([]string, 0, len(res.Trailer))
        for k := range res.Trailer {
            trailerKeys = append(trailerKeys, k)
        }
        rw.Header().Add("Trailer", strings.Join(trailerKeys, ", "))
    }
    // Write Status Code
    rw.WriteHeader(res.StatusCode)

    // Periodically refresh content to response
    err = p.copyResponse(rw, res.Body, p.flushInterval(req, res))
    if err != nil {
        defer res.Body.Close()
        // Since we're streaming the response, if we run into an error all we can do
        // is abort the request. Issue 23643: ReverseProxy should use ErrAbortHandler
        // on read error while copying body.
        if !shouldPanicOnCopyError(req) {
            p.logf("suppressing panic for copyResponse error in test; copy error: %v", err)
            return
        }
        panic(http.ErrAbortHandler)
    }
    res.Body.Close() // close now, instead of defer, to populate res.Trailer
  ......
}

3 Modify Return Content Instance

The core is to modify the response body content and content length in the ModifyResponse method in reverseproxy

package main

import (
    "bytes"
    "fmt"
    "io/ioutil"
    "log"
    "net/http"
    "net/http/httputil"
    "net/url"
    "strings"
)

func main() {
    // Address Rewrite Instance
    // http://127.0.0.1:8888/test?id=1  => http://127.0.0.1:8081/reverse/test?id=1

    rs1 := "http://127.0.0.1:8081/reverse"
    targetUrl , err := url.Parse(rs1)
    if err != nil {
        log.Fatal("err")
    }
    proxy := NewSingleHostReverseProxy(targetUrl)
    log.Println("Reverse proxy server serve at : 127.0.0.1:8888")
    if err := http.ListenAndServe(":8888",proxy);err != nil{
        log.Fatal("Start server failed,err:",err)
    }
}

func singleJoiningSlash(a, b string) string {
    aslash := strings.HasSuffix(a, "/")
    bslash := strings.HasPrefix(b, "/")
    switch {
    case aslash && bslash:
        return a + b[1:]
    case !aslash && !bslash:
        return a + "/" + b
    }
    return a + b
}

func NewSingleHostReverseProxy(target *url.URL) *httputil.ReverseProxy {
    targetQuery := target.RawQuery
    director := func(req *http.Request) {
        req.URL.Scheme = target.Scheme
        req.URL.Host = target.Host
        req.URL.Path = singleJoiningSlash(target.Path, req.URL.Path)
        if targetQuery == "" || req.URL.RawQuery == "" {
            req.URL.RawQuery = targetQuery + req.URL.RawQuery
        } else {
            req.URL.RawQuery = targetQuery + "&" + req.URL.RawQuery
        }
        if _, ok := req.Header["User-Agent"]; !ok {
            // explicitly disable User-Agent so it's not set to default value
            req.Header.Set("User-Agent", "")
        }
    }

    // Customize ModifyResponse
    modifyResp := func(resp *http.Response) error{
        var oldData,newData []byte
        oldData,err := ioutil.ReadAll(resp.Body)
        if err != nil{
            return err
        }
        // Modify the returned content according to different status codes
        if resp.StatusCode == 200 {
            newData = []byte("[INFO] " + string(oldData))

        }else{
            newData = []byte("[ERROR] " + string(oldData))
        }

        // Modify the returned content and ContentLength
        resp.Body = ioutil.NopCloser(bytes.NewBuffer(newData))
        resp.ContentLength = int64(len(newData))
        resp.Header.Set("Content-Length",fmt.Sprint(len(newData)))
        return nil
    }
    // Incoming custom ModifyResponse
    return &httputil.ReverseProxy{Director: director,ModifyResponse:modifyResp}
}

test result

$ curl http://127.0.0.1:8888/test?id=123
[INFO] http://127.0.0.1:8081/reverse/test?id=123

4 Return to client's true IP

For security reasons, we don't normally expose real servers, or realserver s, directly to external users, but expose services through a reverse proxy, as shown in the following figure:

The problem is how the real server should get the real IP of the user after passing through one or more reverse proxy servers between the user and the real server. In other words, how the intermediate reverse proxy server should transfer the real IP of the user to the back-end real server unchanged.

Typically, we implement this based on an HTTP header, commonly using X-Real-IP and X-Forward-For ward fields.
X-Real-IP: Usually set at the nearest proxy point to the user to record the user's true IP. The next reverse proxy node does not need to be set, otherwise it will overwrite the previous reverse proxy's IP
X-Forward-For: Record the IP of each passing node separated by','for example, if the request link is client -> proxy1 -> proxy2 -> webapp, the resulting values are clientip,proxy1,proxy2

if clientIP, _, err := net.SplitHostPort(req.RemoteAddr); err == nil {
        // If we aren't the first proxy retain prior
        // X-Forwarded-For information as a comma+space
        // separated list and fold multiple headers into one.
        if prior, ok := outreq.Header["X-Forwarded-For"]; ok {
            clientIP = strings.Join(prior, ", ") + ", " + clientIP
        }
        outreq.Header.Set("X-Forwarded-For", clientIP)
    }

Tags: curl network

Posted on Sat, 20 Jun 2020 20:02:13 -0400 by warewolfe