How to integrate google/pprof into existing services?

Last week, I received the R & D requirements from the leaders and wrote a monitoring service monitor server to analyze the status of the services registered in etcd. Most of the services in the project have introduced the pprof library. To view the / debug/pprof of these services, you only need to go through one layer of proxy. Here, use the httputil.NewSingleHostReverseProxy in the official httputil library.

func proxy(w http.ResponseWriter, r *http.Request) {

	_ = r.ParseForm()

	URL := r.Form.Get("url")
	profile := r.Form.Get("profile")

	target, _ := url.Parse("http://" + URL + "/debug/pprof/" + profile + "?debug=1")

	proxy := httputil.NewSingleHostReverseProxy(target)

	proxy.Director = func(req *http.Request) {
		req.URL.Scheme = target.Scheme
		req.URL.Host = target.Host
		req.URL.Path = target.Path
		req.URL.RawQuery = target.RawQuery
		if _, ok := req.Header["User-Agent"]; !ok {
			// explicitly disable User-Agent so it's not set to default value
			req.Header.Set("User-Agent", "")
		}
	}

	r.Host = r.URL.Host
	proxy.ServeHTTP(w, r)
}

Up to now, this requirement has been completed by 90%, but so far, if you want to see the flame graph, you must first download the profile file to the local, and then use pprof or go tool pprof.

So is there a way to integrate pprof into existing services? Of course, let's start with pprof's main function, and notice that it's the google/pprof library.

func main() {
	if err := driver.PProf(&driver.Options{UI: newUI()}); err != nil {
		fmt.Fprintf(os.Stderr, "pprof: %v\n", err)
		os.Exit(2)
	}
}

You can see that options are officially provided. The complete options include:

// Options groups all the optional plugins into pprof.
type Options struct {
	Writer        Writer
	Flagset       FlagSet
	Fetch         Fetcher
	Sym           Symbolizer
	Obj           ObjTool
	UI            UI
	HTTPServer    func(*HTTPServerArgs) error
	HTTPTransport http.RoundTripper
}

These option s do not need to be changed completely, let's separate them.

0. UI interface

UI interface UI: newUI() is removed directly. It is mainly used for terminal interaction control.

1. Flagset interface

// A FlagSet creates and parses command-line flags.
// It is similar to the standard flag.FlagSet.
type FlagSet interface {

	Bool(name string, def bool, usage string) *bool
	Int(name string, def int, usage string) *int
	Float64(name string, def float64, usage string) *float64
	String(name string, def string, usage string) *string

	BoolVar(pointer *bool, name string, def bool, usage string)
	IntVar(pointer *int, name string, def int, usage string)
	Float64Var(pointer *float64, name string, def float64, usage string)
	StringVar(pointer *string, name string, def string, usage string)

	StringList(name string, def string, usage string) *[]*string

	ExtraUsage() string

	AddExtraUsage(eu string)

	Parse(usage func()) []string
}

As it is literally, this interface is used to parse flags. Because we don't want to write flag in the execution script of existing services, we need to implement a custom Flagset structure. At the same time, we also need to solve the problem that go does not support to define flag repeatedly.

1)GoFlags

The content of the structure is subject to the required parameters. Different items may be different.

// GoFlags implements the plugin.FlagSet interface.
type GoFlags struct {
	UsageMsgs []string
	Profile   string
	Http      string
	NoBrowser bool
}

2)Bool

Parameters that need to be changed are passed in through variables in the structure body, and those that do not need to be changed can be written to death directly.

// Bool implements the plugin.FlagSet interface.
func (f *GoFlags) Bool(o string, d bool, c string) *bool {

	switch o {
	case "no_browser":
		return &f.NoBrowser
	case "trim":
		t := true
		return &t
	case "flat":
		t := true
		return &t
	case "functions":
		t := true
		return &t
	}

	return new(bool)
}

3)Int

The default value of the parameter can be found in google/pprof/internal/driver/commands.go.

// Int implements the plugin.FlagSet interface.
func (*GoFlags) Int(o string, d int, c string) *int {

	switch o {
	case "nodecount":
		t := -1
		return &t
	}

	return new(int)
}

4)Float64

// Float64 implements the plugin.FlagSet interface.
func (*GoFlags) Float64(o string, d float64, c string) *float64 {

	switch o {
	case "divide_by":
		t := 1.0
		return &t
	case "nodefraction":
		t := 0.005
		return &t
	case "edgefraction":
		t := 0.001
		return &t
	}

	return new(float64)
}

Note that some default values must be assigned, otherwise the image cannot be displayed normally.

5)String

// String implements the plugin.FlagSet interface.
func (f *GoFlags) String(o, d, c string) *string {

	switch o {
	case "http":
		return &f.Http
	case "unit":
		t := "minimum"
		return &t

	}

	return new(string)
}

6) Parse

The Parse method returns the file name without parsing the parameters.

// Parse implements the plugin.FlagSet interface.
func (f *GoFlags) Parse(usage func()) []string {
	// flag.Usage = usage
	// flag.Parse()
	// args := flag.Args()
	// if len(args) == 0 {
	// 	usage()
	// }
	return []string{f.Profile}
}

7)StringList

// StringList implements the plugin.FlagSet interface.
func (*GoFlags) StringList(o, d, c string) *[]*string {
	return &[]*string{new(string)}
}

So far, the first option change is complete.

if err := driver.PProf(&driver.Options{
		Flagset: &internal.GoFlags{
			Profile:   profilePath + profile,
			Http:      "127.0.0.1:" + strconv.Itoa(internal.ListenPort),
			NoBrowser: true,
		}}); err != nil {
    fmt.Fprintf(os.Stderr, "pprof: %v\n", err)
    os.Exit(2)
}

2. HTTP server function

If you do not register the HTTP server function, pprof uses the default web server.

func defaultWebServer(args *plugin.HTTPServerArgs) error {
	ln, err := net.Listen("tcp", args.Hostport)
	if err != nil {
		return err
	}
	isLocal := isLocalhost(args.Host)
	handler := http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
		if isLocal {
			// Only allow local clients
			host, _, err := net.SplitHostPort(req.RemoteAddr)
			if err != nil || !isLocalhost(host) {
				http.Error(w, "permission denied", http.StatusForbidden)
				return
			}
		}
		h := args.Handlers[req.URL.Path]
		if h == nil {
			// Fall back to default behavior
			h = http.DefaultServeMux
		}
		h.ServeHTTP(w, req)
	})

	mux := http.NewServeMux()
	mux.Handle("/ui/", http.StripPrefix("/ui", handler))
	mux.Handle("/", redirectWithQuery("/ui"))
	s := &http.Server{Handler: mux}
	return s.Serve(ln)
}

1)s.Server(ln)

As you can see, pprof automatically listens by default. However, our service has started listening. These codes can be deleted directly, including the routing part. It is recommended to write them in the same form as the project.

2)handler

The handler first determines whether the request is a local request, and then registers the corresponding handler according to the path. Because we deleted the route part mux.Handle() in the previous step, these codes can also be deleted.

It should be noted that the handler cannot be registered repeatedly, so we need to add a flag bit.

So far, the second option is complete.

var switcher bool
if err := driver.PProf(&driver.Options{
		Flagset: &internal.GoFlags{
			Profile:   profilePath + profile,
			Http:      "127.0.0.1:" + strconv.Itoa(internal.ListenPort),
			NoBrowser: true,
        },
        HTTPServer: func(args *driver.HTTPServerArgs) error {

			if switcher {
				return nil
			}

			for k, v := range args.Handlers {
				http.Handle("/ui"+k, v)
			}
			switcher = true

			return nil
		}}); err != nil {
    fmt.Fprintf(os.Stderr, "pprof: %v\n", err)
    os.Exit(2)
}

3. Reuse handler

We package the above code and write it into the http interface.

func readProfile(w http.ResponseWriter, r *http.Request) {

	_ = r.ParseForm()

	go pprof("profile")

	time.Sleep(time.Second * 3)

	http.Redirect(w, r, "/ui/", http.StatusTemporaryRedirect)

	return

}

After starting pprof, delay three seconds before redirecting to the pprof interface.

On the face of it, this need is done, but...

The above pprof is one-time, and re reading the generated web interface after changing the profile will not re register in the handler.

In order to solve the final problem, we have to change the pprof source code. I'm resistant to this. It's not that I won't change it or that it's not easy to change it. It's mainly that pprof is placed in the company's general vendor library. I'm afraid that it will affect other projects (for this reason, I submitted a feature to the official library, hoping to have a better solution).

Make the following changes under internal/driver/webui.go to make webI reusable.

var webI = new(webInterface)

func makeWebInterface(p *profile.Profile, opt *plugin.Options) *webInterface {
	templates := template.New("templategroup")
	addTemplates(templates)
	report.AddSourceTemplates(templates)

	webI.prof = p
	webI.options = opt
	webI.help = make(map[string]string)
	webI.templates = templates

	return webI
}

At this point, we can finally see the flame chart smoothly.

As for etcd service query and profile file generation, we will not discuss it here.

Tags: Programming Google Web Server Permission denied

Posted on Mon, 04 Nov 2019 04:55:18 -0500 by dgudema