Introduction and examples of Go test testing mechanism [Go language Bible notes]

go test

The go test command is a program that tests code according to a certain convention and organization. In the package directory, all_ The source files with the suffix test.go will not be built as part of the package when executing go build. They are part of the go test test.

*_ In the test.go file, there are three types of functions: test function, benchmark test function, and sample function.

1. A Test function is a function with Test as the prefix of the function name, which is used to Test whether some logical behaviors of the program are correct; The go test command calls these Test functions and reports that the Test result is PASS or FAIL.

2. Benchmark functions are functions whose names are prefixed with benchmark. They are used to measure the performance of some functions; The go test command runs the benchmark function multiple times to calculate an average execution time.

3. The Example function is a function with Example as the prefix of the function name. It provides a sample document whose correctness is guaranteed by the compiler.

We will discuss all the details of the test function in section 11.2, the details of the benchmark function in section 11.4, and then the details of the sample function in section 11.6.

The go test command will traverse all the tests*_ For the functions in the test.go file that comply with the above naming rules, generate a temporary main package to call the corresponding test functions, then build, run and report the test results, and finally clean up the temporary files generated in the test.

Test function

Test function starts with test

Each test function must import the testing package. The test function has the following signature:

func TestName(t *testing.T) { // ... }

The name of the Test function must start with Test, and the optional suffix must start with an uppercase letter:

func TestSin(t *testing.T) { /* ... */ } func TestCos(t *testing.T) { /* ... */ } func TestLog(t *testing.T) { /* ... */ }

The t parameter is used to report test failures and additional log information. Let's define an instance package gopl.io/ch11/word1, in which there is only one function IsPalindrome to check whether a string is read from front to back and from back to front (i.e. palindrome). (the following implementation repeats the test twice before and after whether a string is a palindrome string; we will discuss this later.)

gopl.io/ch11/word1

// Package word provides utilities for word games package word // IsPalindrome reports whether s reads the same forward and backward // (Our first attempet.) func IsPalidrome(s string) bool { for i := range s { if s[i] != s[len(s)-1-i] { return false } } return true }

In the same directory, word_ The test.go test file contains two test functions, TestPalindrome and TestNonPalindrome. Each is to test whether IsPalindrome gives correct results and report the failure information using t.Error:

package word import "testing" func TestPalindrome(t *testing.T) { if !IsPalindrome("detartrated") { t.Error(`IsPalindrome("detarated") = false`) } if !IsPalindrome("kayak") { t.Error(`IsPalindrome("kayak") = false`) } } func TestNonPalindrome(t *testing.T) { if IsPalindrome("palindrome") { t.Error(`IsPalindrome("palindrome") = true`) } }

If no package is specified in the go test command, the package corresponding to the current directory will be used by default (the same as the go build command). We can build and run tests with the following commands.

$ cd $GOPATH/src/gopl.io/ch11/word1 $ go test ok gopl.io/ch11/word1 0.008s

(note to the author: a Test function is a function with Test as the prefix. It is supported by go test and can be called through the go test command. In essence, a Test function defines Test data and uses the Test data as parameters to call the tested function to observe whether the returned results meet the expectations. From this point of view, each Test function is an executable Test Use case.)

The results are quite satisfactory. We ran this program, but we didn't quit early because we haven't encountered a BUG report yet. However, a French user named "Noelle Eve Elleon" will complain that the IsPalindrome function does not recognize "e t é". Another complaint from users in Central America is that they can't recognize "A man, a plan, a canal: Panama.". Executing special and small BUG reports provides us with new and more natural test cases.

func TestFrenchPalindrome(t *testing.T) { if !IsPalindrome("été") { t.Error(`IsPalindrome("été") = false`) } } func TestCanalPalindrome(t *testing.T) { input := "A man, a plan, a canal: Panama" if !IsPalindrome(input) { t.Errorf(`IsPalindrome(%q) = false`, input) } }

In order to avoid entering a long string twice, we use the errorfunction with Printf formatting function to report the error results.

After adding these two test cases, go test returns the information of test failure.

$ go test --- FAIL: TestFrenchPalindrome (0.00s) word_test.go:28: IsPalindrome("été") = false --- FAIL: TestCanalPalindrome (0.00s) word_test.go:35: IsPalindrome("A man, a plan, a canal: Panama") = false FAIL FAIL gopl.io/ch11/word1 0.014s

It is a good test habit to write test cases first and observe that the test cases trigger the same description as the errors reported by users. Only in this way can we locate the problem we really want to solve.

Another advantage of writing test cases first is that running tests is usually faster than manually describing reports, which allows us to iterate quickly. If the test set has many slow running tests, we can speed up the test by choosing to run only some specific tests.

The parameter - v can be used to print the name and running time of each test function:

$ go test -v === RUN TestPalindrome --- PASS: TestPalindrome (0.00s) === RUN TestNonPalindrome --- PASS: TestNonPalindrome (0.00s) === RUN TestFrenchPalindrome --- FAIL: TestFrenchPalindrome (0.00s) word_test.go:28: IsPalindrome("été") = false === RUN TestCanalPalindrome --- FAIL: TestCanalPalindrome (0.00s) word_test.go:35: IsPalindrome("A man, a plan, a canal: Panama") = false FAIL exit status 1 FAIL gopl.io/ch11/word1 0.017s

The parameter - run corresponds to a regular expression. Only the test function whose test function name is correctly matched by it will be run by the go test command:

$ go test -v -run="French|Canal" === RUN TestFrenchPalindrome --- FAIL: TestFrenchPalindrome (0.00s) word_test.go:28: IsPalindrome("été") = false === RUN TestCanalPalindrome --- FAIL: TestCanalPalindrome (0.00s) word_test.go:35: IsPalindrome("A man, a plan, a canal: Panama") = false FAIL exit status 1 FAIL gopl.io/ch11/word1 0.014s

Of course, once we have repaired the failed test cases, before submitting the code update, we should run all the test cases with the go test command without parameters to ensure that no new problems are introduced while repairing the failed tests.

Our task now is to fix these errors. After a brief analysis, it is found that the reason for the first BUG is that we use byte rather than run sequence, so non ASCII characters such as é in "é t é" cannot be handled correctly. The second BUG is caused by not ignoring the case of spaces and letters.

For the above two bugs, we carefully rewrite the function:

gopl.io/ch11/word2

// Package word provides utilities for word games. package word import "unicode" // IsPalindrome reports whether s reads the same forward and backward // Letter case is ignored, as are non-letters func IsPalindrome(s string) bool { var letters []rune for _, r := range s { if unicode.IsLetter(r) { letters = append(letters, unicode.ToLower(r)) } } for i := range letters { if letters[i] != letters[len(letters)-1-i] { return false } } return true }

At the same time, we also combine all the previous test data into a table in the test (note to the author: table driven test is a test idea, that is, list a table, and each table item represents a group of considered test data. It is embodied in the code as a user-defined structure array, which allows multiple test data to be stored at one time).

func TestIsPalindrome(t *testing.T) { var tests = []struct { input string want bool } { {"", true}, {"a", true}, {"ab", false}, {"kayak", true}, {"detartrated", true}, {"A man, a plan, a canal: Panama", true} {"Evil I did dwell; lewd did I live.", true}, {"Able was I ere I saw Elba", true}, {"été", true}, {"Et se resservir, ivresse reste.", true}, {"palindrome", false}, // non-palindrome {"desserts", false}, // semi-palindrome } for _, test := range tests { if got := IsPalindrome(test.input); got != test.want { t.Errorf(IsPalindrome(%q) = %v, test.input, got) } } }

Now our new tests have passed:

$ go test gopl.io/ch11/word2 ok gopl.io/ch11/word2 0.015s

This table driven test is very common in Go language. It is easy to add new test data to the table, and there is no redundancy in the subsequent test logic, so we can have more energy to improve the error information.

The output of the failed test does not include the stack call information at the time of calling t.Errorf. Unlike assert assertions in other programming languages or test frameworks, t. errorf calls do not cause panic exceptions or stop test execution. Even if the data in front of the table leads to the failure of the test, the test data behind the table will still run the test. Therefore, we may know the information of multiple failures in a test.

If we really need to stop the test, perhaps because of initialization failure or subsequent errors caused by previous errors, we can use t.Fatal or t.Fatalf to stop the current test function, but they must be called in the same goroutine as the test function.

The general form of test failure information is "f(x) = y, want z", where f(x) explains the failed operation and corresponding input, y is the actual operation result, and z is the expected correct result. As in the previous example of checking the palindrome string, the actual function is used for the f(x) part. Displaying x is an important part of table driven testing because the same assertion may be executed multiple times for different table items. Avoid useless and redundant information. When testing functions like IsPalindrome that return Boolean types, you can ignore z part with theout additional information. If x is y or z is the length of Y, just output a concise summary of the relevant parts. Test authors should try to help programmers diagnose the cause of test failure.

Random test

Table driven testing facilitates the construction of test cases based on carefully selected test data. Another testing idea is random testing, which is to test the behavior of exploration function by constructing a wider range of random inputs.

So how can we know the desired output for a random input? There are two processing strategies:

• The first is to write another control function, using a simple and clear algorithm. Although the efficiency is low, the behavior is consistent with the function to be tested, and then check the output results of both for the same random input.
• The second is that the generated random input data follows a specific pattern, so that we can know the pattern of the desired output.

The following example uses the second method: the random palindrome function is used to randomly generate a palindrome string.

import "math/rand" // randomPalindrome returns a palindrome whose length and contents // are derived from the pseudo-random number generator rng. func randomPalindrome(rng *rand.Rand) string { n := rng.Intn(25) // random length up to 24 runes := make([]rune, n) for i:=0; i<(n+1)/2; i++ { r := rune(rng.Intn(0x1000)) // random rune up to '\u0999' runes[i] = r runes[n-1-i] = r } return string(runes) } func TestRandomPalindromes(t *testing.T) { // Initialize a pseudo-random number generator. seed := time.Now().UTC().UnixNano() t.Logf("Random seed: %d", seed) rng = rand.New(rand.NewSource(seed)) for i:=0; i<1000; i++ { p := randomPalindrome(rng) if !IsPalindrome(p) { t.Errorf("IsPalindrome(%q) = false", p) } } }

Although random testing has uncertainties, it is also very important. We can get enough information from the log of failed tests. In our example, inputting the p parameter of IsPalindrome will tell us the real data, but for the function, it will accept more complex input. It is not necessary to save all the input, as long as the random number seed is simply recorded in the log (like the above method). With these random number initialization seeds, we can easily modify the test code to reproduce failed random tests.

By using the current time as a random seed, new random data will be explored each time the test command is run throughout the process. If you use an automated test integration system that runs regularly, random testing will be particularly valuable.

Translator's note: readers interested in extended reading can learn more about go fuzzy.

Test a command

go test is a useful tool for the test package, but with a little effort, we can also use it to test executable programs. If the name of a package is main, an executable program will be generated during construction, but the main package can be imported by tester code as a package.

Let's write a test for the echo program in section 2.3.2. We first split the program into two functions: the echo function completes the real work, and the main function is used to handle the command line input parameters and the errors that echo may return.

gopl.io/ch11/echo

// Echo prints its command-line arguements package main import ( "flag" "fmt" "io" "os" "strings" ) var ( n = flag.Bool("n". false, "omit trailing newline") s = flag.String("s", " ", "separator") ) var out io.Writer = os.Stdout // modified during testing func main() { flag.Parse() if err := echo(!*n, *s, flag.Args()); err != nil { fmt.Fprintf(os.Stderr, "echo: %v\n", err) os.Exit(1) } } func echo(newline bool, sep string, args []string) error { fmt.Fprint(out, strings.Join(args, sep)) if newline { fmt.Fprintln(out) } return nil }

In the test, we can call the echo function with various parameters and flags, and then check whether its output is correct. We can reduce the dependence of the echo function on global variables by adding parameters. We also added a global variable named out to replace the direct use of os.Stdout, so that the test code can modify out to different objects as needed for inspection. Here is echo_ Test code in test.go file:

package main import ( "bytes" "fmt" "testing" ) func TestEcho(t *testing.T) { var tests = []struct { newline bool sep string args []string want string }{ , "\n"}, , ""}, , "one\ttwo\tthree\n"}, , "a,b,c\n"}, , "1:2:3"}, } for _, test := range tests { descr := fmt.Sprintf("echo(%v, %q, %q)", test.newline, test.sep, test.args) out = new(bytes.Buffer) // captured output if err := echo(test.newline, test.sep, test.args); err != nil { t.Errorf("%s failed: %v", descr, err) continue } got := out.(*bytes.Buffer).String() if got != test.want { t.Errorf("%s = %q, want %q", descr, got, test.want) } } }

Note that the test code and the product code are in the same package. Although it is a main package and has a corresponding main entry function, during testing, the main package is only an ordinary package imported by the TestEcho test function, in which the main function is not exported, but ignored.

By putting tests into a table, it is easy to add new test cases. Let me see what the failure is like by adding the following test cases:

, "a b c\n"}, // NOTE: wrong expectation!

The output of go test is as follows:

$ go test gopl.io/ch11/echo --- FAIL: TestEcho (0.00s) echo_test.go:31: echo(true, ",", ["a" "b" "c"]) = "a,b,c", want "a b c\n" FAIL FAIL gopl.io/ch11/echo 0.006s

The error message describes the attempted operation (using Go like syntax), the actual result and the expected result. With this error message, you can easily locate the cause of the error before viewing the code.

Note that log.Fatal or os.Exit is not called in the test code, because calling such functions will cause the program to exit early; The privilege of calling these functions should be placed in the main function. If something unexpected really causes a panic exception in the function, the test driver should try to catch the exception with recover, and then treat the current test as a failure. If it is a predictable error, such as illegal user input, missing file or improper configuration file, it should be handled by returning a non empty error. Fortunately (the above accident is just an episode), our echo example is relatively simple, and there is no need to return a non empty error.

White box test

A test classification method is based on whether the tester needs to understand the internal working principle of the tested object. Black box testing only needs the documents and API behaviors exposed by the test package, and the internal implementation is invisible to the test code. On the contrary, the white box test has access to the internal functions and data structures of the package, so it can do some tests that ordinary clients cannot achieve. For example, a white box test can detect the data type of the invariant after each operation. (white box test is just a traditional name. In fact, it is more accurate to call it clear box test.)

Black box and white box are complementary. Black box testing is generally more robust. With the improvement of software implementation, the test code rarely needs to be updated. They can help testers understand the needs of real customers and find some deficiencies in API design. On the contrary, white box testing can provide more test coverage for some difficult internal implementations.

We have seen examples of two tests. The TestIsPalindrome test uses only the exported IsPalindrome function, so this is a black box test. The TestEcho test calls the internal echo function and updates the internal out package level variables. Both of them are not exported, so this is a white box test.

When preparing the TestEcho test, we modified the echo function to use the package level out variable as the output object, so the test code can use another implementation to replace the standard output, which can facilitate the comparison of echo output data. Using similar techniques, we can replace other parts of the product code with a pseudo object that is easy to test. The advantage of using pseudo objects is that we can easily configure, predict, be more reliable and observe. At the same time, we can also avoid some adverse side effects, such as updating the production database or credit card consumption behavior.

The following code demonstrates the quota detection logic in a web service that provides network storage for users. When the user uses more than 90% of the storage quota, a reminder message will be sent. (Note: generally, when implementing business machine monitoring, including disk, cpu, network, etc., similar logic of reaching threshold = > triggering alarm is required, so it is a very practical case)

gopl.io/ch11/storage1

package storage import ( "fmt" "log" "net/smtp" ) func bytesInUse(username string) int64 { return 0 /* ... */ } // Email sender configuration. // NOTE: never put passwords in source code! Note: the source code cannot contain any plaintext password, which is the basic security specification const sender = "[email protected]" const password = "correcthorsebatterystaple" const hostname = "smtp.example.com" const template = `Warning: you are using %d bytes of storage, %d%% of your quota.` func CheckQuota(username string) { used := bytesInUse(username) const quota = 1000000000 // 1GB percent := 100 * used / quota if percent < 90 { return // OK } msg := fmt.Sprintf(template, used, percent) auth := smtp.PlainAuth("", sender, password, hostname) err := smtp.SendMail(hostname+":587", auth, sender, []string, []byte(msg)) if err != nil { log.Printf("smtp.SendMail(%s) failed: %s", username, err) } }

We want to test this code, but we don't want to send real mail. So we put the mail processing logic into a private notifyUser function.

gopl.io/ch11/storage2

var notifyUser = func(username, msg string) { auth := smtp.PlainAuth("", sender, password, hostname) err := smtp.SendMail(hostname+":587", auth, sender, []string, []byte(msg)) if err != nil { log.Printf("smtp.SendEmail(%s) failed: %s", username, err) } } func CheckQuota(username string) { used := bytesInUse(username) const quota = 1000000000 // 1GB percent := 100 * used / quota if percent < 90 { return // OK } msg := fmt.Sprintf(template, used, percent) notifyUser(username, msg) }

Now we can replace the real mail sending function with the pseudo mail sending function in the test. It simply records the users to be notified and the contents of the mail.

package storage import ( "strings" "testing" ) func TestCheckQuotaNotifiesUser(t *testing.T) { var notifiedUser, notifiedMsg string notifyUser = func(user, msg string) { // Note to the author: look carefully. Here is notifyUser. It is a function variable used to simulate the process of sending e-mail to users notifiedUser, notifiedMsg = user, msg } // ...simulate a 980MB-used condition... const user = "[email protected]" CheckQuota(user) if notifiedUser == "" && notifiedMsg == "" { t.Fatalf("notifyUser not called") } if notifiedUser != user { t.Errorf("wrong user (%s) notified, want %s", notifiedUser, user) } const wantSubstring = "98% of your quota" if !strings.Contains(notifiedMsg, wantSubstring) { t.Errorf("unexpected notification message <<%s>>, "+ "want substring %q", notifiedMsg, wantSubstring) } }

There is a problem here: when the test function returns, CheckQuota will not work normally, because notifyUsers still uses the pseudo send mail function of the test function (there is always this risk when updating the global object). We must modify the test code to restore the original state of notifyUsers so that other subsequent tests will not be affected. We must ensure that all execution paths can be restored, including test failures or panic exceptions. In this case, we recommend using the defer statement to delay the execution of the code that handles the recovery.

func TestCheckQuotaNotifiesUser(t *testing.T) { // Save and restore original notifyUser. saved := notifyUser defer func() { notifyUser = saved }() // Install the test's fake notifyUser. var notifiedUser, notifiedMsg string notifyUser = func(user, msg string) { notifiedUser, notifiedMsg = user, msg } // ...rest of test... }

This processing mode can be used to temporarily save and restore all global variables, including command line flag parameters, debugging options and optimization parameters; Install and remove hook functions that cause production code to generate some debugging information; There are also some changes that induce production code to enter some important states, such as timeouts, errors, and even deliberate concurrency.

Using global variables in this way is safe because the go test command does not execute multiple tests concurrently.

External test package

Consider the following two packages: net/url package, which provides the function of URL parsing; net/http package, which provides the functions of web service and HTTP client. As expected, the upper net/http package depends on the lower net/url package. Then, one test in the net/url package is to demonstrate the interaction between different URLs and HTTP clients. In other words, the test code of a lower package is imported into the upper package.

Such behavior will lead to package circular dependency in the test code of net/url package, as shown by the up arrow in Figure 11.1. At the same time, as we said in Section 10.1, Go language specification prohibits package circular dependency.

However, we can solve the problem of circular dependency through external test packages, that is, declare an independent URL in the directory where the net/url package is located_ Test package. Where package name_ The test suffix tells the go test tool that it should create an additional package to run tests. We regard the import path of this external test package as net/url_test is easier to understand, but in fact it cannot be imported by any other package.

Because the external test package is an independent package, you can import other auxiliary packages that depend on the code to be tested itself. However, the test code in the package cannot do this. At the design level, the external test package is at the top of all the packages it depends on, as shown in Figure 11.2.

(note to the author: as shown in the figure, there is no circular dependency. Because net/url_test is imported separately from net/http and net/url, net/url import from net/http is avoided.)

By avoiding circular import dependency, external test packages can write tests more flexibly, especially integration tests (which need to test the interaction between multiple components), and can freely import other packages like ordinary applications.

We can use the go list command to view which Go source files in the corresponding directory of the package are product code, which are in package tests, and which are external test packages. We take the fmt package as an example: GoFiles represents the list of Go source files corresponding to the product code; That is, the part to be compiled by the go build command.

$ go list -f={{.GoFiles}} fmt [doc.go format.go print.go scan.go]

TestGoFiles represents the internal test code of fmt package to_ test.go is the suffix file name, but it is only built during testing:

$ go list -f={{.TestGoFiles}} fmt [export_test.go]

The test code of the package is usually in these files, but this is not the case with the fmt package; We'll explain export later_ The function of the test.go file.

XTestGoFiles represents the test code belonging to the external test package, that is, fmt_test package, so they must first import the fmt package. Similarly, these files are only built and run during testing:

$ go list -f={{.XTestGoFiles}} fmt [fmt_test.go scan_test.go stringer_test.go]

Sometimes the external test package also needs to access the code inside the tested package. For example, in a white box test scenario that is independent of the external test package to avoid circular import. In this case, we can solve it through some skills: one in our package_ Export an internal implementation to an external test package from the test.go file. Because these codes are only needed for testing, they are generally placed in export_ In the test.go file.

For example, the fmt.Scanf function of the FMT package requires the functionality provided by the unicode.IsSpace function. However, in order to avoid too many dependencies, FMT package does not import Unicode package containing huge table data; Instead, the FMT package has a simple implementation called isSpace internal.

To ensure that the fmt.isSpace and unicode.IsSpace functions behave consistently, the FMT package carefully includes a test. However, a white box test function in an external test package cannot directly access the internal function isSpace, so FMT exports the isSpace function through a back door. export_ The test.go file is a back door for external test packages.

package fmt var IsSpace = isSpace

This test file does not define test code; It simply exports the internal isspace function through fmt.IsSpace and provides it to the external test package (in essence, the call of isspace is drained to isspace through variable assignment). This technique can be widely used for white box testing in external test packages.

Write valid tests

Many newcomers to Go will be surprised by the minimalist testing framework of Go. Many test frameworks in other languages provide a mechanism to identify test functions (usually using reflection or metadata). Some hook functions of "setup" and "teardown" are set to perform initialization and subsequent cleaning operations of test case operation. At the same time, the test tool box also provides many functions such as assert assertion, value comparison Auxiliary functions such as formatting output error messages and stopping a failed test (usually using an exception mechanism). Although these mechanisms can make the test very concise, the log of test output will be as difficult to understand as Martian. In addition, although the test will eventually output PASS or FAIL reports, the information format provided by them is very disadvantageous to code maintainers to quickly locate problems, because the specific meaning of failure information is very obscure, such as "assert: 0 == 1" or paged massive tracking logs.

The test style of Go language is in sharp contrast. It expects the tester to complete most of the work and define functions to avoid repetition, just like ordinary programming. Writing tests is not a mechanical process of filling in the blanks. A test also has its own interface, although its maintainer is also the only user of the test. A good test should not cause other irrelevant error messages. It only needs to clearly and concisely describe the symptoms of the problem, and sometimes some context information may be required. Ideally, the maintainer can locate the cause of the error according to the error information without looking at the code. A good test should not quit the test immediately when it encounters a small error. It should try to report more relevant error information, because we may find the law of error generation from multiple failed test patterns.

The following assertion function compares the two values, then generates a general error message and stops the program. It's easy to use and effective, but when the test fails, the printed error message is almost worthless. It does not provide a good entry point for quick problem solving.

import ( "fmt" "strings" "testing" ) // A poor assertion function. func assertEqual(x, y int) { if x != y { panic(fmt.Sprintf("%d != %d", x, y)) } } func TestSplit(t *testing.T) { words := strings.Split("a:b:c", ":") assertEqual(len(words), 3) // ... }

In this sense, the assertion function makes the mistake of premature abstraction: it only tests whether two integers are the same, but fails to provide more meaningful error information according to the context. We can print a more valuable error message according to the specific error, as in the following example. Abstraction is used only when duplicate patterns occur in the test.

func TestSplit(t *testing.T) { s, sep := "a:b:c", ":" words := strings.Split(s, sep) if got, want := len(words), 3; got != want { t.Errorf("Split(%q, %q) returned %d words, want %d", s, sep, got, want) } // ... }

Author's note: this example prints sufficient context information.

The current test not only reports the specific function called, its input and the meaning of the result, but also prints the real returned value and the expected returned value, and will continue to try to run more tests even if the assertion fails. Once we write a test with this structure, the next step is not to expand the test cases with more if statements. We can prepare more s and sep test cases like IsPalindrome's table driven test.

The previous example does not require additional auxiliary functions. If there are methods that can make the test code easier, we are also willing to accept them. (we will see an auxiliary function similar to reflect.DeepEqual in section 13.3.) the key to a good test is to realize the specific behavior you expect first, and then consider simplifying the test code and avoiding repetition. If we start directly from the abstract and general test library, it is difficult to achieve good results.

Avoid fragile testing

If an application fails frequently for new but valid input, it indicates that the program is prone to bug s (not robust enough). Similarly, if a test fails with only minor changes to the program, it is called fragile. Just as a program that is not robust enough will frustrate its users, a fragile test will also annoy its maintainers. The most vulnerable test code will produce different results when there is no change in the program, good and bad. It will take a lot of time to deal with them, but it won't get any benefit

When a test function will produce a complex output, such as a long string, a well-designed data structure or a file, it is easy to write down a series of fixed benchmark data for comparison in advance. However, as the project progresses, some outputs may change, although it is likely to be caused by an improved implementation. Moreover, not only the output part, but also the complex input part of the function may change, so the input used in the test is no longer valid.

The way to avoid fragile test code is to detect only the attributes you really care about. Keep the test code concise and the internal structure stable. In particular, we should choose the assertion part. Do not match the whole word of the string, but for those substrings that are relatively stable in the development of the project. Many times, it is worth making efforts to write a function to extract the necessary information for assertion from complex output. Although this may bring a lot of preliminary work, it can help quickly and timely repair the illogical failure tests caused by project evolution.

Test coverage

In terms of the nature of the test, it cannot be complete. Computer scientist Edsger Dijkstra once said, "testing can prove the existence of defects, but can not prove that there are no defects." no amount of testing can prove that a program has no bugs (author: there is another implementation path to prove that the program has no bugs, which is called verification, but the cost is very high, and it is not suitable for the high-speed iterative environment such as the Internet). At best, testing can enhance our confidence that the code works well in many important scenarios.

The extent to which the program under test performs the test is called the coverage of the test. Test coverage is not quantifiable - even the dynamics of the simplest programs are difficult to measure accurately - but there are heuristics (Note: heuristics are the way we think of) to help us write effective test code.

Among these heuristic methods, statement coverage is the simplest and most widely used. Statement coverage refers to the proportion of code that is run at least once in the test to the total number of code. In this section, we use the test coverage tool integrated in the go test command to measure the test coverage of the following code to help us identify the gap between the test and our expectations.

The following code is a table driven test for testing the expression evaluator in Chapter 7:

gopl.io/ch7/eval

func TestCoverage(t *testing.T) { var tests = []struct { input string env Env want string // expected error from Parse/Check or result from Eval }{ {"x % 2", nil, "unexpected '%'"}, {"!true", nil, "unexpected '!'"}, {"log(10)", nil, `unknown function "log"`}, {"sqrt(1, 2)", nil, "call to sqrt has 2 args, want 1"}, {"sqrt(A / pi)", Env{"A": 87616, "pi": math.Pi}, "167"}, {"pow(x, 3) + pow(y, 3)", Env{"x": 9, "y": 10}, "1729"}, {"5 / 9 * (F - 32)", Env{"F": -40}, "-40"}, } for _, test := range tests { expr, err := Parse(test.input) if err == nil { err = expr.Check(map[Var]bool{}) } if err != nil { if err.Error() != test.want { t.Errorf("%s: got %q, want %q", test.input, err, test.want) } continue } got := fmt.Sprintf("%.6g", expr.Eval(test.env)) if got != test.want { t.Errorf("%s: %v => %s, want %s", test.input, test.env, got, test.want) } } }

First of all, we should ensure that all tests pass normally:

$ go test -v -run=Coverage gopl.io/ch7/eval === RUN TestCoverage --- PASS: TestCoverage (0.00s) PASS ok gopl.io/ch7/eval 0.011s

The following command can display the usage of the test coverage tool:

$ go tool cover Usage of 'go tool cover': Given a coverage profile produced by 'go test': go test -coverprofile=c.out Open a web browser displaying annotated source code: go tool cover -html=c.out ...

The go tool command runs the underlying executable of the go tool chain. These underlying executables are placed in $GOROOT/pkg/tool/$_$ directory. Because of the go build command, we rarely call these underlying tools directly.

Now we can rerun the test with the - coverprofile flag parameter:

$ go test -run=Coverage -coverprofile=c.out gopl.io/ch7/eval ok gopl.io/ch7/eval 0.032s coverage: 68.5% of statements

This flag parameter counts the coverage data by inserting a generation hook into the test code. That is, before running each test, it copies and modifies the code to be tested, and sets a boolean flag variable in each lexical block. When the modified code under test exits, write the statistical log data to the c.out file and print a summary of some executed statements. (if you need a summary, use go test -cover.)

If the - covermode=count flag parameter is used, a counter is inserted in each code block instead of a boolean flag. The execution times of each block are recorded in the statistical results, which can be used to measure which hot code is frequently executed.

To collect data, we run the test coverage tool, print the test log, generate an HTML report, and then open it in the browser (Figure 11.3).

$ go tool cover -html=c.out

The green code block is covered by the test, and the red indicates that it is not covered. For clarity, we set the background of the red text to the shadow effect. We can immediately find that the Eval method of the unary operation has not been executed. If we add the following test cases for this part of the uncovered code, and then rerun the above command, we will see that the code in the red part will also turn green:

{"-x * -x", eval.Env{"x": 2}, "4"}

However, the two panic statements are still red. This is no problem because these two statements will not be executed.

Achieving 100% test coverage sounds beautiful, but it is usually not feasible in specific practice and is not recommended. Because that only means that the code has been executed, it does not mean that the code is BUG free. For logically complex statements, you need to execute multiple times for different inputs. Some statements, such as the panic statement above, will never be executed. In addition, there are some obscure errors that are rarely encountered in reality, and it is difficult to write the corresponding test code. Testing is a practical work in essence. The cost comparison between writing test code and writing application code needs to be considered. Test coverage tools can help us quickly identify test weaknesses, but the designed test cases need rigorous thinking as well as writing application code.

Benchmarking

Benchmarking is to measure the performance of a program under a fixed workload. In Go language, Benchmark functions are written similarly to ordinary test functions, but they are prefixed with Benchmark and have a parameter of type * testing.B* The testing.B parameter not only provides methods similar to * testing.T, but also some other methods related to performance measurement. It also provides an integer N that specifies the number of cycles the operation executes.

The following is a benchmark for the IsPalindrome function, where the loop will execute N times.

import "testing" func BenchmarkIsPalindrome(b *testing.B) { for i:=0; i<b.N; i++ { IsPalindrom("A man, a plan, a canal: Panama") } }

We run the benchmark with the following command. Unlike normal tests * *, no benchmark tests are run by default * *. We need to manually specify the benchmark function to run through the - bench command line flag parameter. This parameter is a regular expression used to match the name of the benchmark function to be executed. The default value is empty. The "." mode can match all benchmark functions, but there is only one benchmark function here, Therefore, it is equivalent to the - bench=IsPalindrome parameter.

$ cd $GOPATH/src/gopl.io/ch11/word2 $ go test -bench=. # Equivalent to go test -bench=IsPalindrome PASS BenchmarkIsPalindrome-8 1000000 1035 ns/op ok gopl.io/ch11/word2 2.179s

The numeric suffix of the benchmark name in the result, here 8, represents the value of GOMAXPROCS corresponding to the runtime, which is important information for some benchmark tests related to concurrency.

The report shows that each call to the IsPalindrome function takes 1.035 microseconds, which is the average time of 1000000 executions. Because the benchmark driver does not know the running time of each benchmark function at the beginning, it will try to run the test with a smaller N to estimate the time required for the benchmark function before actually running the benchmark, and then infer a larger time to ensure stable measurement results.

The loop is implemented in the benchmark function rather than in the benchmark framework, which gives each benchmark function the opportunity to execute the initialization code before the loop starts without significantly affecting the average running time of each iteration. If you are still worried that the initialization code will interfere with the measurement time, you can temporarily close or reset the timer through the method provided by the testing.B parameter, but these are rarely used.

Now that we have a benchmark and general test, we can easily test the idea of improving the running speed of the program. Perhaps the most obvious optimization is the stop check of the second loop in the IsPalindrome function, which can avoid doing twice for each comparison:

n := len(letters)/2 for i:=0; i<n; i++ { if letters[i] != letters[len(letters)-1-i] { return false } } return true

However, in many cases, an obvious optimization may not bring the expected results. This improvement resulted in only a 4% performance improvement in the benchmark.

$ go test -bench=. PASS BenchmarkIsPalindrome-8 1000000 992 ns/op ok gopl.io/ch11/word2 2.093s

Another improvement idea is to pre allocate a large enough array for each character at the beginning, so as to avoid multiple reallocation of memory during the append call. Declare a letters array variable and specify the appropriate size, as follows:

letters := make([]rune, 0, len(s)) for _, r := range s { if unicode.IsLetter(r) { letters = append(letters, unicode.ToLower(r)) } }

This improvement improves performance by about 35%, and the reported results are based on the average running time statistics of 2000000 iterations.

$ go test -bench=. PASS BenchmarkIsPalindrome-8 2000000 697 ns/op ok gopl.io/ch11/word2 1.468s

As this example shows, fast programs are often accompanied by less memory allocation- The benchmem command line flag parameter will include memory allocation statistics in the report. We can compare the memory allocation before and after optimization. Before optimization:

$ go test -bench=. -benchmen PASS BenchmarkIsPalindrome 1000000 1026 ns/op 304 B/op 4 allocs/op

This is the result of Optimization:

$ go test -bench. -benchmem PASS BenchmarkIsPalindrome 2000000 807 ns/op 128 B/op 1 allocs/op

Using one memory allocation instead of multiple memory allocation saves 75% of the allocation calls and nearly half of the memory requirements.

This benchmark tells us the absolute time required for a specific operation, but what we often want to know is the time comparison of two different operations. For example, if a function needs to process 1000 elements in 1ms, how long will it take to process 10000 or 1 million? Such a comparison reveals the running time of the asymptotic growth function. Another example: how big should I/O cache be? Benchmarking can help us choose the minimum memory we need when performance is up to standard. The third example: which algorithm is better for a certain job? The benchmark can evaluate the advantages and disadvantages of two different algorithms under different scenarios and loads for the same input.

Comparative benchmarks are ordinary program code. They are usually single parameter functions called by several benchmark functions of different orders of magnitude, like this:

func benchmark(b *testing.B, size int) { /* ... */ } func Benchmark10(b *testing.B) { benchmark(b ,10) } func Benchmark100(b *testing.B) { benchmark(b, 100) } func Benchmark1000(b *testing.B) { benchmark(b, 1000)}

The size of the input is specified by function parameters, but the parameter variables are fixed for each specific benchmark. Avoid directly modifying b.N to control the size of input. Unless you use it as input for a fixed size iteration, the results of the benchmark will be meaningless.

The patterns reflected by comparative benchmarks are very helpful in the program design stage, but the benchmark code should be retained even after the program is completed. Because with the development of the project, or the increase of input, or the deployment to new operating systems or different processors, we can use benchmarking again to help us improve the design.

analyse

Benchmarking is helpful to measure the performance of a specific operation, but when we try to make the program run faster, we usually don't know where to start optimization. Every yard farmer should know Donald Knuth's maxim in "Structured Programming with go to Statements" in 1974. Although it is often interpreted as not paying attention to performance, we can see different meanings from the original text:

There is no doubt that the one-sided pursuit of efficiency will lead to all kinds of abuse. Programmers will waste a lot of time on the speed of non critical programs. In fact, these attempts to improve efficiency may have a great negative impact, especially when debugging and maintenance. We should not dwell too much on the optimization of details. It should be said that about 97% of the scenarios: premature optimization is the source of all evil.

Of course, we should not give up the optimization of that key 3%. A good programmer will not hold back because of this small proportion. They will wisely observe and identify what is the key code; However, optimization will be carried out only when the key code has been confirmed. For many programmers, it is easy to make empirical mistakes to judge which part is the key performance bottleneck, so they should generally be proved with the help of measurement tools.

When we want to carefully observe the running speed of our program, the best way is performance analysis. Analysis technology is based on some automatic sampling during program execution, and then inference at the end. The final statistical results are called analysis data.

Go language supports many types of profiling performance analysis, each focusing on different aspects, but they all involve a series of event messages of interest recorded by each sampling. Each event contains the information of the function call stack during function call. The built-in go test tool supports several analysis methods.

The CPU profiling data identifies the function that consumes the most CPU time. The thread running on each CPU will encounter the interrupt event of the operating system every few milliseconds. Each interrupt will record a profiling data and then resume normal operation.

Heap parsing identifies the most memory consuming statements. The profiling library will record the operation of calling internal memory allocation. On average, one profiling data will be triggered for every 512KB of memory application.

Blocking analysis records the longest blocking goroutine operations, such as system calls, pipeline sending and receiving, and obtaining locks. Whenever goroutine is blocked by these operations, the profiling library records the corresponding events.

You only need to open one of the following flag parameters to generate various analysis files. Be careful when using multiple flag parameters at the same time, because one analysis operation may affect the analysis results of other items.

$ go test -cpuprofile=cpu.out $ go test -blockprofile=block.out $ go test -memprofile=mem.out

It is also easy to analyze some non test programs. The specific implementation method will be very different from whether the program is a short-time running gadget or a long-time running service. Profiling is particularly useful for long-running programs, so you can enable runtime profiling by calling Go's runtime API.

Once we have collected the sampling data for analysis, we can use pprof to analyze the data. This is a tool that comes with the Go toolbox, but it is not a daily tool. It corresponds to the go tool pprof command. The command has many features and options, but the most basic are two parameters: the executable that generates the profile and the corresponding profiling data.

In order to improve analysis efficiency and reduce space, the analysis log itself does not contain the name of the function; It contains only the address corresponding to the function. That is, pprof needs corresponding executable programs to interpret and analyze data. Although go test usually discards the temporary test program after the test is completed, when the analysis is enabled, the test program will be saved as foo.test file, in which the foo part corresponds to the name of the package to be tested.

The following command demonstrates how to collect and display a CPU analysis file. We choose a benchmark of net/http package as an example. It is often best to design specialized benchmarks for parts of business critical code. Because simple benchmarks can hardly represent business scenarios, we use the - run=NONE parameter to prohibit those simple tests.

$ go test -run=NONE -bench=ClientServerParallelTLS64 \ -cpuprofile=cpu.log net/http PASS BenchmarkClientServerParallelTLS64-8 1000 3141325 ns/op 143010 B/op 1747 allocs/op ok net/http 3.395s $ go tool pprof -text -nodecount=10 ./http.test cpu.log 2570ms of 3590ms total (71.59%) Dropped 129 nodes (cum <= 17.95ms) Showing top 10 nodes out of 166 (cum >= 60ms) flat flat% sum% cum cum% 1730ms 48.19% 48.19% 1750ms 48.75% crypto/elliptic.p256ReduceDegree 230ms 6.41% 54.60% 250ms 6.96% crypto/elliptic.p256Diff 120ms 3.34% 57.94% 120ms 3.34% math/big.addMulVVW 110ms 3.06% 61.00% 110ms 3.06% syscall.Syscall 90ms 2.51% 63.51% 1130ms 31.48% crypto/elliptic.p256Square 70ms 1.95% 65.46% 120ms 3.34% runtime.scanobject 60ms 1.67% 67.13% 830ms 23.12% crypto/elliptic.p256Mul 60ms 1.67% 68.80% 190ms 5.29% math/big.nat.montgomery 50ms 1.39% 70.19% 50ms 1.39% crypto/elliptic.p256ReduceCarry 50ms 1.39% 71.59% 60ms 1.67% crypto/elliptic.p256Sum

The parameter - text is used to specify the output format. Here, each line is a function, which is sorted according to the length of CPU use. The - nodecount=10 parameter limits the output of only the first 10 rows of results. For serious performance problems, this text format can basically help find out the cause.

This profile tells us that in the HTTPS benchmark, the crypto/elliptic.p256ReduceDegree function occupies nearly half of the CPU resources and accounts for a large proportion of the performance. In contrast, if a profile is mainly a function of memory allocation of the runtime package, reducing memory consumption may be an optimization strategy worth trying.

For more subtle problems, you may need to use pprof's graphical display function. This requires the GraphViz tool to be installed, which can be downloaded from http://www.graphviz.org Download. Parameter - web is used to generate a directed graph of functions, marked with information such as CPU usage and hottest functions.

In this section, we just have a brief look at the data analysis tools of go language. If you want to know more, you can read the article "profiting Go programs" on the official go blog.

Sample function

The third function specifically treated by go test is the Example function, which starts with Example. The sample function has no function parameters and return values. The following is an Example function corresponding to the IsPalindrome function:

func ExampleIsPalindrome() { fmt.Println(IsPalindrome("A man, a plan, a canal: Panama")) fmt.Println(IsPalindrome("palindrome")) // Output: // true // false }

The sample function has three uses:

1. As document
2. The sample function test will also be run when the test is executed using go test
3. Provide a real drill ground

One of the main functions of the sample function is as a document: the example of a package can demonstrate the usage of the function in a more concise and intuitive way, which is more direct and easy to understand than the text description, especially as a reminder or quick reference. An example function can also easily show the relationship between several types or functions belonging to the same interface. All documents must be associated to one place, just as a type or function declaration is unified into (one) package. At the same time, the sample functions and comments are different. The sample functions are real Go code and need to be checked by the compiler at compile time, so as to ensure that the sample code will not be disconnected when the source code is updated.

According to the suffix part of the sample function, godoc, a web document server, will associate the sample function with a specific function or the package itself. Therefore, the Example IsPalindrome sample function will be a part of the IsPalindrome function document, and the Example sample function will be a part of the package document.

The second use of the sample function is that the sample function test will also be run when go test executes the test. If the sample function contains comments in a format similar to / / Output: in the above example, the test tool will execute the sample function and check whether the standard Output of the sample function matches the comments.

The third purpose of the sample function is to provide a real drill ground. http://golang.org It is the document service provided by godoc. It uses Go Playground to allow users to edit and run each sample function online in the browser, as shown in Figure 11.4. This is usually the quickest way to learn how to use functions or Go language features.

Add new comment

0 comments