Golang Benchmark why different results?

Question

go

Hello
I'm learning the go language and got to the point of using benchmarks.
I have a popcount module (yes, from The Go Programming Language book) with 4 functions.
go version 1.6, I am writing in intellij idea 2016
The question itself, if I run a test to check the function's running time, then I have results

PASS
BenchmarkPopCount1-8            200000000                6.57 ns/op
BenchmarkPopCountByLoop-8       100000000               13.0 ns/op
BenchmarkPopCountByShifting-8   20000000                63.9 ns/op
BenchmarkPopCountByClearing-8   1000000000               2.41 ns/op
ok      github.com/user/popcount        9.110s

And if I run testing for coverage, then the results

PASS
BenchmarkPopCount1-8         	100000000	        13.0 ns/op
BenchmarkPopCountByLoop-8    	20000000	        79.6 ns/op
BenchmarkPopCountByShifting-8	 1000000	      1079 ns/op
BenchmarkPopCountByClearing-8	50000000	        27.6 ns/op
coverage: 100.0% of statements
ok  	github.com/user/popcount	7.157s

What could be the reason for such a large increase in time for the BenchmarkPopCountByShifting function ?
her code

func PopCountByShifting(x uint64) int {
  n := 0
  shifted := x
  for i := uint(0); i < 64; i++ {
    // get left byte
    if ((shifted & 1) == 1) {
      n++
    }
    shifted >>= 1
  }
  return n
}

Answer 1

During the calculation of "coverage" part of the resources is spent on this task and it does not have to be efficient. Therefore, the benchmark does not receive resources at this time and may generally show incorrect measurements, since background procedures can be executed and load the CPU unevenly.
When benchmarking, it is desirable:
Are benchmarks needed? Benchmarks are needed to reject very bad results or choose between two very similar ones . They also give an idea of ​​how much is spent on each iteration of the code that you are testing.
An example of a simple benchmark with initialization placed outside the timer:

package conf // <-- здесь название библиотеки для которой предназначены бенчмарки

import (
    "testing"
)

func BenchmarkLoadGo(b *testing.B) {
    conf := NewConfig()
    conf.SetDefaultFile("properties")
    conf.SetDefaultCatalog("./testdata") // default "./configs"

    b.ResetTimer() // <-- обнуляем таймер что бы вся "сложная" и "единомоментная" работа не попала в подсчёт

    for i := 0; i < b.N; i++ {
        go conf.GetSection("server_1", "storage") // <-- в данном случае тестируется сценарий в конкурентном виде, но чаще всего этого не требуется и конкретно в том коде откуда взят бенчмарк есть отдельный бенч для не конкурентного доступа и сравнивая их можно иметь представление о том как оно себя поведёт в обоих сценариях.
    }
}

Answer 2

And who guarantees that different things will give the same results ?
As far as I remember:
To estimate the coverage, the utility takes the source code, copies it to a temporary directory, embeds its calls inside ....
That is, this is not the same source code already.
Benchmarks are needed in order to find the bottleneck in the program. That is, only for relative measurements of the program with itself, with the same program, run / copied in exactly the same conditions. And not in order to get some absolute numbers that are suitable for comparison with something else.