x/perf/cmd/benchstat: no output

[phenpessoa]

What version of Go are you using (go version)?

$ go version
go version go1.20.1 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\pedro\AppData\Local\go-build
set GOENV=C:\Users\pedro\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\pedro\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\pedro\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.20.1
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\test\go.mod
set GOWORK=
set CGO_CFLAGS=-O2 -g
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-O2 -g
set CGO_FFLAGS=-O2 -g
set CGO_LDFLAGS=-O2 -g
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=C:\Users\pedro\AppData\Local\Temp\go-build281116068=/tmp/go-build -gno-record-gcc-switches

What did you do?

go install golang.org/x/perf/cmd/...@latest
benchstat old.txt new.txt

What did you expect to see?

The benchstat output

What did you see instead?

No output at all

If I run benchstat only with no arguments I get:

Usage: benchstat [flags] inputs...

benchstat computes statistical summaries and A/B comparisons of Go
benchmarks. It shows benchmark medians in a table with a row for each
benchmark and a column for each input file. If there is more than one
input file, it also shows A/B comparisons between the files. If a
difference is likely to be noise, it shows "~".

For details, see https://pkg.go.dev/golang.org/x/perf/cmd/benchstat.
  -alpha α
        consider change significant if p < α (default 0.05)
  -col projection
        split results into columns by distinct values of projection (default ".file")
  -confidence level
        confidence level for ranges (default 0.95)
  -filter query
        use only benchmarks matching benchfilter query (default "*")
  -format format
        print results in format:
          text - plain text
          csv  - comma-separated values (warnings will be written to stderr)
         (default "text")
  -ignore keys
        ignore variations in keys
  -row projection
        split results into rows by distinct values of projection (default ".fullname")
  -table projection
        split results into tables by distinct values of projection (default ".config")

Meaning benchstat is properly installed

I also tried:

benchstat old.txt new.txt > output.txt

Even though the output.txt file was created, it was empty.

Files used

old.txt Content

goos: windows
goarch: amd64
pkg: pptests/invites
cpu: Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
BenchmarkInvitesInString-16             	  742873	      1669 ns/op	     360 B/op	       6 allocs/op
BenchmarkInvitesInString-16             	  703620	      1736 ns/op	     360 B/op	       6 allocs/op
BenchmarkInvitesInString-16             	  691200	      1710 ns/op	     360 B/op	       6 allocs/op
BenchmarkInvitesInString-16             	  732019	      1716 ns/op	     360 B/op	       6 allocs/op
BenchmarkInvitesInString-16             	  686360	      1718 ns/op	     360 B/op	       6 allocs/op
BenchmarkInvitesInString-16             	  726889	      1723 ns/op	     360 B/op	       6 allocs/op
BenchmarkInvitesInString-16             	  690158	      1722 ns/op	     360 B/op	       6 allocs/op
BenchmarkInvitesInString-16             	  779018	      1708 ns/op	     360 B/op	       6 allocs/op
BenchmarkInvitesInString-16             	  778911	      1721 ns/op	     360 B/op	       6 allocs/op
BenchmarkInvitesInString-16             	  631538	      1731 ns/op	     360 B/op	       6 allocs/op
BenchmarkInvitesInString_Parallel-16    	  493700	      2362 ns/op	     360 B/op	       6 allocs/op
BenchmarkInvitesInString_Parallel-16    	  525888	      2312 ns/op	     360 B/op	       6 allocs/op
BenchmarkInvitesInString_Parallel-16    	  514244	      2337 ns/op	     360 B/op	       6 allocs/op
BenchmarkInvitesInString_Parallel-16    	  514758	      2319 ns/op	     360 B/op	       6 allocs/op
BenchmarkInvitesInString_Parallel-16    	  514744	      2336 ns/op	     360 B/op	       6 allocs/op
BenchmarkInvitesInString_Parallel-16    	  514729	      2331 ns/op	     360 B/op	       6 allocs/op
BenchmarkInvitesInString_Parallel-16    	  525934	      2330 ns/op	     360 B/op	       6 allocs/op
BenchmarkInvitesInString_Parallel-16    	  503484	      2348 ns/op	     360 B/op	       6 allocs/op
BenchmarkInvitesInString_Parallel-16    	  503995	      2328 ns/op	     360 B/op	       6 allocs/op
BenchmarkInvitesInString_Parallel-16    	  345625	      3357 ns/op	     360 B/op	       6 allocs/op
PASS
ok  	pptests/invites	29.052s

new.txt Content

goos: windows
goarch: amd64
pkg: pptests/invites
cpu: Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
BenchmarkInvitesInString-16             	 3589579	       328.7 ns/op	     320 B/op	       1 allocs/op
BenchmarkInvitesInString-16             	 3412801	       337.5 ns/op	     320 B/op	       1 allocs/op
BenchmarkInvitesInString-16             	 3455905	       337.1 ns/op	     320 B/op	       1 allocs/op
BenchmarkInvitesInString-16             	 3458821	       347.1 ns/op	     320 B/op	       1 allocs/op
BenchmarkInvitesInString-16             	 3470880	       344.7 ns/op	     320 B/op	       1 allocs/op
BenchmarkInvitesInString-16             	 3436590	       348.3 ns/op	     320 B/op	       1 allocs/op
BenchmarkInvitesInString-16             	 3444130	       346.9 ns/op	     320 B/op	       1 allocs/op
BenchmarkInvitesInString-16             	 3448120	       345.1 ns/op	     320 B/op	       1 allocs/op
BenchmarkInvitesInString-16             	 3458286	       347.2 ns/op	     320 B/op	       1 allocs/op
BenchmarkInvitesInString-16             	 3359350	       350.2 ns/op	     320 B/op	       1 allocs/op
BenchmarkInvitesInString_Parallel-16    	 2404923	       490.9 ns/op	     320 B/op	       1 allocs/op
BenchmarkInvitesInString_Parallel-16    	 2478717	       488.5 ns/op	     320 B/op	       1 allocs/op
BenchmarkInvitesInString_Parallel-16    	 2458682	       483.2 ns/op	     320 B/op	       1 allocs/op
BenchmarkInvitesInString_Parallel-16    	 2473642	       486.4 ns/op	     320 B/op	       1 allocs/op
BenchmarkInvitesInString_Parallel-16    	 2451098	       485.3 ns/op	     320 B/op	       1 allocs/op
BenchmarkInvitesInString_Parallel-16    	 2453709	       492.0 ns/op	     320 B/op	       1 allocs/op
BenchmarkInvitesInString_Parallel-16    	 2491608	       480.1 ns/op	     320 B/op	       1 allocs/op
BenchmarkInvitesInString_Parallel-16    	 2451085	       491.1 ns/op	     320 B/op	       1 allocs/op
BenchmarkInvitesInString_Parallel-16    	 2494173	       492.6 ns/op	     320 B/op	       1 allocs/op
BenchmarkInvitesInString_Parallel-16    	 2448741	       499.1 ns/op	     320 B/op	       1 allocs/op
PASS
ok  	pptests/invites	35.760s

[seankhliao]

can you provide zipped up versions of your input?

[phenpessoa]

@seankhliao sure

inputs.zip

[seankhliao]

That's a utf16le encoded text file, the tools should only work on utf8

[phenpessoa]

I noticed that if I change this line (https://github.com/golang/perf/blob/801c7ef9e5c5eb8fb97c176023267056756fc52d/benchfmt/reader.go#L151) to:

line := bytes.ReplaceAll(r.s.Bytes(), null, nil)

Where null == []byte{0}

It works properly

[phenpessoa]

That's a utf16le encoded text file, the tools should only work on utf8

I'm generating the files using go test -bench . -count 10 > old.txt and go test -bench . -count 10 > new.txt. So I'm not sure why it is generating a utf16le file?

[seankhliao]

that would be a powershell issue https://stackoverflow.com/questions/40098771/changing-powershells-default-output-encoding-to-utf-8

[phenpessoa]

Thanks, that answers it!