Skip to content

cmd/go: coverage profile should be cached with tests #23565

Closed
@bbrks

Description

@bbrks

As briefly discussed here: https://twitter.com/_rsc/status/956888213314068481

I don't see why Go shouldn't cache the results of -coverprofile when running tests, as test coverage shouldn't vary from run to run, given the same set of arguments.

What version of Go are you using (go version)?

go version go1.10rc1 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

$GOOS="darwin"
$GOARCH="amd64"

What did you do?

Call go test -coverprofile=coverprofile.out ./... multiple times.

What did you expect to see?

Test results cached between runs.

$ go test -coverprofile=coverprofile.out ./...
ok  	github.com/bbrks/tmp	2.893s	coverage: 46.1% of statements
$ go test -coverprofile=coverprofile.out ./...
ok  	github.com/bbrks/tmp	(cached)

What did you see instead?

Test results were not cached between runs.

$ go test -coverprofile=coverprofile.out ./...
ok  	github.com/bbrks/tmp	2.893s	coverage: 46.1% of statements
$ go test -coverprofile=coverprofile.out ./...
ok  	github.com/bbrks/tmp	2.893s	coverage: 46.1% of statements

Activity

changed the title [-]Coverage profile should be cached with tests[/-] [+]cmd/go: coverage profile should be cached with tests[/+] on Jan 26, 2018
added
NeedsDecisionFeedback is required from experts, contributors, and/or the community before a change can be made.
on Jan 26, 2018
added this to the Go1.11 milestone on Jan 26, 2018
ianlancetaylor

ianlancetaylor commented on Jan 26, 2018

@ianlancetaylor
Contributor

CC @rsc

added
NeedsFixThe path to resolution is known, but the work has not been done.
on Jan 29, 2018
removed
NeedsDecisionFeedback is required from experts, contributors, and/or the community before a change can be made.
on Jan 29, 2018
gopherbot

gopherbot commented on Jan 30, 2018

@gopherbot
Contributor

Change https://golang.org/cl/90955 mentions this issue: cmd/go: coverage profile use cache if the set of arguements equals

modified the milestones: Go1.11, Go1.12 on Jul 6, 2018
fgrosse

fgrosse commented on Aug 6, 2018

@fgrosse

I was hoping this change would make it into go1.11 but it seems its not working yet in go1.11beta3. Looking at the CL it seems it got stuck due to merge conflicts? Any chance we can pick up this issue and still include it in v11?

bbrks

bbrks commented on Aug 6, 2018

@bbrks
Author

@fgrosse Looks like it's being targeted for 1.12 now. I don't think it will be rescheduled this close to 1.11's release.

fgrosse

fgrosse commented on Aug 6, 2018

@fgrosse
nightlyone

nightlyone commented on Jan 24, 2019

@nightlyone
Contributor

Is this still planned for Go1.12 ? I guess not...

removed this from the Go1.12 milestone on Feb 12, 2019

82 remaining items

chudilka1

chudilka1 commented on Feb 21, 2025

@chudilka1

Right to a bull's eye from @ryancurrah

This change does not affect every Go developer. It does impact those who use Go in a professional environment.

The very same case, one of our pipelines ran up to 45 mins w/o caching, and up to 10 with it. So, it is not a minor feature, it is underevaluated here.

ryancurrah

ryancurrah commented on Feb 27, 2025

@ryancurrah
Contributor

CL 610564 got a thumbs up. Waiting on another review.

ryancurrah

ryancurrah commented on Mar 5, 2025

@ryancurrah
Contributor

@thanm sorry to bug you again. I had to rebase my change because there was a conflict in the docs. I ordered the flags alphabetically which requires your re-review. http://go-review.googlesource.com/c/go/+/610564/16/src/cmd/go/alldocs.go

CL: https://go-review.googlesource.com/c/go/+/610564

thanm

thanm commented on Mar 5, 2025

@thanm
Contributor

@ryancurrah I'll take a look. Thank you for your persistence in getting this CL moved forward, I believe it will be a valuable change to make.

added a commit that references this issue on Mar 6, 2025
6a4bc8d
ruoibmt

ruoibmt commented on Apr 17, 2025

@ruoibmt

@thanm On March 6, this modification was merged to master. Do you intend to update it to the upcoming minor version of 1.23 or 1.24?

thanm

thanm commented on Apr 18, 2025

@thanm
Contributor

It is not my intent to back-port this change. The general policy on back-porting is that the fix needs to address a serious problem (compiler or runtime crash, computing incorrect results, that sort of thing). Performance improvements typically don't meet those criteria. I could nominate the bug for back-porting, but I would be surprised if the release team approved it.

hihoak

hihoak commented on Jun 6, 2025

@hihoak

Hello guys,

Thanks a lot for resolving this issue.

I started testing this feature and encountered a problem that occurs when I use the go test command with the -coverpkg flag.

When coverage is collected with the -coverpkg flag and test result caching is enabled, the generated report may include lines that no longer exist. This happens because:

  • Each test package attempts to collect coverage for all packages matching the -coverpkg pattern.

  • If the test result is loaded from the cache, the coverage data may be outdated—especially if the test package does not directly or indirectly depend on the modified code, leaving the cache uninvalidated.

for example we have project with layout

Project layout

proj/
  some_func.go
  some_func_test.go
  sub/
    sub.go
    sub_test.go
  sum/
    sum.go

Files Content

some_func.go

package proj

import "proj/sum"

func SomeFunc(a, b int) int {
	if a == 0 && b == 0 {
		return 0
	}
	return sum.Sum(a, b)
}

sub.go

package sub

func Sub(a, b int) int {
	if a == 0 && b == 0 {
		return 0
	}
	return a - b
}

sum.go

package sum

func Sum(a, b int) int {
	if a == 0 {
		return b
	}
	return a + b
}

some_func_test.go

package proj

import (
	"github.com/stretchr/testify/require"
	"testing"
)

func Test_SomeFunc(t *testing.T) {
	t.Run("test1", func(t *testing.T) {
		require.Equal(t, 2, SomeFunc(1, 1))
	})
}

sub_test.go

package sub

import (
	"github.com/stretchr/testify/require"
	"testing"
)

func Test_Sub(t *testing.T) {
	t.Run("test_sub1", func(t *testing.T) {
		require.Equal(t, 0, Sub(1, 1))
	})
}

Coverage result of this tests

mode: set
proj/some_func.go:5.29,6.22 1 1
proj/some_func.go:6.22,8.3 1 0
proj/some_func.go:9.2,9.22 1 1
proj/sub/sub.go:3.24,4.22 1 0
proj/sub/sub.go:4.22,6.3 1 0
proj/sub/sub.go:7.2,7.14 1 0
proj/sum/sum.go:3.24,4.12 1 1
proj/sum/sum.go:4.12,6.3 1 0
proj/sum/sum.go:7.2,7.14 1 1
proj/some_func.go:5.29,6.22 1 0
proj/some_func.go:6.22,8.3 1 0
proj/some_func.go:9.2,9.22 1 0
proj/sub/sub.go:3.24,4.22 1 1
proj/sub/sub.go:4.22,6.3 1 0
proj/sub/sub.go:7.2,7.14 1 1
proj/sum/sum.go:3.24,4.12 1 0
proj/sum/sum.go:4.12,6.3 1 0
proj/sum/sum.go:7.2,7.14 1 0
proj/sum/sum.go:3.24,4.12 1 0
proj/sum/sum.go:4.12,6.3 1 0
proj/sum/sum.go:7.2,7.14 1 0

Change a bit sub.go

sub.go

package sub

func Sub(a, b int) int {
	if a == 0 && b == 0 || a == -100 {
		return 0
	}
	return a - b
}

Coverage result after change

mode: set
proj/some_func.go:5.29,6.22 1 1
proj/some_func.go:6.22,8.3 1 0
proj/some_func.go:9.2,9.22 1 1
proj/sub/sub.go:3.24,4.22 1 0
proj/sub/sub.go:4.22,6.3 1 0
proj/sub/sub.go:7.2,7.14 1 0
proj/sum/sum.go:3.24,4.12 1 1
proj/sum/sum.go:4.12,6.3 1 0
proj/sum/sum.go:7.2,7.14 1 1
proj/some_func.go:5.29,6.22 1 0
proj/some_func.go:6.22,8.3 1 0
proj/some_func.go:9.2,9.22 1 0
proj/sub/sub.go:3.24,4.35 1 1
proj/sub/sub.go:4.35,6.3 1 0
proj/sub/sub.go:7.2,7.14 1 1
proj/sum/sum.go:3.24,4.12 1 0
proj/sum/sum.go:4.12,6.3 1 0
proj/sum/sum.go:7.2,7.14 1 0
proj/sum/sum.go:3.24,4.12 1 0
proj/sum/sum.go:4.12,6.3 1 0
proj/sum/sum.go:7.2,7.14 1 0

Conclusion

so there are now 2 versions of sub.go:3.24 line in one report and proj/sub/sub.go:3.24,4.22 1 0 doesn't exist

as a result it can breaks up coverage tools to generate cobertura coverage

ryancurrah

ryancurrah commented on Jun 6, 2025

@ryancurrah
Contributor

Can you provide the go test commands your running to get these results.

hihoak

hihoak commented on Jun 6, 2025

@hihoak

first run

go test -coverpkg=proj/... -coverprofile=cover.out ./proj/...
ok  	proj	(cached)	coverage: 44.4% of statements in proj/...
ok  	proj/sub	(cached)	coverage: 22.2% of statements in proj/...
	proj/sum		coverage: 0.0% of statements

second run

go test -coverpkg=proj/... -coverprofile=cover.out ./proj/...
ok  	proj	(cached)	coverage: 44.4% of statements in proj/...
ok  	proj/sub	0.005s	coverage: 22.2% of statements in proj/...
	proj/sum		coverage: 0.0% of statements
ryancurrah

ryancurrah commented on Jun 6, 2025

@ryancurrah
Contributor

My immediate thought here is to disable coverage caching when -coverpkg is used. Otherwise we will need to figure out how to determine if any of the packages found in coverpkg changed. Which I'm not sure how to do. If none of these options sound good we should revert this CL.

hihoak

hihoak commented on Jun 6, 2025

@hihoak

All of these "phantom" lines always have 0 hits; otherwise, they would be invalidated (since they would have been imported by modified code).

This fact can be used to ignore them in tools that generate reports.

That said, I agree, this behavior does seem unintuitive and non-obvious at first glance

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    GoCommandcmd/goNeedsFixThe path to resolution is known, but the work has not been done.

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

      Participants

      @rsc@karelbilek@dinvlad@andybons@fgrosse

      Issue actions

        cmd/go: coverage profile should be cached with tests · Issue #23565 · golang/go