Skip to content

cmd/go: coverage profile should be cached with tests #23565

New issue
New issue
Closed
#65657
Closed
cmd/go: coverage profile should be cached with tests#23565
#65657
Labels
GoCommandcmd/goNeedsFixThe path to resolution is known, but the work has not been done.
Milestone
 
Backlog
@bbrks

Description

@bbrks
bbrks
opened on Jan 26, 2018 · edited by bbrks

As briefly discussed here: https://twitter.com/_rsc/status/956888213314068481

I don't see why Go shouldn't cache the results of -coverprofile when running tests, as test coverage shouldn't vary from run to run, given the same set of arguments.

What version of Go are you using (go version)?

go version go1.10rc1 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

$GOOS="darwin"
$GOARCH="amd64"

What did you do?

Call go test -coverprofile=coverprofile.out ./... multiple times.

What did you expect to see?

Test results cached between runs.

$ go test -coverprofile=coverprofile.out ./...
ok  	github.com/bbrks/tmp	2.893s	coverage: 46.1% of statements
$ go test -coverprofile=coverprofile.out ./...
ok  	github.com/bbrks/tmp	(cached)

What did you see instead?

Test results were not cached between runs.

$ go test -coverprofile=coverprofile.out ./...
ok  	github.com/bbrks/tmp	2.893s	coverage: 46.1% of statements
$ go test -coverprofile=coverprofile.out ./...
ok  	github.com/bbrks/tmp	2.893s	coverage: 46.1% of statements

Activity

ALTree
changed the title [-]Coverage profile should be cached with tests[/-] [+]cmd/go: coverage profile should be cached with tests[/+] on Jan 26, 2018
ALTree
added
NeedsDecisionFeedback is required from experts, contributors, and/or the community before a change can be made.
on Jan 26, 2018
ALTree
added this to the Go1.11 milestone on Jan 26, 2018
ianlancetaylor

ianlancetaylor commented on Jan 26, 2018

@ianlancetaylor
ianlancetaylor
on Jan 26, 2018
Contributor

CC @rsc

rsc
added
NeedsFixThe path to resolution is known, but the work has not been done.
on Jan 29, 2018
gopherbot
removed
NeedsDecisionFeedback is required from experts, contributors, and/or the community before a change can be made.
on Jan 29, 2018
gopherbot

gopherbot commented on Jan 30, 2018

@gopherbot
gopherbot
on Jan 30, 2018
Contributor

Change https://golang.org/cl/90955 mentions this issue: cmd/go: coverage profile use cache if the set of arguements equals

ianlancetaylor
modified the milestones: Go1.11, Go1.12 on Jul 6, 2018
fgrosse

fgrosse commented on Aug 6, 2018

@fgrosse
fgrosse
on Aug 6, 2018

I was hoping this change would make it into go1.11 but it seems its not working yet in go1.11beta3. Looking at the CL it seems it got stuck due to merge conflicts? Any chance we can pick up this issue and still include it in v11?

bbrks

bbrks commented on Aug 6, 2018

@bbrks
bbrks
on Aug 6, 2018
Author

@fgrosse Looks like it's being targeted for 1.12 now. I don't think it will be rescheduled this close to 1.11's release.

fgrosse

fgrosse commented on Aug 6, 2018

@fgrosse
fgrosse
on Aug 6, 2018 via email
krhubert
mentioned this on Nov 7, 2018
nightlyone

nightlyone commented on Jan 24, 2019

@nightlyone
nightlyone
on Jan 24, 2019
Contributor

Is this still planned for Go1.12 ? I guess not...

andybons
removed this from the Go1.12 milestone on Feb 12, 2019

82 remaining items

chudilka1

chudilka1 commented on Feb 21, 2025

@chudilka1
chudilka1
on Feb 21, 2025

Right to a bull's eye from @ryancurrah

This change does not affect every Go developer. It does impact those who use Go in a professional environment.

The very same case, one of our pipelines ran up to 45 mins w/o caching, and up to 10 with it. So, it is not a minor feature, it is underevaluated here.

ryancurrah

ryancurrah commented on Feb 27, 2025

@ryancurrah
ryancurrah
on Feb 27, 2025 · edited by ryancurrah
Contributor

CL 610564 got a thumbs up. Waiting on another review.

ryancurrah

ryancurrah commented on Mar 5, 2025

@ryancurrah
ryancurrah
on Mar 5, 2025 · edited by ryancurrah
Contributor

@thanm sorry to bug you again. I had to rebase my change because there was a conflict in the docs. I ordered the flags alphabetically which requires your re-review. http://go-review.googlesource.com/c/go/+/610564/16/src/cmd/go/alldocs.go

CL: https://go-review.googlesource.com/c/go/+/610564

thanm

thanm commented on Mar 5, 2025

@thanm
thanm
on Mar 5, 2025
Contributor

@ryancurrah I'll take a look. Thank you for your persistence in getting this CL moved forward, I believe it will be a valuable change to make.

gopherbot
closed this as completedin 6a4bc8don Mar 6, 2025
pull
added a commit that references this issue on Mar 6, 2025

cmd/go: cache coverage profile with tests

6a4bc8d
ruoibmt

ruoibmt commented on Apr 17, 2025

@ruoibmt
ruoibmt
on Apr 17, 2025

@thanm On March 6, this modification was merged to master. Do you intend to update it to the upcoming minor version of 1.23 or 1.24?

thanm

thanm commented on Apr 18, 2025

@thanm
thanm
on Apr 18, 2025
Contributor

It is not my intent to back-port this change. The general policy on back-porting is that the fix needs to address a serious problem (compiler or runtime crash, computing incorrect results, that sort of thing). Performance improvements typically don't meet those criteria. I could nominate the bug for back-porting, but I would be surprised if the release team approved it.

taratorio
mentioned this on Apr 23, 2025
yperbasis
mentioned this on Apr 23, 2025
hihoak

hihoak commented on Jun 6, 2025

@hihoak
hihoak
on Jun 6, 2025 · edited by hihoak

Hello guys,

Thanks a lot for resolving this issue.

I started testing this feature and encountered a problem that occurs when I use the go test command with the -coverpkg flag.

When coverage is collected with the -coverpkg flag and test result caching is enabled, the generated report may include lines that no longer exist. This happens because:

  • Each test package attempts to collect coverage for all packages matching the -coverpkg pattern.

  • If the test result is loaded from the cache, the coverage data may be outdated—especially if the test package does not directly or indirectly depend on the modified code, leaving the cache uninvalidated.

for example we have project with layout

Project layout

proj/
  some_func.go
  some_func_test.go
  sub/
    sub.go
    sub_test.go
  sum/
    sum.go

Files Content

some_func.go

package proj

import "proj/sum"

func SomeFunc(a, b int) int {
	if a == 0 && b == 0 {
		return 0
	}
	return sum.Sum(a, b)
}

sub.go

package sub

func Sub(a, b int) int {
	if a == 0 && b == 0 {
		return 0
	}
	return a - b
}

sum.go

package sum

func Sum(a, b int) int {
	if a == 0 {
		return b
	}
	return a + b
}

some_func_test.go

package proj

import (
	"github.com/stretchr/testify/require"
	"testing"
)

func Test_SomeFunc(t *testing.T) {
	t.Run("test1", func(t *testing.T) {
		require.Equal(t, 2, SomeFunc(1, 1))
	})
}

sub_test.go

package sub

import (
	"github.com/stretchr/testify/require"
	"testing"
)

func Test_Sub(t *testing.T) {
	t.Run("test_sub1", func(t *testing.T) {
		require.Equal(t, 0, Sub(1, 1))
	})
}

Coverage result of this tests

mode: set
proj/some_func.go:5.29,6.22 1 1
proj/some_func.go:6.22,8.3 1 0
proj/some_func.go:9.2,9.22 1 1
proj/sub/sub.go:3.24,4.22 1 0
proj/sub/sub.go:4.22,6.3 1 0
proj/sub/sub.go:7.2,7.14 1 0
proj/sum/sum.go:3.24,4.12 1 1
proj/sum/sum.go:4.12,6.3 1 0
proj/sum/sum.go:7.2,7.14 1 1
proj/some_func.go:5.29,6.22 1 0
proj/some_func.go:6.22,8.3 1 0
proj/some_func.go:9.2,9.22 1 0
proj/sub/sub.go:3.24,4.22 1 1
proj/sub/sub.go:4.22,6.3 1 0
proj/sub/sub.go:7.2,7.14 1 1
proj/sum/sum.go:3.24,4.12 1 0
proj/sum/sum.go:4.12,6.3 1 0
proj/sum/sum.go:7.2,7.14 1 0
proj/sum/sum.go:3.24,4.12 1 0
proj/sum/sum.go:4.12,6.3 1 0
proj/sum/sum.go:7.2,7.14 1 0

Change a bit sub.go

sub.go

package sub

func Sub(a, b int) int {
	if a == 0 && b == 0 || a == -100 {
		return 0
	}
	return a - b
}

Coverage result after change

mode: set
proj/some_func.go:5.29,6.22 1 1
proj/some_func.go:6.22,8.3 1 0
proj/some_func.go:9.2,9.22 1 1
proj/sub/sub.go:3.24,4.22 1 0
proj/sub/sub.go:4.22,6.3 1 0
proj/sub/sub.go:7.2,7.14 1 0
proj/sum/sum.go:3.24,4.12 1 1
proj/sum/sum.go:4.12,6.3 1 0
proj/sum/sum.go:7.2,7.14 1 1
proj/some_func.go:5.29,6.22 1 0
proj/some_func.go:6.22,8.3 1 0
proj/some_func.go:9.2,9.22 1 0
proj/sub/sub.go:3.24,4.35 1 1
proj/sub/sub.go:4.35,6.3 1 0
proj/sub/sub.go:7.2,7.14 1 1
proj/sum/sum.go:3.24,4.12 1 0
proj/sum/sum.go:4.12,6.3 1 0
proj/sum/sum.go:7.2,7.14 1 0
proj/sum/sum.go:3.24,4.12 1 0
proj/sum/sum.go:4.12,6.3 1 0
proj/sum/sum.go:7.2,7.14 1 0

Conclusion

so there are now 2 versions of sub.go:3.24 line in one report and proj/sub/sub.go:3.24,4.22 1 0 doesn't exist

as a result it can breaks up coverage tools to generate cobertura coverage

ryancurrah

ryancurrah commented on Jun 6, 2025

@ryancurrah
ryancurrah
on Jun 6, 2025
Contributor

Can you provide the go test commands your running to get these results.

hihoak

hihoak commented on Jun 6, 2025

@hihoak
hihoak
on Jun 6, 2025

first run

go test -coverpkg=proj/... -coverprofile=cover.out ./proj/...
ok  	proj	(cached)	coverage: 44.4% of statements in proj/...
ok  	proj/sub	(cached)	coverage: 22.2% of statements in proj/...
	proj/sum		coverage: 0.0% of statements

second run

go test -coverpkg=proj/... -coverprofile=cover.out ./proj/...
ok  	proj	(cached)	coverage: 44.4% of statements in proj/...
ok  	proj/sub	0.005s	coverage: 22.2% of statements in proj/...
	proj/sum		coverage: 0.0% of statements
ryancurrah

ryancurrah commented on Jun 6, 2025

@ryancurrah
ryancurrah
on Jun 6, 2025 · edited by ryancurrah
Contributor

My immediate thought here is to disable coverage caching when -coverpkg is used. Otherwise we will need to figure out how to determine if any of the packages found in coverpkg changed. Which I'm not sure how to do. If none of these options sound good we should revert this CL.

hihoak

hihoak commented on Jun 6, 2025

@hihoak
hihoak
on Jun 6, 2025

All of these "phantom" lines always have 0 hits; otherwise, they would be invalidated (since they would have been imported by modified code).

This fact can be used to ignore them in tools that generate reports.

That said, I agree, this behavior does seem unintuitive and non-obvious at first glance

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    GoCommandcmd/goNeedsFixThe path to resolution is known, but the work has not been done.

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

      Development

      Participants

      @rsc@karelbilek@dinvlad@andybons@fgrosse

      Issue actions
        cmd/go: coverage profile should be cached with tests · Issue #23565 · golang/go