x/build/cmd/golangbuild: potential send on closed channel during cleanup

[dmitshur]

LUCI build https://ci.chromium.org/b/8737704283378549569 uncovers a potential panic in a golangbuild dependency during cleanup:

[...]
[I2024-09-04T18:03:18.887542Z 266889 0 butler.go:395] adding stream                               {"stream":"u/step/22/log/1"}
[I2024-09-04T18:03:18.887829Z 266889 0 butler.go:395] adding stream                               {"stream":"u/step/22/log/2"}
[I2024-09-04T18:04:51.847543Z 266889 0 butler.go:395] adding stream                               {"stream":"u/step/22/log/3"}
[I2024-09-04T18:04:52.849612Z 266889 0 main.go:797] Final build status from subprocess: STATUS_UNSPECIFIED
[I2024-09-04T18:04:52.849715Z 266889 0 cleanup.go:37] running cleanup "butler.Activate"
[I2024-09-04T18:04:52.849733Z 266889 0 cleanup.go:42] cleanup "butler.Activate" succeeded
[I2024-09-04T18:04:52.849743Z 266889 0 cleanup.go:37] running cleanup "flush u/"
[I2024-09-04T18:04:52.849777Z 266889 0 host.go:126] waiting up to 30s for user logs to flush
[I2024-09-04T18:04:56.107788Z 266889 0 cleanup.go:42] cleanup "flush u/" succeeded
[I2024-09-04T18:04:56.107875Z 266889 0 cleanup.go:37] running cleanup "buildmerge spy"
[I2024-09-04T18:04:56.113395Z 266889 0 host.go:106] waiting for buildmerge spy to finish
[I2024-09-04T18:04:56.114393Z 266889 0 options.go:227] dropped 3 items over 3m27.716270372s
[I2024-09-04T18:04:56.116713Z 266889 0 cleanup.go:42] cleanup "buildmerge spy" succeeded
[I2024-09-04T18:04:56.117164Z 266889 0 cleanup.go:37] running cleanup "butler"
[I2024-09-04T18:04:56.225876Z 266889 0 butler.go:265] Message output has closed                   {"stats":"{SentBytes:1105439 SentMessages:22 DiscardedMessages:0 Errors:0}"}
[I2024-09-04T18:04:56.225917Z 266889 0 cleanup.go:42] cleanup "butler" succeeded
[I2024-09-04T18:04:56.225929Z 266889 0 cleanup.go:37] running cleanup "LUCI_CONTEXT"
[I2024-09-04T18:04:56.225985Z 266889 0 cleanup.go:42] cleanup "LUCI_CONTEXT" succeeded
[I2024-09-04T18:04:56.225994Z 266889 0 cleanup.go:37] running cleanup "authctx"
[I2024-09-04T18:04:56.226333Z 266889 0 cleanup.go:42] cleanup "authctx" succeeded
[I2024-09-04T18:04:56.226347Z 266889 0 cleanup.go:37] running cleanup "restoreEnv"
[I2024-09-04T18:04:56.226382Z 266889 0 cleanup.go:42] cleanup "restoreEnv" succeeded
[I2024-09-04T18:04:56.226389Z 266889 0 cleanup.go:37] running cleanup "cleanupComplete"
[I2024-09-04T18:04:56.226421Z 266889 0 cleanup.go:42] cleanup "cleanupComplete" succeeded
[I2024-09-04T18:04:59.554303Z 266889 0 options.go:227] dropped 2 items over 3.439905472s
panic: send on closed channel

goroutine 666 [running]:
go.chromium.org/luci/common/sync/dispatcher.(*coordinatorState).sendBatches.func2()
	go.chromium.org/luci/common/sync/dispatcher/coordinator.go:125 +0x68
created by go.chromium.org/luci/common/sync/dispatcher.(*coordinatorState).sendBatches in goroutine 139
	go.chromium.org/luci/common/sync/dispatcher/coordinator.go:124 +0x3c8

Visible in the raw output of the failed backend task. Originally reported by watchflakes in #69266.

[gabyhelp]

Related Issues and Documentation

• x/text/message/pipeline: TestFullCycle/ssa fails with "panic: Cannot range over: func(yield func(E) bool)" on Go tip #67348 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[dmitshur]

We have a report of another instance at https://ci.chromium.org/b/8735636356859002161/infra, this time on windows/amd64. So this likely applies to all platforms rather than linux/ppc64le specifically.

[riannucci]

Hi! The CL to fix this is here - https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/5896786

It is, indeed, not specific to any platform :). Thanks for raising this issue!

[dmitshur]

Thanks @riannucci! The aforementioned CL is submitted and released by now, so closing as fixed.