x/build: instances of all internet-facing tests failing

[dmitshur]

#!watchflakes
post <- repo == "build" && `https:\/\/maintner\.golang\.org\/.+: 502 Bad Gateway`

Issue created to collect instances where internet-facing tests fail with errors:

[...]
2022/07/28 00:33:59 syncing segment 136 did not succeed on attempt 3, will try again in 9s: https://maintner.golang.org/logs/136: 502 Bad Gateway
2022/07/28 00:34:08 Downloading 15228181 bytes of https://maintner.golang.org/logs/136 ...
2022/07/28 00:34:17 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs/136: 502 Bad Gateway"}
2022/07/28 00:34:17 syncing segment 136 did not succeed on attempt 4, will try again in 16s: https://maintner.golang.org/logs/136: 502 Bad Gateway
2022/07/28 00:34:33 Downloading 15228181 bytes of https://maintner.golang.org/logs/136 ...
2022/07/28 00:34:34 Unknown error type *errors.errorString: &errors.errorString{s:"corrupt download"}
2022/07/28 00:34:34 Corpus GetMutations: corrupt download
--- FAIL: TestCorpusCheck (112.99s)
    godata_test.go:52: getting corpus: corrupt download
2022/07/28 00:34:34 Loading data from log *maintner.netMutSource ...
2022/07/28 00:34:34 Downloading 15186069 bytes of https://maintner.golang.org/logs/136 ...
2022/07/28 00:34:35 wrote /root/.cache/golang-maintner/0136.growing.mutlog
2022/07/28 00:35:16 Reloaded data from log *maintner.netMutSource.
FAIL
FAIL	golang.org/x/build/maintner/godata	156.319s
[...]
--- FAIL: TestFindWork (9.05s)
    coordinator_test.go:309: bad http response: 500 Internal Server Error
2022/07/28 00:33:04 gRPCServer.ClearResults: bad status from dashboard: 400 ("400 Bad Request")
2022/07/28 00:33:04 gRPCServer.ClearResults: bad status from dashboard: 308 ("308 Permanent Redirect")
FAIL
FAIL	golang.org/x/build/cmd/coordinator	9.164s
[...]

• 2022-07-28T00:31:10-b32452c-462b78f/linux-amd64-longtest
• 2022-07-28T00:31:10-b32452c-462b78f/windows-amd64-longtest

[dmitshur]

The likely explanation here is that the k8s cluster was undergoing an upgrade, resulting in brief windows of service unavailability.

Let's reuse this issue to catch future instances, so that watchflakes doesn't keep opening new issues per package/test.

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- repo == "build" && `https:\/\/maintner\.golang\.org\/.+: 502 Bad Gateway`

2024-06-17 17:07 x_build-go1.21-windows-amd64-longtest build@def93586 release-branch.go1.21@74ac37e9 x/build/maintner/godata.TestCorpusCheck (log)

=== RUN   TestCorpusCheck
2024/06/17 10:16:19 Loading data from log *maintner.netMutSource ...
2024/06/17 10:16:28 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:16:28 fetching server segments did not succeed on attempt 1, will try again in 1s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:16:38 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:16:38 fetching server segments did not succeed on attempt 2, will try again in 4s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:16:51 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:16:51 fetching server segments did not succeed on attempt 3, will try again in 9s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:09 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:09 fetching server segments did not succeed on attempt 4, will try again in 16s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:34 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:34 Corpus GetMutations: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway
    godata_test.go:52: getting corpus: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway
--- FAIL: TestCorpusCheck (75.14s)

2024-06-17 17:07 x_build-go1.21-windows-amd64-longtest build@def93586 release-branch.go1.21@74ac37e9 x/build/maintner/maintnerd/maintapi.TestGetRef (log)

=== RUN   TestGetRef
2024/06/17 10:16:20 Loading data from log *maintner.netMutSource ...
2024/06/17 10:16:29 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:16:29 fetching server segments did not succeed on attempt 1, will try again in 1s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:16:39 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:16:39 fetching server segments did not succeed on attempt 2, will try again in 4s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:16:52 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:16:52 fetching server segments did not succeed on attempt 3, will try again in 9s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:10 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:10 fetching server segments did not succeed on attempt 4, will try again in 16s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:35 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:35 Corpus GetMutations: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway
    api_test.go:475: getting corpus: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway
--- FAIL: TestGetRef (75.12s)

2024-06-17 17:07 x_build-go1.22-darwin-amd64-longtest build@def93586 release-branch.go1.22@fe9b3c33 x/build/maintner/godata.TestCorpusCheck (log)

=== RUN   TestCorpusCheck
2024/06/17 10:16:26 Loading data from log *maintner.netMutSource ...
2024/06/17 10:16:35 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:16:35 fetching server segments did not succeed on attempt 1, will try again in 1s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:16:45 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:16:45 fetching server segments did not succeed on attempt 2, will try again in 4s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:16:58 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:16:58 fetching server segments did not succeed on attempt 3, will try again in 9s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:16 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:16 fetching server segments did not succeed on attempt 4, will try again in 16s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:41 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:41 Corpus GetMutations: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway
    godata_test.go:52: getting corpus: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway
--- FAIL: TestCorpusCheck (75.37s)

2024-06-17 17:07 x_build-gotip-windows-amd64-longtest build@def93586 go@8f5c6904 x/build/maintner/godata.TestGerritForeachNonChangeRef (log)

=== RUN   TestGerritForeachNonChangeRef
2024/06/17 10:16:43 Loading data from log *maintner.netMutSource ...
2024/06/17 10:16:52 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:16:52 fetching server segments did not succeed on attempt 1, will try again in 1s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:02 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:02 fetching server segments did not succeed on attempt 2, will try again in 4s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:15 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:15 fetching server segments did not succeed on attempt 3, will try again in 9s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:33 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:33 fetching server segments did not succeed on attempt 4, will try again in 16s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:58 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:58 Corpus GetMutations: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway
    godata_test.go:52: getting corpus: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway
--- FAIL: TestGerritForeachNonChangeRef (75.02s)

2024-06-17 17:07 x_build-gotip-windows-amd64-longtest build@def93586 go@8f5c6904 x/build/maintner/maintnerd/maintapi.TestTryBotStatus (log)

=== RUN   TestTryBotStatus
2024/06/17 10:16:43 Loading data from log *maintner.netMutSource ...
2024/06/17 10:16:52 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:16:52 fetching server segments did not succeed on attempt 1, will try again in 1s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:02 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:02 fetching server segments did not succeed on attempt 2, will try again in 4s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:15 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:15 fetching server segments did not succeed on attempt 3, will try again in 9s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:33 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:33 fetching server segments did not succeed on attempt 4, will try again in 16s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:58 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:58 Corpus GetMutations: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway
    api_test.go:475: getting corpus: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway
--- FAIL: TestTryBotStatus (75.04s)

— watchflakes