x/telemetry/internal/upload: TestRun_Concurrent failures

[gopherbot]

#!watchflakes
default <- pkg == "golang.org/x/telemetry/internal/upload" && test == "TestRun_Concurrent"

Issue created automatically to collect these failures.

Example (log):

=== RUN   TestRun_Concurrent
    run_test.go:66: 17:16:43.368589 findwork.go:35: Finding work: mode on, asof 2023-05-31 00:00:00 +0000 UTC
    run_test.go:66: 17:16:43.368622 findwork.go:35: Finding work: mode on, asof 2023-05-31 00:00:00 +0000 UTC
    run_test.go:66: 17:16:43.368646 findwork.go:35: Finding work: mode on, asof 2023-05-31 00:00:00 +0000 UTC
    run_test.go:66: 17:16:43.369582 findwork.go:35: Finding work: mode on, asof 2023-05-31 00:00:00 +0000 UTC
    run_test.go:66: 17:16:43.369752 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-05-08.v1.count
    run_test.go:66: 17:16:43.369861 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-05-08.v1.count
    run_test.go:66: 17:16:43.369616 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-05-08.v1.count
    run_test.go:66: 17:16:43.370231 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-05-09.v1.count
    run_test.go:66: 17:16:43.370245 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-05-09.v1.count
...
    run_test.go:66: 17:16:43.388046 reports.go:113: remove /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2569094852/001/local/upload.test-devel-darwin-amd64-2024-05-21.v1.count: no such file or directory failed to remove /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2569094852/001/local/upload.test-devel-darwin-amd64-2024-05-21.v1.count
    run_test.go:66: 17:16:43.388069 reports.go:64: Failed to create report for 2024-05-23: local report /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2569094852/001/local/local.2024-05-23.json already exists
    run_test.go:66: 17:16:43.388083 run.go:157: Uploading 0 reports
    run_test.go:66: 17:16:43.387658 reports.go:113: remove /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2569094852/001/local/upload.test-devel-darwin-amd64-2024-05-21.v1.count: no such file or directory failed to remove /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2569094852/001/local/upload.test-devel-darwin-amd64-2024-05-21.v1.count
    run_test.go:66: 17:16:43.388107 reports.go:64: Failed to create report for 2024-05-23: local report /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2569094852/001/local/local.2024-05-23.json already exists
    run_test.go:66: 17:16:43.388133 run.go:157: Uploading 1 reports
    run_test.go:66: 17:16:43.389401 upload.go:111: Uploaded 2024-05-23.json to "http://127.0.0.1:51325/2024-05-23"
    run_test.go:66: 17:16:43.389493 upload.go:111: Uploaded 2024-05-16.json to "http://127.0.0.1:51325/2024-05-16"
    run_test.go:593: uploaded 13 total observations, want 14
--- FAIL: TestRun_Concurrent (0.46s)

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "golang.org/x/telemetry/internal/upload" && test == "TestRun_Concurrent"

2024-05-22 23:36 x_telemetry-go1.21-darwin-amd64_13 telemetry@39ace7a4 release-branch.go1.21@9488a444 x/telemetry/internal/upload.TestRun_Concurrent (log)

=== RUN   TestRun_Concurrent
    run_test.go:66: 17:16:43.368589 findwork.go:35: Finding work: mode on, asof 2023-05-31 00:00:00 +0000 UTC
    run_test.go:66: 17:16:43.368622 findwork.go:35: Finding work: mode on, asof 2023-05-31 00:00:00 +0000 UTC
    run_test.go:66: 17:16:43.368646 findwork.go:35: Finding work: mode on, asof 2023-05-31 00:00:00 +0000 UTC
    run_test.go:66: 17:16:43.369582 findwork.go:35: Finding work: mode on, asof 2023-05-31 00:00:00 +0000 UTC
    run_test.go:66: 17:16:43.369752 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-05-08.v1.count
    run_test.go:66: 17:16:43.369861 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-05-08.v1.count
    run_test.go:66: 17:16:43.369616 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-05-08.v1.count
    run_test.go:66: 17:16:43.370231 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-05-09.v1.count
    run_test.go:66: 17:16:43.370245 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-05-09.v1.count
...
    run_test.go:66: 17:16:43.388046 reports.go:113: remove /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2569094852/001/local/upload.test-devel-darwin-amd64-2024-05-21.v1.count: no such file or directory failed to remove /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2569094852/001/local/upload.test-devel-darwin-amd64-2024-05-21.v1.count
    run_test.go:66: 17:16:43.388069 reports.go:64: Failed to create report for 2024-05-23: local report /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2569094852/001/local/local.2024-05-23.json already exists
    run_test.go:66: 17:16:43.388083 run.go:157: Uploading 0 reports
    run_test.go:66: 17:16:43.387658 reports.go:113: remove /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2569094852/001/local/upload.test-devel-darwin-amd64-2024-05-21.v1.count: no such file or directory failed to remove /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2569094852/001/local/upload.test-devel-darwin-amd64-2024-05-21.v1.count
    run_test.go:66: 17:16:43.388107 reports.go:64: Failed to create report for 2024-05-23: local report /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2569094852/001/local/local.2024-05-23.json already exists
    run_test.go:66: 17:16:43.388133 run.go:157: Uploading 1 reports
    run_test.go:66: 17:16:43.389401 upload.go:111: Uploaded 2024-05-23.json to "http://127.0.0.1:51325/2024-05-23"
    run_test.go:66: 17:16:43.389493 upload.go:111: Uploaded 2024-05-16.json to "http://127.0.0.1:51325/2024-05-16"
    run_test.go:593: uploaded 13 total observations, want 14
--- FAIL: TestRun_Concurrent (0.46s)

2024-05-22 23:36 x_telemetry-go1.22-darwin-amd64_11 telemetry@39ace7a4 release-branch.go1.22@74565750 x/telemetry/internal/upload.TestRun_Concurrent (log)

=== RUN   TestRun_Concurrent
    run_test.go:66: 18:17:30.218262 findwork.go:35: Finding work: mode on, asof 2023-05-31 00:00:00 +0000 UTC
    run_test.go:66: 18:17:30.218366 findwork.go:35: Finding work: mode on, asof 2023-05-31 00:00:00 +0000 UTC
    run_test.go:66: 18:17:30.218650 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-05-08.v1.count
    run_test.go:66: 18:17:30.218810 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-05-08.v1.count
    run_test.go:66: 18:17:30.218918 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-05-09.v1.count
    run_test.go:66: 18:17:30.218911 findwork.go:35: Finding work: mode on, asof 2023-05-31 00:00:00 +0000 UTC
    run_test.go:66: 18:17:30.219034 findwork.go:35: Finding work: mode on, asof 2023-05-31 00:00:00 +0000 UTC
    run_test.go:66: 18:17:30.219058 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-05-08.v1.count
    run_test.go:66: 18:17:30.219069 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-05-09.v1.count
...
    run_test.go:66: 18:17:30.226116 reports.go:64: Failed to create report for 2024-05-23: local report /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2249583122/001/local/local.2024-05-23.json already exists
    run_test.go:66: 18:17:30.226158 reports.go:113: remove /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2249583122/001/local/upload.test-devel-darwin-amd64-2024-05-20.v1.count: no such file or directory failed to remove /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2249583122/001/local/upload.test-devel-darwin-amd64-2024-05-20.v1.count
    run_test.go:66: 18:17:30.226175 reports.go:113: remove /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2249583122/001/local/upload.test-devel-darwin-amd64-2024-05-21.v1.count: no such file or directory failed to remove /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2249583122/001/local/upload.test-devel-darwin-amd64-2024-05-21.v1.count
    run_test.go:66: 18:17:30.226184 reports.go:64: Failed to create report for 2024-05-23: local report /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2249583122/001/local/local.2024-05-23.json already exists
    run_test.go:66: 18:17:30.226191 run.go:157: Uploading 0 reports
    run_test.go:66: 18:17:30.226164 run.go:157: Uploading 0 reports
    run_test.go:66: 18:17:30.226540 upload.go:111: Uploaded 2024-05-23.json to "http://127.0.0.1:58936/2024-05-23"
    run_test.go:66: 18:17:30.226624 upload.go:111: Uploaded 2024-05-16.json to "http://127.0.0.1:58936/2024-05-16"
    run_test.go:593: uploaded 13 total observations, want 14
--- FAIL: TestRun_Concurrent (0.20s)

2024-05-22 23:36 x_telemetry-go1.22-darwin-amd64_14 telemetry@39ace7a4 release-branch.go1.22@74565750 x/telemetry/internal/upload.TestRun_Concurrent (log)

=== RUN   TestRun_Concurrent
    run_test.go:66: 18:58:02.893007 findwork.go:35: Finding work: mode on, asof 2023-05-31 00:00:00 +0000 UTC
    run_test.go:66: 18:58:02.895442 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-05-08.v1.count
    run_test.go:66: 18:58:02.896315 findwork.go:35: Finding work: mode on, asof 2023-05-31 00:00:00 +0000 UTC
    run_test.go:66: 18:58:02.897332 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-05-09.v1.count
    run_test.go:66: 18:58:02.897724 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-05-08.v1.count
    run_test.go:66: 18:58:02.904669 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-05-10.v1.count
    run_test.go:66: 18:58:02.908755 findwork.go:35: Finding work: mode on, asof 2023-05-31 00:00:00 +0000 UTC
    run_test.go:66: 18:58:02.908466 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-05-09.v1.count
    run_test.go:66: 18:58:02.909854 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-05-08.v1.count
...
    run_test.go:66: 18:58:02.982806 findwork.go:97: Already uploaded: 2024-05-23.json
    run_test.go:66: 18:58:02.983172 reports.go:34: Last week: 2024-05-23, today: 2024-05-30
    run_test.go:66: 18:58:02.983244 run.go:157: Uploading 0 reports
    run_test.go:66: 18:58:02.994336 findwork.go:35: Finding work: mode on, asof 2023-05-31 00:00:00 +0000 UTC
    run_test.go:66: 18:58:02.994948 findwork.go:97: Already uploaded: 2024-05-16.json
    run_test.go:66: 18:58:02.995046 findwork.go:97: Already uploaded: 2024-05-23.json
    run_test.go:66: 18:58:02.995576 reports.go:34: Last week: 2024-05-23, today: 2024-05-30
    run_test.go:66: 18:58:02.995652 run.go:157: Uploading 0 reports
    run_test.go:593: uploaded 13 total observations, want 14
--- FAIL: TestRun_Concurrent (0.98s)

2024-05-22 23:36 x_telemetry-go1.22-windows-amd64 telemetry@39ace7a4 release-branch.go1.22@74565750 x/telemetry/internal/upload.TestRun_Concurrent (log)

=== RUN   TestRun_Concurrent
    run_test.go:66: 18:25:34.050596 findwork.go:35: Finding work: mode on, asof 2023-05-31 00:00:00 +0000 UTC
    run_test.go:66: 18:25:34.050596 findwork.go:35: Finding work: mode on, asof 2023-05-31 00:00:00 +0000 UTC
    run_test.go:66: 18:25:34.050596 findwork.go:50: Collecting count file upload.test-devel-windows-amd64-2024-05-08.v1.count
    run_test.go:66: 18:25:34.050596 findwork.go:50: Collecting count file upload.test-devel-windows-amd64-2024-05-08.v1.count
    run_test.go:66: 18:25:34.050596 findwork.go:35: Finding work: mode on, asof 2023-05-31 00:00:00 +0000 UTC
    run_test.go:66: 18:25:34.051487 findwork.go:50: Collecting count file upload.test-devel-windows-amd64-2024-05-09.v1.count
    run_test.go:66: 18:25:34.051487 findwork.go:35: Finding work: mode on, asof 2023-05-31 00:00:00 +0000 UTC
    run_test.go:66: 18:25:34.051487 findwork.go:50: Collecting count file upload.test-devel-windows-amd64-2024-05-08.v1.count
    run_test.go:66: 18:25:34.051487 findwork.go:35: Finding work: mode on, asof 2023-05-31 00:00:00 +0000 UTC
...
    run_test.go:66: 18:25:34.061359 reports.go:113: remove C:\b\s\w\ir\x\t\TestRun_Concurrent2928643688\001\local\upload.test-devel-windows-amd64-2024-05-18.v1.count: The system cannot find the file specified. failed to remove C:\b\s\w\ir\x\t\TestRun_Concurrent2928643688\001\local\upload.test-devel-windows-amd64-2024-05-18.v1.count
    run_test.go:66: 18:25:34.061359 reports.go:113: remove C:\b\s\w\ir\x\t\TestRun_Concurrent2928643688\001\local\upload.test-devel-windows-amd64-2024-05-19.v1.count: The system cannot find the file specified. failed to remove C:\b\s\w\ir\x\t\TestRun_Concurrent2928643688\001\local\upload.test-devel-windows-amd64-2024-05-19.v1.count
    run_test.go:66: 18:25:34.061359 reports.go:113: remove C:\b\s\w\ir\x\t\TestRun_Concurrent2928643688\001\local\upload.test-devel-windows-amd64-2024-05-20.v1.count: The system cannot find the file specified. failed to remove C:\b\s\w\ir\x\t\TestRun_Concurrent2928643688\001\local\upload.test-devel-windows-amd64-2024-05-20.v1.count
    run_test.go:66: 18:25:34.062229 reports.go:113: remove C:\b\s\w\ir\x\t\TestRun_Concurrent2928643688\001\local\upload.test-devel-windows-amd64-2024-05-21.v1.count: The system cannot find the file specified. failed to remove C:\b\s\w\ir\x\t\TestRun_Concurrent2928643688\001\local\upload.test-devel-windows-amd64-2024-05-21.v1.count
    run_test.go:66: 18:25:34.062229 reports.go:64: Failed to create report for 2024-05-23: local report C:\b\s\w\ir\x\t\TestRun_Concurrent2928643688\001\local\local.2024-05-23.json already exists
    run_test.go:66: 18:25:34.062229 run.go:157: Uploading 1 reports
    run_test.go:66: 18:25:34.063844 upload.go:111: Uploaded 2024-05-16.json to "http://127.0.0.1:49870/2024-05-16"
    run_test.go:66: 18:25:34.063844 upload.go:111: Uploaded 2024-05-23.json to "http://127.0.0.1:49870/2024-05-23"
    run_test.go:593: uploaded 13 total observations, want 14
--- FAIL: TestRun_Concurrent (0.30s)

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "golang.org/x/telemetry/internal/upload" && test == "TestRun_Concurrent"

2024-06-03 22:45 x_telemetry-gotip-linux-arm telemetry@f2b69109 go@535f81e7 x/telemetry/internal/upload.TestRun_Concurrent (log)

=== RUN   TestRun_Concurrent
    run_test.go:66: 15:37:57.528458 findwork.go:35: Finding work: mode on, asof 2023-06-05 00:00:00 +0000 UTC
    run_test.go:66: 15:37:57.528573 findwork.go:50: Collecting count file upload.test-devel-linux-arm-2024-05-14.v1.count
    run_test.go:66: 15:37:57.528621 findwork.go:50: Collecting count file upload.test-devel-linux-arm-2024-05-15.v1.count
    run_test.go:66: 15:37:57.528463 findwork.go:35: Finding work: mode on, asof 2023-06-05 00:00:00 +0000 UTC
    run_test.go:66: 15:37:57.528753 findwork.go:50: Collecting count file upload.test-devel-linux-arm-2024-05-14.v1.count
    run_test.go:66: 15:37:57.528814 findwork.go:50: Collecting count file upload.test-devel-linux-arm-2024-05-15.v1.count
    run_test.go:66: 15:37:57.528863 findwork.go:50: Collecting count file upload.test-devel-linux-arm-2024-05-16.v1.count
    run_test.go:66: 15:37:57.528913 findwork.go:50: Collecting count file upload.test-devel-linux-arm-2024-05-17.v1.count
    run_test.go:66: 15:37:57.528961 findwork.go:50: Collecting count file upload.test-devel-linux-arm-2024-05-18.v1.count
...
    run_test.go:66: 15:37:57.531860 reports.go:113: remove /home/swarming/.swarming/w/ir/x/t/TestRun_Concurrent3059370928/001/local/upload.test-devel-linux-arm-2024-05-24.v1.count: no such file or directory failed to remove /home/swarming/.swarming/w/ir/x/t/TestRun_Concurrent3059370928/001/local/upload.test-devel-linux-arm-2024-05-24.v1.count
    run_test.go:66: 15:37:57.531877 reports.go:113: remove /home/swarming/.swarming/w/ir/x/t/TestRun_Concurrent3059370928/001/local/upload.test-devel-linux-arm-2024-05-25.v1.count: no such file or directory failed to remove /home/swarming/.swarming/w/ir/x/t/TestRun_Concurrent3059370928/001/local/upload.test-devel-linux-arm-2024-05-25.v1.count
    run_test.go:66: 15:37:57.531896 reports.go:113: remove /home/swarming/.swarming/w/ir/x/t/TestRun_Concurrent3059370928/001/local/upload.test-devel-linux-arm-2024-05-26.v1.count: no such file or directory failed to remove /home/swarming/.swarming/w/ir/x/t/TestRun_Concurrent3059370928/001/local/upload.test-devel-linux-arm-2024-05-26.v1.count
    run_test.go:66: 15:37:57.531913 reports.go:113: remove /home/swarming/.swarming/w/ir/x/t/TestRun_Concurrent3059370928/001/local/upload.test-devel-linux-arm-2024-05-27.v1.count: no such file or directory failed to remove /home/swarming/.swarming/w/ir/x/t/TestRun_Concurrent3059370928/001/local/upload.test-devel-linux-arm-2024-05-27.v1.count
    run_test.go:66: 15:37:57.531931 reports.go:64: Failed to create report for 2024-05-29: local report /home/swarming/.swarming/w/ir/x/t/TestRun_Concurrent3059370928/001/local/local.2024-05-29.json already exists
    run_test.go:66: 15:37:57.531944 run.go:157: Uploading 0 reports
    run_test.go:66: 15:37:57.531905 upload.go:111: Uploaded 2024-05-22.json to "http://127.0.0.1:39247/2024-05-22"
    run_test.go:66: 15:37:57.531994 upload.go:54: open /home/swarming/.swarming/w/ir/x/t/TestRun_Concurrent3059370928/001/local/2024-05-29.json: no such file or directory reading /home/swarming/.swarming/w/ir/x/t/TestRun_Concurrent3059370928/001/local/2024-05-29.json
    run_test.go:593: uploaded 21 total observations, want 14
--- FAIL: TestRun_Concurrent (0.07s)

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "golang.org/x/telemetry/internal/upload" && test == "TestRun_Concurrent"

2024-06-21 19:41 x_telemetry-gotip-darwin-amd64_11 telemetry@a740542b go@e8ee1dc4 x/telemetry/internal/upload.TestRun_Concurrent (log)

=== RUN   TestRun_Concurrent
    run_test.go:66: 17:56:25.996832 findwork.go:35: Finding work: mode on, asof 2023-06-24 00:00:00 +0000 UTC
    run_test.go:66: 17:56:25.997280 findwork.go:35: Finding work: mode on, asof 2023-06-24 00:00:00 +0000 UTC
    run_test.go:66: 17:56:25.997360 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-06-01.v1.count
    run_test.go:66: 17:56:25.997388 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-06-01.v1.count
    run_test.go:66: 17:56:25.997675 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-06-02.v1.count
    run_test.go:66: 17:56:25.997719 findwork.go:35: Finding work: mode on, asof 2023-06-24 00:00:00 +0000 UTC
    run_test.go:66: 17:56:25.997793 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-06-01.v1.count
    run_test.go:66: 17:56:25.997819 findwork.go:35: Finding work: mode on, asof 2023-06-24 00:00:00 +0000 UTC
    run_test.go:66: 17:56:25.997837 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-06-02.v1.count
...
    run_test.go:66: 17:56:26.004049 findwork.go:35: Finding work: mode on, asof 2023-06-24 00:00:00 +0000 UTC
    run_test.go:66: 17:56:26.004074 findwork.go:71: Uploadable: 2024-06-09.json
    run_test.go:66: 17:56:26.004125 findwork.go:71: Uploadable: 2024-06-16.json
    run_test.go:66: 17:56:26.004188 reports.go:34: Last week: , today: 2024-06-23
    run_test.go:66: 17:56:26.004197 run.go:157: Uploading 2 reports
    run_test.go:66: 17:56:26.004518 upload.go:80: Failed to acquire lock /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2517570491/001/upload/2024-06-09.json.lock: open /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2517570491/001/upload/2024-06-09.json.lock: file exists
    run_test.go:66: 17:56:26.004645 upload.go:111: Uploaded 2024-06-09.json to "http://127.0.0.1:50119/2024-06-09"
    run_test.go:66: 17:56:26.004818 upload.go:111: Uploaded 2024-06-16.json to "http://127.0.0.1:50119/2024-06-16"
    run_test.go:593: uploaded 13 total observations, want 14
--- FAIL: TestRun_Concurrent (0.18s)

2024-06-21 19:41 x_telemetry-gotip-darwin-arm64_11 telemetry@a740542b go@e8ee1dc4 x/telemetry/internal/upload.TestRun_Concurrent (log)

=== RUN   TestRun_Concurrent
    run_test.go:66: 17:56:43.716603 findwork.go:35: Finding work: mode on, asof 2023-06-24 00:00:00 +0000 UTC
    run_test.go:66: 17:56:43.716686 findwork.go:50: Collecting count file upload.test-devel-darwin-arm64-2024-06-01.v1.count
    run_test.go:66: 17:56:43.716715 findwork.go:35: Finding work: mode on, asof 2023-06-24 00:00:00 +0000 UTC
    run_test.go:66: 17:56:43.716726 findwork.go:50: Collecting count file upload.test-devel-darwin-arm64-2024-06-02.v1.count
    run_test.go:66: 17:56:43.716758 findwork.go:50: Collecting count file upload.test-devel-darwin-arm64-2024-06-01.v1.count
    run_test.go:66: 17:56:43.716761 findwork.go:50: Collecting count file upload.test-devel-darwin-arm64-2024-06-03.v1.count
    run_test.go:66: 17:56:43.716788 findwork.go:50: Collecting count file upload.test-devel-darwin-arm64-2024-06-02.v1.count
    run_test.go:66: 17:56:43.716821 findwork.go:50: Collecting count file upload.test-devel-darwin-arm64-2024-06-04.v1.count
    run_test.go:66: 17:56:43.716826 findwork.go:35: Finding work: mode on, asof 2023-06-24 00:00:00 +0000 UTC
...
    run_test.go:66: 17:56:43.719520 reports.go:113: remove /opt/s/w/ir/x/t/TestRun_Concurrent4277224640/001/local/upload.test-devel-darwin-arm64-2024-06-11.v1.count: no such file or directory failed to remove /opt/s/w/ir/x/t/TestRun_Concurrent4277224640/001/local/upload.test-devel-darwin-arm64-2024-06-11.v1.count
    run_test.go:66: 17:56:43.719527 reports.go:113: remove /opt/s/w/ir/x/t/TestRun_Concurrent4277224640/001/local/upload.test-devel-darwin-arm64-2024-06-12.v1.count: no such file or directory failed to remove /opt/s/w/ir/x/t/TestRun_Concurrent4277224640/001/local/upload.test-devel-darwin-arm64-2024-06-12.v1.count
    run_test.go:66: 17:56:43.719537 reports.go:113: remove /opt/s/w/ir/x/t/TestRun_Concurrent4277224640/001/local/upload.test-devel-darwin-arm64-2024-06-13.v1.count: no such file or directory failed to remove /opt/s/w/ir/x/t/TestRun_Concurrent4277224640/001/local/upload.test-devel-darwin-arm64-2024-06-13.v1.count
    run_test.go:66: 17:56:43.719545 reports.go:113: remove /opt/s/w/ir/x/t/TestRun_Concurrent4277224640/001/local/upload.test-devel-darwin-arm64-2024-06-14.v1.count: no such file or directory failed to remove /opt/s/w/ir/x/t/TestRun_Concurrent4277224640/001/local/upload.test-devel-darwin-arm64-2024-06-14.v1.count
    run_test.go:66: 17:56:43.719549 reports.go:64: Failed to create report for 2024-06-16: local report /opt/s/w/ir/x/t/TestRun_Concurrent4277224640/001/local/local.2024-06-16.json already exists
    run_test.go:66: 17:56:43.719553 run.go:157: Uploading 0 reports
    run_test.go:66: 17:56:43.719748 upload.go:111: Uploaded 2024-06-09.json to "http://127.0.0.1:50679/2024-06-09"
    run_test.go:66: 17:56:43.719831 upload.go:111: Uploaded 2024-06-16.json to "http://127.0.0.1:50679/2024-06-16"
    run_test.go:593: uploaded 13 total observations, want 14
--- FAIL: TestRun_Concurrent (0.07s)

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "golang.org/x/telemetry/internal/upload" && test == "TestRun_Concurrent"

2024-06-24 21:43 x_telemetry-go1.21-darwin-amd64-nocgo telemetry@210d439e release-branch.go1.21@dfb00670 x/telemetry/internal/upload.TestRun_Concurrent (log)

=== RUN   TestRun_Concurrent
    run_test.go:66: 19:13:28.334433 findwork.go:35: Finding work: mode on, asof 2023-06-26 00:00:00 +0000 UTC
    run_test.go:66: 19:13:28.334727 findwork.go:35: Finding work: mode on, asof 2023-06-26 00:00:00 +0000 UTC
    run_test.go:66: 19:13:28.334868 findwork.go:35: Finding work: mode on, asof 2023-06-26 00:00:00 +0000 UTC
    run_test.go:66: 19:13:28.335085 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-06-03.v1.count
    run_test.go:66: 19:13:28.335225 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-06-03.v1.count
    run_test.go:66: 19:13:28.335280 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-06-03.v1.count
    run_test.go:66: 19:13:28.335449 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-06-04.v1.count
    run_test.go:66: 19:13:28.335464 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-06-04.v1.count
    run_test.go:66: 19:13:28.335556 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-06-04.v1.count
...
    run_test.go:66: 19:13:28.346808 upload.go:80: Failed to acquire lock /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2987023683/001/upload/2024-06-11.json.lock: open /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2987023683/001/upload/2024-06-11.json.lock: file exists
    run_test.go:66: 19:13:28.347559 upload.go:111: Uploaded 2024-06-11.json to "http://127.0.0.1:49287/2024-06-11"
    run_test.go:66: 19:13:28.347867 upload.go:111: Uploaded 2024-06-18.json to "http://127.0.0.1:49287/2024-06-18"
    run_test.go:66: 19:13:28.359325 findwork.go:35: Finding work: mode on, asof 2023-06-26 00:00:00 +0000 UTC
    run_test.go:66: 19:13:28.359426 findwork.go:97: Already uploaded: 2024-06-11.json
    run_test.go:66: 19:13:28.359443 findwork.go:97: Already uploaded: 2024-06-18.json
    run_test.go:66: 19:13:28.359506 reports.go:34: Last week: 2024-06-18, today: 2024-06-25
    run_test.go:66: 19:13:28.359519 run.go:157: Uploading 0 reports
    run_test.go:593: uploaded 13 total observations, want 14
--- FAIL: TestRun_Concurrent (0.40s)

2024-06-24 21:43 x_telemetry-go1.21-darwin-amd64_14 telemetry@210d439e release-branch.go1.21@dfb00670 x/telemetry/internal/upload.TestRun_Concurrent (log)

=== RUN   TestRun_Concurrent
    run_test.go:66: 19:11:04.892926 findwork.go:35: Finding work: mode on, asof 2023-06-26 00:00:00 +0000 UTC
    run_test.go:66: 19:11:04.893910 findwork.go:35: Finding work: mode on, asof 2023-06-26 00:00:00 +0000 UTC
    run_test.go:66: 19:11:04.894110 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-06-03.v1.count
    run_test.go:66: 19:11:04.894714 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-06-03.v1.count
    run_test.go:66: 19:11:04.894845 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-06-04.v1.count
    run_test.go:66: 19:11:04.895121 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-06-04.v1.count
    run_test.go:66: 19:11:04.895161 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-06-05.v1.count
    run_test.go:66: 19:11:04.895300 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-06-05.v1.count
    run_test.go:66: 19:11:04.895521 findwork.go:50: Collecting count file upload.test-devel-darwin-amd64-2024-06-06.v1.count
...
    run_test.go:66: 19:11:04.908140 reports.go:64: Failed to create report for 2024-06-11: local report /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2257027360/001/local/local.2024-06-11.json already exists
    run_test.go:66: 19:11:04.908298 upload.go:111: Uploaded 2024-06-18.json to "http://127.0.0.1:52771/2024-06-18"
    run_test.go:66: 19:11:04.908367 reports.go:113: remove /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2257027360/001/local/upload.test-devel-darwin-amd64-2024-06-11.v1.count: no such file or directory failed to remove /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2257027360/001/local/upload.test-devel-darwin-amd64-2024-06-11.v1.count
    run_test.go:66: 19:11:04.908401 reports.go:113: remove /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2257027360/001/local/upload.test-devel-darwin-amd64-2024-06-12.v1.count: no such file or directory failed to remove /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2257027360/001/local/upload.test-devel-darwin-amd64-2024-06-12.v1.count
    run_test.go:66: 19:11:04.908509 reports.go:113: remove /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2257027360/001/local/upload.test-devel-darwin-amd64-2024-06-13.v1.count: no such file or directory failed to remove /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2257027360/001/local/upload.test-devel-darwin-amd64-2024-06-13.v1.count
    run_test.go:66: 19:11:04.908541 reports.go:113: remove /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2257027360/001/local/upload.test-devel-darwin-amd64-2024-06-14.v1.count: no such file or directory failed to remove /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2257027360/001/local/upload.test-devel-darwin-amd64-2024-06-14.v1.count
    run_test.go:66: 19:11:04.908554 reports.go:64: Failed to create report for 2024-06-18: local report /Users/swarming/.swarming/w/ir/x/t/TestRun_Concurrent2257027360/001/local/local.2024-06-18.json already exists
    run_test.go:66: 19:11:04.908567 run.go:157: Uploading 0 reports
    run_test.go:593: uploaded 13 total observations, want 14
--- FAIL: TestRun_Concurrent (0.40s)

— watchflakes

[findleyr]

Looks like a real race. I will take a look.

[findleyr]

Not a race: looks like a timezone / day boundary bug 🤦.

[gopherbot]

Change https://go.dev/cl/596316 mentions this issue: internal/upload: use UTC time in TestRun_Concurrent

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "golang.org/x/telemetry/internal/upload" && test == "TestRun_Concurrent"

2024-07-01 17:54 x_telemetry-go1.21-windows-arm64 telemetry@4e29c787 release-branch.go1.21@12e9b968 x/telemetry/internal/upload.TestRun_Concurrent (log)

=== RUN   TestRun_Concurrent
    run_test.go:66: 21:03:41.860501 findwork.go:35: Finding work: mode on, asof 2023-07-03 00:00:00 +0000 UTC
    run_test.go:66: 21:03:41.861858 findwork.go:50: Collecting count file upload.test-devel-windows-arm64-2024-06-11.v1.count
    run_test.go:66: 21:03:41.861858 findwork.go:50: Collecting count file upload.test-devel-windows-arm64-2024-06-12.v1.count
    run_test.go:66: 21:03:41.861858 findwork.go:50: Collecting count file upload.test-devel-windows-arm64-2024-06-13.v1.count
    run_test.go:66: 21:03:41.861858 findwork.go:50: Collecting count file upload.test-devel-windows-arm64-2024-06-14.v1.count
    run_test.go:66: 21:03:41.861858 findwork.go:50: Collecting count file upload.test-devel-windows-arm64-2024-06-15.v1.count
    run_test.go:66: 21:03:41.861858 findwork.go:50: Collecting count file upload.test-devel-windows-arm64-2024-06-16.v1.count
    run_test.go:66: 21:03:41.861858 findwork.go:50: Collecting count file upload.test-devel-windows-arm64-2024-06-17.v1.count
    run_test.go:66: 21:03:41.862877 findwork.go:50: Collecting count file upload.test-devel-windows-arm64-2024-06-18.v1.count
...
    run_test.go:66: 21:03:41.870249 upload.go:80: Failed to acquire lock C:\Users\swarming\.swarming\w\ir\x\t\TestRun_Concurrent2162911338\001\upload\2024-06-19.json.lock: open C:\Users\swarming\.swarming\w\ir\x\t\TestRun_Concurrent2162911338\001\upload\2024-06-19.json.lock: The file exists.
    run_test.go:66: 21:03:41.870781 upload.go:80: Failed to acquire lock C:\Users\swarming\.swarming\w\ir\x\t\TestRun_Concurrent2162911338\001\upload\2024-06-12.json.lock: open C:\Users\swarming\.swarming\w\ir\x\t\TestRun_Concurrent2162911338\001\upload\2024-06-12.json.lock: The file exists.
    run_test.go:66: 21:03:41.869954 upload.go:80: Failed to acquire lock C:\Users\swarming\.swarming\w\ir\x\t\TestRun_Concurrent2162911338\001\upload\2024-06-12.json.lock: open C:\Users\swarming\.swarming\w\ir\x\t\TestRun_Concurrent2162911338\001\upload\2024-06-12.json.lock: The file exists.
    run_test.go:66: 21:03:41.871311 upload.go:111: Uploaded 2024-06-12.json to "http://127.0.0.1:59857/2024-06-12"
    run_test.go:66: 21:03:41.871859 upload.go:111: Uploaded 2024-06-19.json to "http://127.0.0.1:59857/2024-06-19"
    run_test.go:66: 21:03:41.871859 upload.go:80: Failed to acquire lock C:\Users\swarming\.swarming\w\ir\x\t\TestRun_Concurrent2162911338\001\upload\2024-06-26.json.lock: open C:\Users\swarming\.swarming\w\ir\x\t\TestRun_Concurrent2162911338\001\upload\2024-06-26.json.lock: The file exists.
    run_test.go:66: 21:03:41.872907 upload.go:111: Uploaded 2024-06-26.json to "http://127.0.0.1:59857/2024-06-26"
    run_test.go:66: 21:03:41.872907 upload.go:111: Uploaded 2024-06-19.json to "http://127.0.0.1:59857/2024-06-19"
    run_test.go:593: uploaded 21 total observations, want 14
--- FAIL: TestRun_Concurrent (0.59s)

— watchflakes