CI test output `results.json` contains a lot of duplicate entries

[inkydragon]

1. Open https://buildkite.com/julialang/julia-master/builds/33133#018d7b6a-ce86-45be-800e-8f0498598629
2. Locate the "Artifacts" tab
3. Download julia-28db6c7a11/share/julia/test/results.json
4. Open results.json
5. LinearAlgebra/addmul repeated 4000+ times

JSON Output code:

julia/test/runtests.jl

Lines 431 to 435 in 28db6c7

	if Base.get_bool_env("CI", false)
	testresults = joinpath(@__DIR__, "results.json")
	@info "Writing test result data to $testresults"
	open(io -> write_testset_json(io, o_ts), testresults, "w")
	end

julia/test/buildkitetestjson.jl

Line 11 in 28db6c7

export write_testset_json

cc: @tecosaur

[IanButterworth]

It has entries for all non-Pass results, and some tests do @test_broken in loops.

This isn't a bug, but arguably it could de-duplicate test failures in the reporting itself.

[tecosaur]

This is a bit of a tangent, but for cases where there's a test run in a loop, I wonder if we could somehow do better than

for i in [...]
     @test f(i)
end

with something like

@bulktest for i in [...]
    @test f(i)
end # Combine into a single test result somehow

or similar

[tecosaur]

This isn't a bug, but arguably it could de-duplicate test failures in the reporting itself.

Yea, I think that would be good. I'll see about implementing that.

[vtjnash]

We do have @testset for already, so maybe that can somehow combine them? Or just automatically match up all tests with the same line number, and combine them into a single report? I don't know if that needs to be done before generating, or if the later UI tools are capable of merging those already.

[IanButterworth]

@tecosaur on slack I proposed that we just make the results.json contain testset objects.
If the testset has any failures, then group them into the testset somehow.

Recording Results objects seems too tricky because Test.jl doesn't retain Pass info for memory reasons (that seem hard to avoid) and recording testsets are good because they always exist, pass or fail or otherwise, which seems better for the way buildkite analytics is set up.

[tecosaur]

Ah right, I see the sense in that. From a "tracking test results over time" perspective I still think there's value to be had in test-level granularity, but am not sure what's the best balance of concerns.

One PR I'm most of the way through spinning up is just merging duplicate entries, which seems like a completely clear improvement.