Tracking issue for libtest JSON output

[Gilnaa]

Added in #46450
Available in nightly behind a -Z flag.

[johnterickson]

@nrc You mentioned "The long-term solution is going to be deeply intertwined with pluggable test runners" in #46450. Does that mean that JSON won't be stabilized until there are pluggable test runners?

I'm trying to figure out if it makes sense for me to continue looking at #51924 right now, or if I should hold off for the time being. For example, I'm looking at adding test durations to the JSON (master...johnterickson:testjson) and converting the output to JUnit (https://github.com/johnterickson/cargo2junit/).

[djrenren]

Yeah I think we should stabilize something like this independently of custom test frameworks. It makes sense that the included test framework would have a programmatically readable output. Libtest's conceptual model is pretty simple and most likely won't undergo any drastic changes so I'm all for stabilizing something. I'd prefer to have a JSON Schema that describes the structure before stabilization. We'd also need to audit to ensure it's resilient to any minor changes or additions over time.

@alexcrichton thoughts?

[alexcrichton]

Seems like a plausible addition to me! So long as it's designed careful I think it'd be good to go

[epage]

From the experience of creating naiive serde structs / enums for libtest's output, here are my thoughts from translating it exactly how it is written in libtest (rather than inventing my own schema that happens to be compatible):

• For suites / tests, the data is two enums nested. For some cases, this might work well, for others, it is unnecessary nesting to deal with.
• Probably the most annoying aspect is that the type field conflates suite/test being finished and why it finished. This is annoying for if you want to get information regardless of the completion status/.
• No bench start even is sent
• My gut is telling me that the bench event is too narrowly focused on the output of the current implementation and not on what might be wanted

My PR where you can see the data structure's I created: https://github.com/crate-ci/escargot/pull/24/files

Ideas to consider

• Combine event and type fields. Unsure of the value of this
• Split type field into type and status and define more fields as being available for all types. For example, it could be useful to programmatically report stdout for a successful test and leave it to the rendering engine to decide whether to ignore it or not. This doesn't mean all test implementations need to report all of the fields, but define it as possible.
• Get someone from other bench implementations, like criterion, to review the bench schema.

[andoriyu]

I think it's worth adding things like package and type of test. Type of test: unit test, doc test, integration test.

The package is something like: module for unit test, resource path for doc test (crate::foo::Bar), and filename for integration test.

[epage]

Since libtest tracks test times (at minimum, reporting if its taking too long), it would be nice if that got included in the programmatic output so we can report it to the user, e.g. in JUnit. Alternatively, we'd have to timestamp the events as we read them.

[johnterickson]

@epage / @andoriyu Are one of you planning on proposing a better schema? I like what you're proposing.

[rbtcollins]

If I may add a semi-random thing; https://github.com/testing-cabal/subunit is a testing protocol thats been around for a while (one of a number) - the V2 even model is very similar in nature to your proposed JSON output schema. There is a rust implementation of the binary serialisation https://github.com/mtreinish/subunit-rust and I'm interested / planning on doing a JSON variant of V2 at some point for use with web stacks.

All of which to say two things I guess - consider thinking about chunking stdout/err, allowing additional streams, tags, and possibly, if its interesting, consider adopting the same model - I think it would be a super set of libtests current needs.

Of course, doing a new schema and having folk write adapters is complete fine too; I just saw this ticket and thought I'd offer this kibbitz :)