parse test output and suggest re-running failed subtests

[ramya-rao-a]

microsoft/vscode-go#2850 contains the discussion that lead to the conclusion that there is a need to re-run failed tests from the last test run.

This feature request is to parse the test output, identify failed tests and provide a command to re-run just the failed tests.

Code Pointers from the old repo:

• Set up local dev environment for the extension
• goTest() is the function that calls go test and parses the output. It is here that you will need to keep track of the failed test names and return them in the rejected promise.
• Use the code for the Go: Test Previous command at https://github.com/microsoft/vscode-go/blob/0.12.0/src/goTest.ts#L226-L237 as a reference to have a new command Go: Test Previously failed tests.
• You will need a copy of the latestTestConfig which has its functions updated to be reflect the name of the failed tests.

[smyrman]

In more detail, what is not possible today, is to (re-)run an individually (failed) sub-test.

Inserting links to run sub-tests in the code is hard because sub-tests can be defined in many ways. E.g by calling t.Run(...) within a for loop. In other words the full test-name of sub-tests is only known run-time.

The case to solve (in my mind), is when you have many failing sub-tests (within a single test function), and you want to focus on one of those failures and try to fix it. Re-running the full test function can create a lot of noice if there are many sub-tests. Running a sub-test trough go test with a -name filter is possible, but it would be a lot easier if there where some sort of short-cut for doing it from within vs code.

Re-running a (failed) sub-test from the test output via a link, might just work...

[stamblerre]

I believe that https://go-review.googlesource.com/c/vscode-go/+/235447 should at least address the issue of running subtests. The next step would be adding a link if a single test fails.

[hyangah]

FYI I think go test -json can be useful here instead of trying to develop our own parsing if a couple of blockers such as golang/go#40588 can be addressed.

[smyrman]

Indeed, go test -json would be the way to do.

I think other tools, such as https://github.com/gotestyourself/gotestsum have already worked around golang/go#40588, so don't think it's a blocker.

[hyangah]

@smyrman Yes, the heuristics still need to be implemented in typescript/javascript. (and I also noticed noticeable slowdown by switching to -json , but definitely not a critical blocker)

[hyangah]

With @firelizzard18 's impressive work that added test explorer UI, I think this is already covered - the test explorer UI processes the output and marks failed tests clearly with an option to rerun individual tests/subtests. @firelizzard18 what do you think?

[firelizzard18]

The test explorer UI definitely shows the pass/fail status of subtests, and CL 343433 adds the ability to run a subtest by itself. That CL does not allow running more than subtest, but some scenarios could be supported: subtests TestFoo/A and TestFoo/B can be run together via -run '^TestFoo$/^(A|B)$. However, TestFoo/A and TestBar/B cannot be run together, and TestFoo/A and TestFoo/B/C cannot be run together, due to limitations in the regexp syntax for the -run flag: golang/go#39904.

The test explorer UI does not currently support debugging. That is something I plan to add, but I probably won't do that until the code lens/command functionality and the test explorer functionality have been more closely integrated with each other.

Is there some additional capability desired, beyond the ability to see the pass/fail status of individual subtests, and to run a specific subtest?

[smyrman]

Is there some additional capability desired, beyond the ability to see the pass/fail status of individual subtests, and to run a specific subtest?

From my stand-point, then no. This is a perfect limitation of scope.

If anything, it might be nice to be able to re-run at any level.

TestFoo, TestFoo/A or TestFoo/A/C.

But I don't see a strong enough use-case for running ^TestFoo/(A|B)$ to try to optimize the UI for it.

[firelizzard18]

With CL 343433, you will be able to run a subtest at any level.

You just can't run multiple subtests together. And go test -run syntax precludes simultaneously running two subtests at different levels.

[hyangah]

I confirm that v0.28.0 dynamically finds subtests and provides UI to rerun the subtests.
Running multiple subtests is tracked in #1772.
Thanks @firelizzard18 for the awesome work!

Debugging support is in progress AFAIK.

[firelizzard18]

Debugging support is in progress AFAIK.

There's a CL that implements basic debugging; beyond that, I intend to wait until after #1702.