testing: testing with -json option is not debuggable

[zolotov]

Continuation of #22789 and relates to #22996

Suggested solution doesn't allow debug tests and use JSON output simultaneously.

[bradfitz]

I don't understand.

What do you mean by debuggable?

[zolotov]

Literally, I cannot debug a test with Delve.

[ianlancetaylor]

Sorry, can you spell out in more detail what you want to do?

Do you want to run Delve on a test, while looking at the JSON output from the test? Why is that an important use case?

[zolotov]

Yes, I want to debug my tests and have JSON output at the same time, not sure what I can add to this.

It's important because otherwise IDE or any other tool that provides debug support for tests won't be able to use brand new JSON output. Particularly I won't be able to fix a lot of issues that users asked us to fix in testing.

[ianlancetaylor]

Your IDE is going to be parsing the output of the program you are debugging? I'll just take your word for it. I can't picture that at all.

In that case, why not have your IDE pipe the test output through go tool test2json?

[zolotov]

Your IDE is going to be parsing the output of the program you are debugging?

Yes, it's not different in any way from the current situation when we have to parse messages like ===RUN.

In that case, why not have your IDE pipe the test output through go tool test2json?

Because of #22789, because it's much more complicated and because the issue not only about the IDE: it might be tools for remote debugging on continuous integrations servers or just for a user who wants to see json reports while debugging from the CLI.

[zolotov]

Is this still waiting for info?

[ianlancetaylor]

Well, yes, this is still waiting for info. Can you say precisely what change you want?

If the change you want is for tests to be able to generate JSON output without going through the test2json program, that is not going to happen for the reasons given over at #22996.

[zolotov]

Well, yes, this is still waiting for info. Can you say precisely what change you want?

I'm not asking for particular changes, I'm asking for a cross-platform way to debug tests with json output. Could you suggest one?

that is not going to happen for the reasons given over at #22996.

Having a test-binary that reports JSON would fix the issue automatically, but having such binary is not the goal, this is why it's different from #22996

[ianlancetaylor]

I'm asking for a cross-platform way to debug tests with json output. Could you suggest one?

I'm sorry for asking the same question over and over again, but I honestly and genuinely do not understand what you mean. I can debug a test by running delve. Whether the test output JSON or not is entirely irrelevant to my debugging experience.

Let me try this a different way. Can you explain, in as much detail as possible, precisely what you do, what you expect to happen, and precisely what happens instead? Thanks.

[zolotov]

I can debug a test by running delve

This is what I want to know. How do you run it?

This is what I can imagine about running Delve and having JSON output:

• dlv <binary> doesn't work because, as we know from cmd/test2json: support emitting JSON directly from the test binary #22996, a testing binary cannot produce JSON output.

• dlv test -json <etc> doesn't work because dlv test is a shortcut for making testing binary and then running dlv <binary>.

• dlv ./foo.test -test.v 2>&1 | go tool test2json is not a cross-platform solution (cmd/test2json: handle running command and processing output #22789). Also, it converts Delve's debug output, which is not desirable.

• dlv go tool test2json ./foo.test -test.v produces JSON output but it doesn't stop at breakpoints, likely it tries to debug the wrong process

[zolotov]

go tool test2json dlv <binary> should work but it converts Delve's output