debug: toggling breakpoints hangs while focus is on an ephemeral goroutine

[lucky-wolf]

For asking questions, see:

• GitHub Discussions (Q&A)
• #vscode channel in Gophers Slack

Before filing an issue, please review our troubleshooting guides

• Troubleshooting problems with debugging
• Troubleshooting other problems

Please answer these questions before submitting your issue. Thanks!

What version of Go, VS Code & VS Code Go extension are you using?

• Run go version to get version of Go from the VS Code integrated terminal.
  • go version go1.16.5 linux/amd64
• Run gopls -v version to get version of Gopls from the VS Code integrated terminal.
  • Build info

    golang.org/x/tools/gopls v0.7.0
    golang.org/x/tools/gopls@v0.7.0 h1:JQBHW81Gsyim6iDjUwGoPeSpXrSqwen3isPJLfDfaYU=
    github.com/BurntSushi/toml@v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
    github.com/google/go-cmp@v0.5.5 h1:Khx7svrCpmxxtHBq5j2mp/xVjsi8hQMfNLvJFAlrGgU=
    github.com/sergi/go-diff@v1.1.0 h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0=
    golang.org/x/mod@v0.4.2 h1:Gz96sIWK3OalVv/I/qNygP42zyoKp3xptRVCWRFEBvo=
    golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c h1:5KslGYwFpkhGh+Q16bwMP3cOontH8FOep7tGV86Y7SQ=
    golang.org/x/sys@v0.0.0-20210510120138-977fb7262007 h1:gG67DSER+11cZvqIMb8S8bt0vZtiN6xWYARwirrOSfE=
    golang.org/x/tools@v0.1.3-0.20210608163600-9ed039809d4c h1:Pv9gNyJFYVdpUAVZYJ1BDSU4eGgXQ+0f3DIGAdolO5s=
    golang.org/x/xerrors@v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
    honnef.co/go/tools@v0.2.0 h1:ws8AfbgTX3oIczLPNPCu5166oBg9ST2vNs0rcht+mDE=
    mvdan.cc/gofumpt@v0.1.1 h1:bi/1aS/5W00E2ny5q65w9SnKpWEF/UIOqDYBILpo9rA=
    mvdan.cc/xurls/v2@v2.2.0 h1:NSZPykBXJFCetGZykLAxaL6SIpvbVy/UFEniIfHAa8A=

• Run code -v or code-insiders -v to get version of VS Code or VS Code Insiders.
  • 1.58.2
    c3f126316369cd610563c75b1b1725e0679adfb3
    x64
• Check your installed extensions to get the version of the VS Code Go extension
  • N/A
• Run Ctrl+Shift+P (Cmd+Shift+P on Mac OS) > Go: Locate Configured Go Tools command.
  • Checking configured tools....

GOBIN: undefined
toolsGopath: 
gopath: /home/swolf/go
GOROOT: /home/swolf/go-sdk
PATH: /home/swolf/go-sdk/bin:/home/swolf/go/bin:/home/swolf/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/swolf/protobuf/bin

   gopkgs: /home/swolf/go/bin/gopkgs installed
   go-outline: /home/swolf/go/bin/go-outline installed
   gotests: /home/swolf/go/bin/gotests installed
   gomodifytags: /home/swolf/go/bin/gomodifytags installed
   impl: /home/swolf/go/bin/impl installed
   goplay: /home/swolf/go/bin/goplay installed
   dlv: /home/swolf/go/bin/dlv installed
   dlv-dap: /home/swolf/go/bin/dlv-dap installed
   staticcheck: /home/swolf/go/bin/staticcheck installed
   gopls: /home/swolf/go/bin/gopls installed

go env
Workspace Folder (pb): /home/swolf/Projects/pb
	GO111MODULE=""
	GOARCH="amd64"
	GOBIN=""
	GOCACHE="/home/swolf/.cache/go-build"
	GOENV="/home/swolf/.config/go/env"
	GOEXE=""
	GOFLAGS=""
	GOHOSTARCH="amd64"
	GOHOSTOS="linux"
	GOINSECURE=""
	GOMODCACHE="/home/swolf/go/pkg/mod"
	GONOPROXY="github.com/luckypointlabs,github.com/Arkadia-tech"
	GONOSUMDB="github.com/luckypointlabs,github.com/Arkadia-tech"
	GOOS="linux"
	GOPATH="/home/swolf/go"
	GOPRIVATE="github.com/luckypointlabs,github.com/Arkadia-tech"
	GOPROXY="https://proxy.golang.org,direct"
	GOROOT="/home/swolf/go-sdk"
	GOSUMDB="sum.golang.org"
	GOTMPDIR=""
	GOTOOLDIR="/home/swolf/go-sdk/pkg/tool/linux_amd64"
	GOVCS=""
	GOVERSION="go1.16.5"
	GCCGO="gccgo"
	AR="ar"
	CC="gcc"
	CXX="g++"
	CGO_ENABLED="1"
	GOMOD="/home/swolf/Projects/pb/go.mod"
	CGO_CFLAGS="-g -O2"
	CGO_CPPFLAGS=""
	CGO_CXXFLAGS="-g -O2"
	CGO_FFLAGS="-g -O2"
	CGO_LDFLAGS="-g -O2"
	PKG_CONFIG="pkg-config"
	GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2546632551=/tmp/go-build -gno-record-gcc-switches"

Share the Go related settings you have added/edited

Nothing very interesting...

Describe the bug

When I set a breakpoint in a running session using dlv-dap, the process hangs (assuming it wasn't paused when I set the BP).

When I clear a BP in a paused session using dlv-dap, I thrown into src/runtime/sigqueue.go:168

Eitherway - unable to change BPs in an active debug session. I can change them if I exit the process first, and start a new session.

Steps to reproduce the behavior:

launch a go server
to set a BP on something you know will trigger in a moment / after an action
do that action to trigger that code
nothing will ever happen (the process is frozen)

Screenshots or recordings

If applicable, add screenshots or recordings to help explain your problem.

[lucky-wolf]

I'd give you more logging info - but I've always found selecting text & copying from output to be nigh-impossible (linux mint / cinnamon vm, if that matters).

[lucky-wolf]

Note: I was informed that using dlv-dap the behavior of setting a BP causes the process to be paused. Simply resuming the process after setting the BP does appear to work.

[lucky-wolf]

Hmm... having that "jumps to runtime/sema.go" over and over bug (similar to #1617).

I simply set a BP - started a session - after it trapped into debugger - stepped a few lines, then stepped into something - and it began happening on each new step...

[hyangah]

@lucky-wolf Seems like you already figured out how to handle the original issue - resuming after changing breakpoints - and now you are experiencing a different issue - jumping to runtime/sema.go. Am I understanding it correctly?

Re: #1648 (comment)
What is the dlv-dap version (go version -m /home/swolf/go/bin/dlv-dap)? Do you reproduce the issue if you update dlv-dap ("Go: Install/Update Tools" -> choose dlv-dap)?

[lucky-wolf]

@hyangah apologies for the swerve - but yes, I was asked on #1648 to start another report for the toggling BP issues - but for the most part those are just not automatic anymore - if I manually resume the debug process after toggling one, it does appear to work as expected. So that could be said to be "working as intended"

However, while using the debugger yesterday - I did run into essentially that same bug (well, nearly same behavior, if the same cause or simply something like it) - where each step over / step into would flash over to a go runtime file - seemingly to do with goroutine state assessment?

in 1648 it was a different file - this time it jumps into sema.go.

there's a lot of context switching in the debug output from dlv-dap - but nothing more interesting than that. screenshots might be easier than copying the text from the debug output window - not sure why that is so user-unfriendly - but I cannot find a good way to simply select multiple lines & copy them out - it refuses to allow me to select them properly and becomes impossible to interface to when I do have a selection...

[polinasok]

@lucky-wolf

Setting breakpoints while the program is running

Just to confirm the expected behavior (which you already seem to have figured out):

1. When setting a breakpoint during one of the "step" operations,
   Legacy: halts execution as if you clicked pause first, sets breakpoint, user must resume manually
   Dlv-dap: halts execution as if you clicked pause first, sets breakpoint, user must resume manually
2. When setting a breakpoint during "continue"
   Legacy: halts execution, sets breakpoint, execution is resumed automatically (can be flaky due to concurrency issues)
   Dlv-dap: halts execution as if you clicked pause first, sets breakpoint, user must resume manually
   So yes, for now this is working as intended. We will continue looking into automating program execution after setting a breakpoint (tricky), but in the meantime we can make this behavior more clear with additional user messages.

Jumping goroutines/files while stepping

This is likely caused by the same underlying issues as #1617 and #1647. Since everybody's goroutine set up iss differemt, it is not surprising that flickering files are not always the same. We have submitted an additional fixes for the issue since it was reported, so please do make sure that you are using the most recent version of dlv-dap.

Coping logs

You should be able to right click on the Debug Console and choose Copy All. You can also use logDest in launch.json to redirect to a file. From dlv help log:

--log-dest can be used to specify where the logs should be written. 
If the argument is a number it will be interpreted as a file descriptor,
otherwise as a file path.

[lucky-wolf]

@polinasok thanks for the feedback!
I just fixed the font-size issue in debug-console - so that helps. And yes - copy all will be my goto from now on. It would be nice to be able to click to set a cursor position, and shift click on a new position to select between those!

Similarly, if I could drag above / below the current window so it scrolled while selecting - that'd be useful.

But neither of those exist, so selecting a chunk is nigh-impossible.

All will work (and I can edit that in another text editor after that).

Any improvements to make the setting of a BP during "running" state auto-resume would be nice. Messaging me to say "app paused to set new BP - press continue to resume..." would be helpful. Thanks.

[lucky-wolf]

Since I've got a platform to make some comments, I'll add one more:

It would be nice if the dlv-dap captured the remaining debug output once I hit "stop".

I think you disconnect before the app is actually shut down? So any final log output is missed from the debug console.

That or dlv-dap still only kills dlv and fails to send a stop interrupt to the underlying debug process? My graceful shutdown code (which triggers on such an interrupt) is not visible in the debug console, but I am certain works if I send a stop from some other mechanism (e.g. a rest call that issues it against itself).

Preferrable to have dlv-dap send a terminate / stop interrupt rather than just hard killing the process immediately. Or for that to be a configurable option anyway (in the launch settings + user + project defaults?)

[hyangah]

@lucky-wolf you can access equivalent tracing info from "Go Debug" output channel which in my opinion is a bit easier to select/copy/paste than the Debug Console where various javascript runs because it's primarily targeted for repl use.
https://github.com/golang/vscode-go/blob/master/docs/dlv-dap.md#testing

I think you disconnect before the app is actually shut down? So any final log output is missed from the debug console.

Triaging and inspecting the issue will be much easier if you can share the trace/log.

That or dlv-dap still only kills dlv and fails to send a stop interrupt to the underlying debug process? My graceful shutdown code (which triggers on such an interrupt) is not visible in the debug console, but I am certain works if I send a stop from some other mechanism (e.g. a rest call that issues it against itself).

For graceful shutdown, there is an open issue #120 I believe dlv-dap has the same issue.

[polinasok]

@lucky-wolf Just to add to what @hyangah wrote. The logging in Go Debug output channel is enabled with "trace" in your launch.json. It is true that after things start shutting down the Debug Console stops logging. Its life span is associated with the interactive user portion of the debug session, which is truncated, comparing to the life span of all the moving parts involved in setting up and shutting down this debug session. That's why we also send logging to Go Debug output channel. But it would not have the output from your application. We do not do anything special to shut it down, just whatever delve normally does (e.g. what you would get with dlv on the command-line). You can see exactly what gets triggered by running the dlv dap server externally and connecting to it from the editor. Then all the logging from dlv-dap and from your process will go to your terminal window.

dlv dap --listen=:12345 --log --log-output=dap,debugger

Then add "debugServer" to your launch configuration to tell vscode to connect to the above server instead of launching a new one.

Edit: recommend using debugServer to attach to debug server.

[polinasok]

Just to add about shutdown. Dlv-dap is supposed to shut down gracefully if it receives an explicit disconnect request from the editor or is killed with SIGTERM or SIGINT (which is what we do to force-kill it from the extension if the server becomes unresponsive). If target was launched, it should be taken down by default as well (unless you toggle the stop/disconnect option in the client to change the default). If you attached to an application, it should keep running (unless you toggle stop/disconnect option in the client to change the default).