runtime: gdb command "goroutine 1 bt" fails on core file

[kayuuzu]

What version of Go are you using (go version)?

go version go1.5.3 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/ldata/comp/project/go"
GORACE=""
GOROOT="/ldata/bin/go"
GOTOOLDIR="/ldata/bin/go/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT=""
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

What did you do?

1. export GOTRACEBACK=crash
2. ulimit -c unlimited
3. write a buggy program and run it to generate a core file
4. use gdb to load core file
5. source runtime-gdb.py
6. run goroutine 1 bt

What did you expect to see?

print stack trace of goroutine 1

What did you see instead?

print error msg:
Python Exception <class 'gdb.error'> You can't do that without a process to debug.:
Error occurred in Python command: You can't do that without a process to debug.

[kayuuzu]

runtime-gdb.py seems to set pc&sp registers when you run goroutine 1 bt, but registers are read-only after you load core file.

[quentinmit]

I'm not sure what option we have here instead of setting $pc and $sp.

/cc @aclements @cherrymui

[aclements]

Unfortunately, before GDB 7.10, setting $pc and $sp is our only option. For GDB 7.10 and up, I believe we can use "unwinder" support, which is a much better way to do this. Not having GDB 7.10, I can't easily try this out. :) Ubuntu 16.04LTS has GDB 7.11, so this is probably becoming more widely available.

[aclements]

BTW, if someone does want to try this out with GDB 7.10+, I think the code would be something like:

try:
    from gdb.unwinders import Unwinder
except ImportError:
    Unwinder = None

if Unwinder is not None:
    class FrameID(object):
        def __init__(self, sp, pc):
            self.sp = sp
            self.pc = pc

    class GoUnwinder(Unwinder):
        def __init__(self):
            super(GoUnwinder, self).__init___("go-unwinder")

        def __call__(pending_frame):
            # This only applies to the first frame.
            self.enabled = False

            # Ignore registers in pending_frame. Use stashed PC/SP.
            return pending_frame.create_unwind_info(self.frame_id)
    goUnwinder = GoUnwinder()

# ... in GoroutineCmd ...
goUnwinder.frame_id = FrameID(sp, pc)
goUnwinder.enabled = True
try:
    gdb.execute(cmd)
finally:
    goUnwinder.enabled = False

This is completely untested.

[aclements]

FWIW, I recently wrote a gdb script that addresses this by copying the core file and rewriting the PC/SP saved in the core file for thread 1. You can then open this new core file and backtrace from it. It's awful, but it works (at least, on linux/amd64). https://gist.github.com/aclements/8d2d6a1d1ade4bc4fd492db6d3eb07a5

[kayuuzu]

@aclements Good job. I will try it.

[mail2fish]

Is there any update for this issue?

[bradfitz]

@mail2fish, any updates would be posted here. This is not happening for Go 1.9 apparently.

[aclements]

I got my hands on a more recent version of GDB that supports the frame unwinders API and completely failed to get it to do anything useful. If anyone wants to pick up where I left off, here's the code that monkey-patches in the unwinder: https://gist.github.com/aclements/e8b4b3d887ccc9fd2907e696a8b4b2a2. It gets into the custom unwinder, but GDB seems to ignore the updated unwind info it returns. My next step would probably be to run GDB under GDB and figure out what's going on inside.