runtime: gdb command "goroutine 1 bt" fails on core file

[kayuuzu]

What version of Go are you using (go version)?

go version go1.5.3 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/ldata/comp/project/go"
GORACE=""
GOROOT="/ldata/bin/go"
GOTOOLDIR="/ldata/bin/go/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT=""
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

What did you do?

1. export GOTRACEBACK=crash
2. ulimit -c unlimited
3. write a buggy program and run it to generate a core file
4. use gdb to load core file
5. source runtime-gdb.py
6. run goroutine 1 bt

What did you expect to see?

print stack trace of goroutine 1

What did you see instead?

print error msg:
Python Exception <class 'gdb.error'> You can't do that without a process to debug.:
Error occurred in Python command: You can't do that without a process to debug.

[kayuuzu]

runtime-gdb.py seems to set pc&sp registers when you run goroutine 1 bt, but registers are read-only after you load core file.

[quentinmit]

I'm not sure what option we have here instead of setting $pc and $sp.

/cc @aclements @cherrymui

[aclements]

Unfortunately, before GDB 7.10, setting $pc and $sp is our only option. For GDB 7.10 and up, I believe we can use "unwinder" support, which is a much better way to do this. Not having GDB 7.10, I can't easily try this out. :) Ubuntu 16.04LTS has GDB 7.11, so this is probably becoming more widely available.

[aclements]

BTW, if someone does want to try this out with GDB 7.10+, I think the code would be something like:

try:
    from gdb.unwinders import Unwinder
except ImportError:
    Unwinder = None

if Unwinder is not None:
    class FrameID(object):
        def __init__(self, sp, pc):
            self.sp = sp
            self.pc = pc

    class GoUnwinder(Unwinder):
        def __init__(self):
            super(GoUnwinder, self).__init___("go-unwinder")

        def __call__(pending_frame):
            # This only applies to the first frame.
            self.enabled = False

            # Ignore registers in pending_frame. Use stashed PC/SP.
            return pending_frame.create_unwind_info(self.frame_id)
    goUnwinder = GoUnwinder()

# ... in GoroutineCmd ...
goUnwinder.frame_id = FrameID(sp, pc)
goUnwinder.enabled = True
try:
    gdb.execute(cmd)
finally:
    goUnwinder.enabled = False

This is completely untested.

[aclements]

FWIW, I recently wrote a gdb script that addresses this by copying the core file and rewriting the PC/SP saved in the core file for thread 1. You can then open this new core file and backtrace from it. It's awful, but it works (at least, on linux/amd64). https://gist.github.com/aclements/8d2d6a1d1ade4bc4fd492db6d3eb07a5

[kayuuzu]

@aclements Good job. I will try it.

[mail2fish]

Is there any update for this issue?

[bradfitz]

@mail2fish, any updates would be posted here. This is not happening for Go 1.9 apparently.

[aclements]

I got my hands on a more recent version of GDB that supports the frame unwinders API and completely failed to get it to do anything useful. If anyone wants to pick up where I left off, here's the code that monkey-patches in the unwinder: https://gist.github.com/aclements/e8b4b3d887ccc9fd2907e696a8b4b2a2. It gets into the custom unwinder, but GDB seems to ignore the updated unwind info it returns. My next step would probably be to run GDB under GDB and figure out what's going on inside.

[occia]

So can I say, so far, there isn't a way to look the stack trace from a core dump of an exe written in Golang? I really need the answer, and a more detail question is at here thanks : )

[aclements]

@occia, the closest thing we have right now is the hack in #17575 (comment). Unfortunately, this is a limitation of GDB and there's not much we can do about it. In theory a custom unwinder should make it possible, but all of my experiments with custom unwinding failed to produce any results at all.

[heschi]

For posterity, dlv should work fine for this. dlv core <binary> <core file>.

[ptsneves]

Have been burning my few balding hairs trying to get a meaningful bt. gdb displays a bt but it is useless as it mostly shows OS threads bt not really go routines.

I was able to get an unwinder working but I think the way @aclements did it is not how the unwinders are supposed to work. We are supposed to use the frames passed and process them, not completely ignore them. I do not think that is a feasible approach.
I was able to get the core rewriter mentioned by @aclements working but there are some points to mention:

• One needs to manually get the sp and pc from the 'runtime.allgs' and plug them in the core generator
• The script is amd64 only
• I think that it does not take into account the thread, so registers may be not matching the go routines thread.

If i make further progress I will report

[ptsneves]

I finally was able to get a stack trace from a core dump with the unwinder work @aclements did. I am now trying to find if it is possible to use it for other arches. Unfortunately when I use the "pc" mnemonic it crashes GDB

btg <address of the g>

import gdb
import gdb.xmethod
from gdb.unwinder import Unwinder, register_unwinder


def command(fn):
    name = fn.__name__.replace('_', '-')
    dct = {
        '__doc__': fn.__doc__,
        '__init__': lambda self: super(cls, self).__init__(name, gdb.COMMAND_USER),
        'invoke': lambda self, *args, **kw: fn(*args, **kw),
    }
    cls = type(fn.__name__ + 'Cls', (gdb.Command,), dct)
    cls()
    return fn


class FrameId(object):
    def __init__(self, sp, pc):
        self._sp = sp
        self._pc = pc

    @property
    def sp(self):
        return self._sp

    @property
    def pc(self):
        return self._pc


class GoUnwinder(Unwinder):
    frameId = None

    def __init__(self):
        super(GoUnwinder, self).__init__("go-unwinder")
        self.enabled = False
        gdb.invalidate_cached_frames()

    def setFrameId(self, frame_id):
        self.enabled = True
        GoUnwinder.frameId = frame_id

    def __call__(self, pending_frame):
        pc = pending_frame.read_register("pc")
        sp = pending_frame.read_register("sp")

        unwind_info = pending_frame.create_unwind_info(FrameId(pc, sp))

        unwind_info.add_saved_register('rip', GoUnwinder.frameId.pc)
        unwind_info.add_saved_register('sp', GoUnwinder.frameId.sp)

        self.enabled = False
        return unwind_info


goUnwinder = GoUnwinder()
register_unwinder(None, goUnwinder, replace=True)


def btg1(g, m):
    # If the G is active on an M, find the thread of that M.
    if m is not None:
        for thr in gdb.selected_inferior().threads():
            if thr.ptid[1] == m['procid']:
                break
        else:
            thr = None
        if thr:
            # If this is the current goroutine, use a regular
            # backtrace since the saved state may be stale.
            curthr = gdb.selected_thread()
            try:
                thr.switch()
                cursp = gdb.parse_and_eval('$sp')
            finally:
                curthr.switch()
            if g['stack']['lo'] < cursp <= g['stack']['hi']:
                # if g['syscallsp'] != 0:
                #     sp, pc = g['syscallsp'], g['syscallpc']
                # else:
                #     sp, pc = g['sched']['sp'], g['sched']['pc']
                # goUnwinder.setFrameId(FrameID(sp, pc))
                gdb.execute('thread apply %d backtrace' % thr.num)
                return
        else:
            print("thread %d not found; stack may be incorrect (try import _ \"rutime/cgo\")" % m['procid'])

    # TODO: LR register on LR machines.
    if g['syscallsp'] != 0:
        sp, pc = g['syscallsp'], g['syscallpc']
    else:
        sp, pc = g['sched']['sp'], g['sched']['pc']

    goUnwinder.setFrameId(FrameId(sp, pc))
    gdb.invalidate_cached_frames()
    gdb.execute('backtrace')


#
# Go runtime helpers
#

class SliceValue:
    """Wrapper for slice values."""

    def __init__(self, val):
        self.val = val

    @property
    def len(self):
        return int(self.val['len'])

    @property
    def cap(self):
        return int(self.val['cap'])

    def __getitem__(self, i):
        if i < 0 or i >= self.len:
            raise IndexError(i)
        ptr = self.val["array"]
        return (ptr + i).dereference()


_Gdead = 6


def getg(sp=None, n=None) -> []:
    def checkg(GProc):
        if GProc == 0:
            return
        if GProc['atomicstatus'] == _Gdead:
            return
        if sp is not None and GProc['stack']['lo'] < sp <= GProc['stack']['hi']:
            if found[0] is not None:
                raise gdb.GdbError('multiple Gs with overlapping stacks!')
            found[0] = GProc
        if n is not None and GProc['goid'] == n:
            if found[0] is not None:
                raise gdb.GdbError('multiple Gs with same goid!')
            found[0] = GProc

    if sp is None and n is None:
        sp = gdb.parse_and_eval('$sp')

    found = [None]

    # Check allgs.
    for gp in SliceValue(gdb.parse_and_eval("'runtime.allgs'")):
        checkg(gp)

    # Check g0s and gsignals, which aren't on allgs.
    if sp is not None:
        mp = gdb.parse_and_eval("'runtime.allm'")
        while mp != 0:
            checkg(mp['g0'])
            checkg(mp['gsignal'])
            mp = mp['alllink']

    return found[0]


@command
def btg(arg, from_tty):
    """btg [g]: print a backtrace for G."""
    g = gdb.parse_and_eval(arg)

    if g is None or g == 0:
        print("no goroutine")
        return

    m = g['m']
    if m == 0:
        m = None
    else:
        if g == m['g0']:
            print("g0 stack:")
            btg1(g, m)
            print()
            g, m = m['curg'], None
        elif g == m['gsignal']:
            print("gsignal stack:")
            btg1(g, m)
            print()
            g, m = m['curg'], None

        if g == 0:
            print("no user goroutine")
            return

    print("goroutine %d stack:" % g['goid'])
    btg1(g, m)

[Scusemua]

I finally was able to get a stack trace from a core dump with the unwinder work @aclements did. I am now trying to find if it is possible to use it for other arches. Unfortunately when I use the "pc" mnemonic it crashes GDB
...

How does one get the address of the Goroutine in this context, for the script you provided? @ptsneves