runtime: add 'created by goroutine number' to stack traces

[firelizzard18]

Per @ianlancetaylor's comment on #35178 (comment), I propose improving the debugging experience by including goroutine tree information in stack dumps and providing that information to debuggers (delve).

I propose that created by file.go:col in stack dumps is expanded to created by goroutine # at file.go:col, and that goroutine tree information is provided to debuggers like delve.

Ian's original comment:

If the problem we are trying to address is better support for understanding large numbers of goroutines in stack dumps and when debugging, then let's discuss that problem. Let's not jump to the idea of goroutine names, which have many drawbacks in a language like Go where goroutines are started casually. Maybe goroutine names are the best idea we can come up with, but that conclusion seems premature given that we haven't even started talking about the actual problem.

For example, one thing that might help is giving the stack dump, and debuggers, access to the goroutine tree, so that you can see clearly that goroutine N was started by goroutine M. You can see this a bit today by using GODEBUG=gotracebackancestors=N for various integer N. Or I'm sure there are other better ideas out there.

In general debuggers do not do well when there are many separate threads of executions, because most languages do not make it trivial to start many separate threads of executions. We need to do better in this area.

[rsc]

We'd need to add this to the g struct and print it.
Debuggers can then fetch it because they look at the g structs.
Sounds like a great idea.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[rsc]

Does anyone object to adding this?

[kaey]

I'd rather want to see pprof.Labels in stacktraces, as I suggested in linked issue.

But if adding parent goroutine ID is easy to do then why not I guess.

[kortschak]

It seems to me that this would not give access to a full tree view into the goroutine's origins since when we have the creator's ID, we cannot easily step beyond that to its creator since we have an ID, not a g.

[firelizzard18]

@kortschak The only time I can think of that you'd be building a tree view of goroutines is during a debug session. In that case the debugger could build a map from ID to g and the performance impact on a human scale should be negligible. But that's only relevant if there's a downside to adding parent *g to g instead of parentId int (I have no idea).

[kaey]

Parent goroutine can very well be gone at any point while the child is still running. Will knowing ID be even useful in that case?

[firelizzard18]

@kaey Good point and probably a good reason why we couldn't add parent *g to g.

It's been a while since I created this and I don't honestly remember what my motivation was. I think the main issue is figuring out what caused a goroutine to be spawned. If the parent is dead and I don't know the parent's parent, then knowing the parent's ID is probably useless. However if I am able to reconstruct the goroutine chain, I might be able to figure out the chain of events that lead to the last goroutine getting spawned.

The cases I can think of are things like an event loop spawning goroutines to process events, or a socket accept loop spawning goroutines to handle connections. In those cases the parent goroutine should be long-lived.

[rsc]

This is very simple to implement. It has limitations that more comprehensive schemes might not, but it's still very likely useful on its own, far in excess of its cost. So it sounds like we should do this, and we can leave any suggested heavier schemes for other proposals.

[rsc]

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

[rsc]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

[gopherbot]

Change https://go.dev/cl/435337 mentions this issue: runtime: record parent goroutine ID, and print it in stack traces

[gopherbot]

Change https://go.dev/cl/501336 mentions this issue: doc/go1.21: add release notes for parent goroutine in stack traces