pprof: add support for profiler labels

[matloob]

I propose adding a mechanism to the runtime and profiling code to allow for annotating profile samples with key-value labels. These labels are an already-existing feature of pprof that we do not support yet.

The user interface would look something like this:

    package context

    type ProfileLabel struct {
        Key string
        Value string
    }

    // DoWithLabels calls f with a copy of the parent context with the
    // given labels appended. The set of labels on that context will be set
    // for the duration of the call to f and restored once f returns.
    func DoWithLabels(parent Context, labels []ProfileLabel, f func(ctx Context))

More details in the proposal document

[gopherbot]

CL https://golang.org/cl/30015 mentions this issue.

[matloob]

The proposal document has been submitted to the proposal repo. Please take a look.

https://github.com/golang/proposal/blob/master/design/17280-profile-labels.md

[quentinmit]

At first glance, I find the high-level API confusing. You're proposing taking the Context type, which already encapsulates a key-value store, and adding a second separate key-value store to it. I especially find the name "DoWithLabels" confusing - I naively expect it to behave similarly to WithValue, but it doesn't.

To my mind, this would be resolved by moving this outside the context package. I think this can just be a regular consumer of the Context type. i.e. we could move this to pprof.DoWithContext, and it can store the profiling labels in a regular Context key using context.WithValue.

It feels unfortunate to me that this has to be DoWithLabels and not just WithLabels returning a new Context. I understand how that is necessary, though, given the runtime API.

The fact that the argument is func(context.Context) means it is cumbersome to pass back any values such as an error from the function. Perhaps the func type should be func(context.Context) error? "Generics would help here".

Finally, don't you need a way to read out the current labels to pass them along with an RPC? It is my understanding that this is one of the underlying goals of the proposal.

[ianlancetaylor]

I don't see how it makes sense to store all the Context keys as profiling labels.

Personally I think it's fine to use func(context.Context) as the closure can be used for any other values. I don't feel strongly about it, though.

For an RPC I think the expectation is that the specific appropriate labels would be applied by the RPC server. There is no necessary or appropriate correspondence between the complete set of labels used by the client and those used by the server.

[matloob]

I don't have a strong opinion about the package DoWithLabels appears in, but I think context is a sensible place to place it. runtime/pprof is okay also.

I have a slight preference for func(context.Context) just because it's a shorter signature. I think it's a wash between the clunkiness of assigning an error in the closure vs. returning an error and immediately assigning it when DoWithContext returns. But the context is necessary, so what we have now is the bare minimum.

[matloob]

Here's a change with a prototype of the interface (but not actually applying the labels to profiles): https://go-review.googlesource.com/c/31145/

[gopherbot]

CL https://golang.org/cl/31145 mentions this issue.

[quentinmit]

@ianlancetaylor Not sure if that was directed at me, but I did not propose storing all the Context keys as profiling labels. I proposed, roughly:

package pprof
type ctxKeyType struct{}
var ctxKey ctxKeyType

func DoWithLabels(ctx context.Context, new Labels, f func(context.Context)) {
  labels := ctx.Value(ctxKey)
  newLabels = mergeLabels(labels, new)
  setLabels(newLabels)
  defer setLabels(labels)
  f(context.WithValue(ctx, ctxKey, newLabels))
}

This can be done in any package, and does not require any change to the Context interface or implementation.

[matloob]

@quentinmit I think we're all in agreement then... That's exactly how we want DoWithLabels to work. See the prototype here: https://go-review.googlesource.com/c/31145/6/src/context/context.go

I think the only question here is where DoWithLabels belongs. It coulb be in package context or package pprof. I don't have a strong preference.

[quentinmit]

Yep, I think we're all in agreement about how DoWithLabels should work. I still think that it needs a better name and/or a different package, because otherwise I find the current state confusing.

As for reading labels, your proposal currently says:

// ProfileLabels is an immutable map of profiler labels. A nil
// *ProfileLabels is an empty map of labels.
// There is intentionally no way to access the profile labels contained
// inside the ProfLabels because doing so could create a goroutine-local
// storage mechanism.
type ProfileLabels struct { /* runtime-internal unexported fields */ }

From talking to @ianlancetaylor, the thing we are trying to prevent is goroutine-local storage. I missed the fact that runtime.SetProfileLabels returns a pointer to the previous ProfileLabels struct. I don't see why we need that instead of just restoring the labels from the context as my code above does. Then ProfileLabels would not need to be opaque.

Either way, we need to add a way to get at the labels in a context. This could be a func pprof.LabelsFromContext(context.Context) [][2]string or similar (in the same package that DoWithLabels ends up in).

[matloob]

@quentinmit That comment is obsolete. I've sent a cl to update it. The "compromise" solution is to allow reading labels, but not on ProfileLabels returned by the runtime. So that allows us to read labels we keep in the context (because they're never set from a ProfileLabels returned by the runtime), while disallowing GLS.

It would be nice to have LabelsFromContext, but I don't think it's strictly necessary (at least for now). You can get all the nice pprof benefits of adding labels without needing to read the labels on a context. It would be really easy to add LabelsFromContext later. -- I'm not saying that we shouldn't add it, but we can (if we want) wait until later to do so.

Another option is to have LabelsFromContext return a runtime.ProfileLabels. Any ProfileLabels set on a context using DoWithLabels should be readable.

[matloob]

@quentinmit Here's an example of why we'd want to restore the previous ProfileLabels set on the goroutine rather than the parent context's ProfileLabels: Let's say a user wants to do some work that they don't want to have accounted to the current context. They might have code that looks like this:

 DoWithLabels(context.Background(), nil, func(ctx context.Context) { ... } )

When the DoWithLabels returns, they probably want to revert to the context already set on the goroutine rather than the background context.