Add request to Observation Context at creation to enable filtering by the request

[jonatan-ivanov]

This issue was originally discussed in spring-projects/spring-boot#34400.
The scenario is the following: if Spring Security is used and users want to ignore /actuator endpoints, there is no easy way to ignore Observations created by Spring Security because the details to make this decision are missing when the ObservationPredicate is tested.

One potential solution would be adding the request to the Context before the Observation is created, one example here:

spring-security/web/src/main/java/org/springframework/security/web/ObservationFilterChainDecorator.java

Lines 195 to 202 in ac1d269

	private AroundFilterObservation parent(HttpServletRequest request) {
	FilterChainObservationContext beforeContext = FilterChainObservationContext.before();
	FilterChainObservationContext afterContext = FilterChainObservationContext.after();
	Observation before = Observation.createNotStarted(this.convention, () -> beforeContext, this.registry);
	Observation after = Observation.createNotStarted(this.convention, () -> afterContext, this.registry);
	AroundFilterObservation parent = AroundFilterObservation.create(before, after);
	request.setAttribute(ATTRIBUTE, parent);
	return parent;

Would be something like:

FilterChainObservationContext beforeContext = FilterChainObservationContext.before(request);
FilterChainObservationContext afterContext = FilterChainObservationContext.after(request);

Expected behavior: having extra details on the Observations created by Spring Security that allows the users to ignore these Observation based on the request.

There is a reproducer in the issue mentioned above.

[jzheaux]

Thanks for the suggestion, @jonatan-ivanov. Can you elaborate on how the request details would be read when evaluating the ObservationPredicate?

Based on spring-projects/spring-boot#34400 I wonder if it should add http.url.

[jonatan-ivanov]

It can be an uri field on the context, a tag or for full flexibility, does the whole request make sense?

I think something like this:

@Bean
ObservationPredicate actuatorServerContextPredicate() {
    return (name, context) -> {
        if (name.startsWith("spring.security.") && context instanceof SecurityObservationContext secObsContext) {
                return !secObsContext.getCarrier().getRequestURI().startsWith("/actuator");
        }
        return true;
    };
}

But as an uri field this would be quite similar: secObsContext.getRequestURI().startsWith("/actuator")

Here is a working example excluding the actuator http spans.

[jzheaux]

In that case, would someone that is trying to turn off spans by request need to know how specifically how to turn off each project's spans? I'm wondering, for example, if FilterChainObservationContext should instead implement some sort of request-aware observability interface.

Perhaps I'm not seeing the whole picture, but from where I'm standing, I'm seeing an application that needs to know Spring Security specifics -- as well as the specifics of other modules -- to perform an observability-level task of filtering out spans from a request.

[braunsonm]

In that case, would someone that is trying to turn off spans by request need to know how specifically how to turn off each project's spans?

That's the exact problem I have with this solution. And once that this change in micrometer will also not solve: micrometer-metrics/micrometer#3678

It seems like the only way to do this properly is as mentioned on the original issue, by adding an attribute on every span you have in order to ignore them with a predicate.

(From Jonathan on the original issue)

// This adds the http.url keyvalue to security observations from the root (mvc) observation
// You add an ignoreSpan=true keyValue instead if you want, or something that can signal to the SpanExportingPredicate what to ignore
@Bean
ObservationFilter urlObservationFilter() {
    return context -> {
        if (context.getName().startsWith("spring.security.")) {
            Context root = getRoot(context);
            if (root.getName().equals("http.server.requests")) {
                context.addHighCardinalityKeyValue(root.getHighCardinalityKeyValue("http.url"));
            }
        }

        return context;
    };
}

private Observation.Context getRoot(Observation.Context context) {
    if (context.getParentObservation() == null) {
        return context;
    }
    else {
        return getRoot((Context) context.getParentObservation().getContextView());
    }
}

// This ignores actuator spans but its logic depends on the ObservationFilter above
// Auto-configuration for SpanExportingPredicate was added in 3.1.0-M1
// So either you use 3.1.x or you can add the same to your config : https://github.com/spring-projects/spring-boot/pull/34002
@Bean
SpanExportingPredicate actuatorSpanExportingPredicate() {
    return span -> !span.getTags().get("http.url").startsWith("/actuator");
}

[jonatan-ivanov]

In that case, would someone that is trying to turn off spans by request need to know how specifically how to turn off each project's spans?

Filtering out spans from anywhere in the flow can be a valid use-case I think (this is about Observations but that does not change much), I think saying that http Observations are off for actuator but Observation XYZ that is triggered by that HTTP call is not is a completely valid use-case though I feel not as frequent.

[jzheaux]

Filtering out spans from anywhere in the flow can be a valid use-case

Yes, that's the case I'm referring to. I imagine something like the following:

1. Look up the request in the context or its ancestry
2. Make a decision based on the request material

Or, in pseudocode, something like this:

ObservationPredicate isNotActuator = (name, context) -> {
    RequestAwareObservationContext requestAware = context.getFirstAncestorOfType(RequestAwareObservationContext.class);
    return requestAware == null || !requestAware.getCarrier().getRequestURI().endsWith("/actuator");
};

I'm not sure what other complexities I am ignoring or unaware of, so please take the code as just an observation.

Regardless of the best way to solve this, I think it would be better to expose the request in a non-Spring-Security-specific way.

[jonatan-ivanov]

@braunsonm Could you please try out one thing?
I talked to Josh who came up with the idea of getting the current request through RequestContextHolder.getRequestAttributes(). Since the "parent" http observation is attached to it, you can use it to filter out actuator, or any observations that was triggered by an HTTP request but its parent http observation is a noop (disabled):

@Bean
ObservationPredicate noRootlessHttpObservations() {
    return (name, context) -> {
        RequestAttributes requestAttributes = RequestContextHolder.getRequestAttributes();
        if (requestAttributes instanceof ServletRequestAttributes servletRequestAttributes) {
            Observation observation = (Observation) requestAttributes.getAttribute(ServerHttpObservationFilter.class.getName() + ".observation", SCOPE_REQUEST);
            return observation == null || !observation.isNoop();
        }
        return true;
    };
}