Should the `server.address` tag in the HttpClient metric counters incorporate the Host header of the request?

[Tragetaschen]

The MetricsHandler of the HttpClient currently sets the server.address tag and extracts the value from the request's URL. On the other hand, requests can explicitly set the Host header to overwrite the hostname information from the one in the given URL.

When using client-side load balancing in https://github.com/grpc/grpc-dotnet, the implementation does DNS lookups for the given hostname and then constructs IP-URLs that are the input for multiple round-robin http requests. These are configured to overwrite the Host header with the original hostname.

When looking at the exported metrics in that scenario, there is basically no way to recognize from the counters which gRPC service was actually contacted. Especially in a k8s cluster, the information which service had the IP might even be long gone.

By incorporating the overwritten Host header of the HttpRequest when generating the tag, that information could be retained.

Alternatively, an additional (non-standard?) tag might help as well.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[ManickaP]

This seems like a bug on our side based the OTel docs: https://github.com/open-telemetry/semantic-conventions/blob/main/docs/http/http-metrics.md#metric-httpclientactive_requests

But it also seems easy to fix. @antonfirsov do you think we should fix it now for 10.0?

[antonfirsov]

The metrics specs seem to contradict the http client span specs, which state that server.address should be the

Host identifier of the "URI origin" HTTP request is sent to. [2]

@lmolkova do you know if this is intentional?

[antonfirsov]

Alternatively, an additional (non-standard?) tag might help as well.

Note that it might be possible to inject such a tag via Enrichment. Since you are not using HttpClient directly, this might be more tricky AFAIK Grpc.Net API has IHttpClientFactory integration meaning you should be able to inject a custom handler that can intercept a request and add an enrichment callback.

[Tragetaschen]

As far as I can tell from the implementation, only the duration histogram is affected by the custom tags, not the active requests counter.

[antonfirsov]

Yes, but it might be possible to correlate them in your queries. I know the workaround is not ideal, I'm suggesting it because even if we decide to fix this, the fix will only ship in .NET 10 and unlikely to be backported.