Skip to content

Adds resource attribute enricher #376

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

noob-se7en
Copy link

@noob-se7en noob-se7en commented Mar 10, 2023

Description

This PR enriches the spans using their resource tags adding additional infra metadata context. Infra metadata context can be quite useful in scenarios like:

  1. Users want to debug an application behaving abnormally in a certain cluster. The application could be running in multiple clusters but Hypertrace ignores the resource tags and users cannot differentiate via resource tags.
  2. Users want to see metrics of applications running in different environments. (Example: P99 latency of service running in stage cluster or any k8s node etc.)

Resource attributes can be attached to spans using agents or collectors before the span reaches the Hypertrace ingestion pipeline. Few ways to attach infra metadata context to spans:

  1. Using the K8s attribute processor of otel-collector to append resource tags with infra tags.
  2. Using agents running on nodes like Otel-agent or Jaeger agent to attach tags like - (cluster.name, node.name) with the help of environment variables.

Screenshot 2023-03-11 at 12 45 10 AM

Sample Enricher config:

ResourceAttributeEnricher {
    class = "org.hypertrace.traceenricher.enrichment.enrichers.ResourceAttributeEnricher"
    attributes = ["pod.name","node.name","cluster.name","ip","service.version","node.selector"]
    attributesToMatch {
        pod.name = "host.name"
    }
  }

_ class_ : Java class to refer for this enricher.
attributes : Resource attributes which would be added to the span attribute if they exist in the process tags.
attributesToMatch: Could be used if a resource attribute needs to refer to some other attribute (For example the Jaeger client library would add k8s pod name to span process tags as 'host.name').

Testing

Added required test cases and the change is live on the production cluster for 3+ months now.

Checklist:

  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • Any dependent changes have been merged and published in downstream modules

Documentation

N/A

@noob-se7en noob-se7en requested a review from a team March 10, 2023 19:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants