feat(otel): add OpenTelemetry SpanProcessor, Propagator, Extractor #779

lcian · 2025-04-30T12:48:22Z

Creates a new crate sentry-opentelemetry for integration with OTEL.

The implementation is based on a SpanProcessor that creates and finishes Sentry spans on OTEL span start/end.
There's also a Propagator/Extractor that injects the metadata (e.g. HTTP headers) required for distributed tracing to work.
It's based on our implementation for JS and other SDKs (https://develop.sentry.dev/sdk/telemetry/traces/opentelemetry/).

The goal with this initial version is:

support OTEL <> Sentry span conversion and capturing
capture OTEL span attributes
link any messages/errors captured through the Sentry API to the correct spans, assuming that only the OTEL API is used to manage spans

Feedback appreciated on:

SPAN_MAP, general Rust stuff, additions to performance API (@Swatinem)
OTEL/SDKs things (@adinauer & @sl0thentr0py)

There are some todo items which will be addressed in follow-up PRs (not necessarily in this release).

lcian · 2025-04-30T12:55:01Z

sentry-opentelemetry/src/processor.rs

+/// A mapping from Sentry span IDs to Sentry spans/transactions.
+/// Sentry spans are created with the same SpanId as the corresponding OTEL span, so this is used
+/// to track OTEL spans across start/end calls.
+type SpanMap = Arc<Mutex<HashMap<sentry_core::protocol::SpanId, TransactionOrSpan>>>;
+
+static SPAN_MAP: LazyLock<SpanMap> = LazyLock::new(Default::default);


Is there a better way?
We need this to have a static lifetime for it to be used in the event processor.

as we discussed, at least for the event_processor, all we need are the span_id and trace_id, and we can losslessly convert those directly from the otel_span.

sentry-opentelemetry/src/converters.rs

sentry-opentelemetry/src/processor.rs

sentry-opentelemetry/src/propagator.rs

lcian · 2025-04-30T15:24:45Z

Seems like Windows CI is getting stuck on test_captures_transaction for some reason. I'll stop it for now.

sentry-opentelemetry/src/processor.rs

adinauer

Just gave this a quick glance and left some comments on what I noticed so far.

sentry-opentelemetry/src/propagator.rs

sentry-opentelemetry/src/processor.rs

sentry-opentelemetry/src/propagator.rs

lcian · 2025-05-02T08:05:48Z

Just gave this a quick glance and left some comments on what I noticed so far.

Thank you!

lcian · 2025-05-02T10:12:39Z

sentry-opentelemetry/src/propagator.rs

+    Context, SpanId, TraceId,
+};
+
+const SENTRY_TRACE_HEADER: &str = "sentry-trace";


I guess we should also consider traceparent according to https://develop.sentry.dev/sdk/telemetry/traces/distributed-tracing/

lcian · 2025-05-02T17:13:26Z

Apparently we need to put each test in a different file or they will get stuck on Windows.

Swatinem

this looks reasonable to me.
Its a bit unfortunate how the lack of "onlyspans" means that you have to still deal with needing access to a TransactionOrSpan object instead of just being able to push finished spans with opaque parent ids into a global span sink.

Swatinem · 2025-05-05T12:19:35Z

sentry-core/src/performance.rs

+    #[cfg(feature = "client")]
+    {
+        let client = Hub::with_active(|hub| hub.client());
+        let transaction = Transaction::new(client, ctx);
+        if let Some(tx) = transaction.inner.lock().unwrap().transaction.as_mut() {
+            tx.start_timestamp = timestamp;
+        }
+        transaction
+    }
+    #[cfg(not(feature = "client"))]
+    {
+        let transaction = Transaction::new_noop(ctx);
+        if let Some(tx) = transaction.inner.lock().unwrap().transaction.as_mut() {
+            tx.start_timestamp = timestamp;
+        }
+        transaction
+    }


Suggested change

#[cfg(feature = "client")]

{

let client = Hub::with_active(|hub| hub.client());

let transaction = Transaction::new(client, ctx);

if let Some(tx) = transaction.inner.lock().unwrap().transaction.as_mut() {

tx.start_timestamp = timestamp;

}

transaction

}

#[cfg(not(feature = "client"))]

{

let transaction = Transaction::new_noop(ctx);

if let Some(tx) = transaction.inner.lock().unwrap().transaction.as_mut() {

tx.start_timestamp = timestamp;

}

transaction

}

let transaction = start_transaction(ctx);

if let Some(tx) = transaction.inner.lock().unwrap().transaction.as_mut() {

tx.start_timestamp = timestamp;

}

transaction

lets deduplicate the hub/feature logic here

Swatinem · 2025-05-05T12:20:21Z

sentry-core/src/performance.rs

+        ctx: TransactionContext,
+        timestamp: SystemTime,
+    ) -> Transaction {
+        #[cfg(feature = "client")]


same thing here

Swatinem · 2025-05-05T12:32:15Z

sentry-opentelemetry/src/converters.rs

+    let mut res: BTreeMap<String, Value> = BTreeMap::new();
+    attributes.iter().for_each(|key_val| {
+        res.insert(
+            key_val.key.as_str().into(),
+            convert_value(key_val.value.clone()),
+        );
+    });
+    res


Suggested change

let mut res: BTreeMap<String, Value> = BTreeMap::new();

attributes.iter().for_each(|key_val| {

res.insert(

key_val.key.as_str().into(),

convert_value(key_val.value.clone()),

);

});

res

attributes.into_iter().map(|key_val| {

(

key_val.key.as_str().into(),

convert_value(key_val.value),

)

}).collect()

Swatinem · 2025-05-05T12:35:17Z

sentry-opentelemetry/src/processor.rs

+/// A mapping from Sentry span IDs to Sentry spans/transactions.
+/// Sentry spans are created with the same SpanId as the corresponding OTEL span, so this is used
+/// to track OTEL spans across start/end calls.
+type SpanMap = Arc<Mutex<HashMap<sentry_core::protocol::SpanId, TransactionOrSpan>>>;
+
+static SPAN_MAP: LazyLock<SpanMap> = LazyLock::new(Default::default);


as we discussed, at least for the event_processor, all we need are the span_id and trace_id, and we can losslessly convert those directly from the otel_span.

Swatinem · 2025-05-05T12:42:19Z

sentry-opentelemetry/src/processor.rs

+            span_data
+                .as_ref()
+                .map(|data| &data.parent_span_id)


you are doing this span_data.as_ref().map() in a bunch of places. maybe moving all those into a if let Some(span_data) = span.exported_data() block might make things a bit clearer?

Swatinem · 2025-05-05T12:49:58Z

sentry-opentelemetry/src/processor.rs

+                resource: convert_attributes(
+                    resource
+                        .iter()
+                        .map(|(key, value)| KeyValue::new(key.clone(), value.clone()))
+                        .collect(),
+                ),


Suggested change

resource: convert_attributes(

resource

.iter()

.map(|(key, value)| KeyValue::new(key.clone(), value.clone()))

.collect(),

),

resource: resource

.iter()

.map(|(key, value)| (key.as_str().into(), convert_value(value.clone())))

.collect(),

Maybe this can avoid an intermediate Vec allocation and having to clone the keys.

Swatinem · 2025-05-05T13:01:53Z

sentry-opentelemetry/Cargo.toml

+
+[features]
+default = ["release-health"]
+release-health = ["sentry-core/release-health"]


is this used anywhere?

This brings in release health features from core, it's used under the hood after you init the SDK. I guess we can also add release-health directly on the sentry-core dependency instead of making it a separate feature flag for this crate as well.

Swatinem · 2025-05-05T13:03:12Z

sentry-opentelemetry/src/context.rs

+impl FromStr for SentrySpanContext {
+    type Err = ParseContextError;
+
+    fn from_str(sentry_trace: &str) -> Result<Self, Self::Err> {


I’m a bit surprised we don’t have this yet? This, including the logic of the Propagator should ideally be covered by our from_headers and iter_headers functions.

Swatinem · 2025-05-05T13:04:38Z

sentry-opentelemetry/tests/associates_event_with_span.rs

+    // Create root span
+    let root_span = tracer.start("root_span");
+    let cx = Context::current_with_span(root_span);
+
+    // Create child span
+    let child_span = tracer.start_with_context("child_span", &cx);
+    let child_cx = cx.with_span(child_span);
+
+    // Set child span as active on current thread
+    let child_cx_guard = child_cx.attach();
+
+    // Capture an event while the child span is active
+    sentry::capture_message("Test message", sentry::Level::Error);
+
+    // End the spans
+    drop(child_cx_guard);
+    cx.span().end();


does opentelemetry not have some kind of attribute macro that decorates functions, so you wan just create a callchain that way?

these tests look a bit fragile when you have to manually create the span hierarchy.

It appears that there are no such macros, this means that this is the only possible way for end users to instrument their app when using OTEL in Rust 😅
There is this proposal to add macros similar to those in tracing open-telemetry/opentelemetry-rust#1388 but it didn't get much activity
I suppose most people are using tracing-opentelemetry, because this API just sucks

Swatinem · 2025-05-05T13:07:31Z

sentry-opentelemetry/src/processor.rs

+            .map(|data| data.start_time)
+            .unwrap_or_else(SystemTime::now);
+
+        let mut span_map = SPAN_MAP.lock().unwrap();


Instead of this being a static, this can live on the SentrySpanProcessor.

lcian added 3 commits April 30, 2025 14:32

feat(otel): add OpenTelemetry SpanProcessor, Propagator, Extractor

18fcd62

merge master

6b89829

update

b7f83a7

lcian marked this pull request as ready for review April 30, 2025 12:52

lcian requested review from Swatinem, adinauer and sl0thentr0py April 30, 2025 12:52

lcian commented Apr 30, 2025

View reviewed changes

sentry-opentelemetry/src/propagator.rs Outdated Show resolved Hide resolved

lcian added 3 commits April 30, 2025 16:48

set kind on transaction as well

dddec25

replace unknown span op with empty string

351cfae

replace empty op with description

9a4add0

lcian added 2 commits April 30, 2025 17:31

try serializing tests

5b068c6

on test init, close existing client if any

51cf84f

lcian commented Apr 30, 2025

View reviewed changes

sentry-opentelemetry/src/processor.rs Outdated Show resolved Hide resolved

remove useless line

cde1ada

lcian linked an issue Apr 30, 2025 that may be closed by this pull request

OpenTelemetry initial support #780

Open

adinauer reviewed May 2, 2025

View reviewed changes

lcian commented May 2, 2025

View reviewed changes

lcian added 5 commits May 2, 2025 12:34

address comments

919c4c3

simplify logic and capture span attributes

357c8bc

fix tests

820416a

restructure tests

4ac1fa0

clippy

532f398

lcian added 2 commits May 3, 2025 11:12

Update shared.rs

707dfc0

Update shared.rs

2b16c11

Swatinem reviewed May 5, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(otel): add OpenTelemetry SpanProcessor, Propagator, Extractor #779

feat(otel): add OpenTelemetry SpanProcessor, Propagator, Extractor #779

lcian commented Apr 30, 2025 •

edited

Loading

lcian Apr 30, 2025

Swatinem May 5, 2025

lcian commented Apr 30, 2025

adinauer left a comment

lcian commented May 2, 2025

lcian May 2, 2025

lcian commented May 2, 2025

Swatinem left a comment

Swatinem May 5, 2025

Swatinem May 5, 2025

Swatinem May 5, 2025

Swatinem May 5, 2025

Swatinem May 5, 2025

Swatinem May 5, 2025

Swatinem May 5, 2025

lcian May 5, 2025

Swatinem May 5, 2025

Swatinem May 5, 2025

lcian May 5, 2025

Swatinem May 5, 2025

feat(otel): add OpenTelemetry SpanProcessor, Propagator, Extractor #779

Are you sure you want to change the base?

feat(otel): add OpenTelemetry SpanProcessor, Propagator, Extractor #779

Conversation

lcian commented Apr 30, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lcian commented Apr 30, 2025

adinauer left a comment

Choose a reason for hiding this comment

lcian commented May 2, 2025

Choose a reason for hiding this comment

lcian commented May 2, 2025

Swatinem left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lcian commented Apr 30, 2025 •

edited

Loading