Fix tracer OpenLineage instrumentation #9487

pawel-big-lebowski · 2025-09-08T08:35:07Z

What Does This Do

Fix service name tagging in OpenLineage events
Currently, the ol_service tag in OpenLineage events is derived from the sparkServiceName property, which only becomes available after OpenLineage instrumentation. This delay causes events to miss the service name, making backend correlation difficult.

This PR ensures that OpenLineage events always contain a service name:

If the user provides one, it is used directly.
If not, spark.app.name is used as a fallback.

This serves as a short-term solution. For the long term, the PR also introduces the _dd.ol_intake.base_hash tag, which backend systems can later leverage for robust service mapping.

Fix missing output statistics in OpenLineage events

OpenLineage output statistics depend on onTask events emitted by the Spark listener. The tracer instruments the OpenLineage listener, but currently it does not handle onTaskEnd correctly—the method returns without invoking the OpenLineage listener. As a result, outputStatistics are absent from OL events.

This PR addresses the issue by ensuring that onTaskEnd properly updates task metrics via the OpenLineage listener. The method is lightweight, as it only updates counters for task metrics.

Motivation

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

datadog-official · 2025-09-08T09:01:36Z

🎯 Code Coverage
• Patch Coverage: 100.00%
• Total Coverage: 57.79% (-0.03%)

View detailed report

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 0e9b724 | Docs | Was this helpful? Give us feedback!}

pr-commenter · 2025-09-08T09:12:30Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	pawel.leszczynski/fix-openlineae-instrumentation
git_commit_date	1757338973	1757401832
git_commit_sha	`03d997e`	`0e9b724`
release_version	1.54.0-SNAPSHOT~03d997e2fd	1.54.0-SNAPSHOT~0e9b72482d

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1757403664	1757403664
ci_job_id	1119856391	1119856391
ci_pipeline_id	75916439	75916439
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-8esvw843 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-8esvw843 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 46 metrics, 13 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.54.0-SNAPSHOT~0e9b72482d, baseline=1.54.0-SNAPSHOT~03d997e2fd

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.046 s) : 0, 1046424
Total [baseline] (10.712 s) : 0, 10712226
Agent [candidate] (1.052 s) : 0, 1052100
Total [candidate] (10.769 s) : 0, 10768526
section appsec
Agent [baseline] (1.222 s) : 0, 1221882
Total [baseline] (10.816 s) : 0, 10815665
Agent [candidate] (1.224 s) : 0, 1224340
Total [candidate] (10.856 s) : 0, 10856473
section iast
Agent [baseline] (1.182 s) : 0, 1181817
Total [baseline] (10.914 s) : 0, 10913540
Agent [candidate] (1.19 s) : 0, 1189592
Total [candidate] (10.998 s) : 0, 10998386
section profiling
Agent [baseline] (1.2 s) : 0, 1199755
Total [baseline] (10.902 s) : 0, 10901881
Agent [candidate] (1.199 s) : 0, 1198745
Total [candidate] (10.851 s) : 0, 10851150

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.046 s	-
Agent	appsec	1.222 s	175.458 ms (16.8%)
Agent	iast	1.182 s	135.393 ms (12.9%)
Agent	profiling	1.2 s	153.331 ms (14.7%)
Total	tracing	10.712 s	-
Total	appsec	10.816 s	103.44 ms (1.0%)
Total	iast	10.914 s	201.315 ms (1.9%)
Total	profiling	10.902 s	189.656 ms (1.8%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.052 s	-
Agent	appsec	1.224 s	172.241 ms (16.4%)
Agent	iast	1.19 s	137.492 ms (13.1%)
Agent	profiling	1.199 s	146.645 ms (13.9%)
Total	tracing	10.769 s	-
Total	appsec	10.856 s	87.947 ms (0.8%)
Total	iast	10.998 s	229.86 ms (2.1%)
Total	profiling	10.851 s	82.624 ms (0.8%)

gantt
    title petclinic - break down per module: candidate=1.54.0-SNAPSHOT~0e9b72482d, baseline=1.54.0-SNAPSHOT~03d997e2fd

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.454 ms) : 0, 1454
crashtracking [candidate] (1.445 ms) : 0, 1445
BytebuddyAgent [baseline] (731.34 ms) : 0, 731340
BytebuddyAgent [candidate] (735.559 ms) : 0, 735559
GlobalTracer [baseline] (242.094 ms) : 0, 242094
GlobalTracer [candidate] (243.623 ms) : 0, 243623
AppSec [baseline] (30.045 ms) : 0, 30045
AppSec [candidate] (30.358 ms) : 0, 30358
Debugger [baseline] (6.083 ms) : 0, 6083
Debugger [candidate] (6.098 ms) : 0, 6098
Remote Config [baseline] (692.621 µs) : 0, 693
Remote Config [candidate] (673.887 µs) : 0, 674
Telemetry [baseline] (13.75 ms) : 0, 13750
Telemetry [candidate] (13.137 ms) : 0, 13137
section appsec
crashtracking [baseline] (1.452 ms) : 0, 1452
crashtracking [candidate] (1.452 ms) : 0, 1452
BytebuddyAgent [baseline] (754.412 ms) : 0, 754412
BytebuddyAgent [candidate] (756.15 ms) : 0, 756150
GlobalTracer [baseline] (235.048 ms) : 0, 235048
GlobalTracer [candidate] (235.644 ms) : 0, 235644
IAST [baseline] (23.593 ms) : 0, 23593
IAST [candidate] (23.757 ms) : 0, 23757
AppSec [baseline] (168.915 ms) : 0, 168915
AppSec [candidate] (167.563 ms) : 0, 167563
Debugger [baseline] (6.612 ms) : 0, 6612
Debugger [candidate] (8.049 ms) : 0, 8049
Remote Config [baseline] (632.087 µs) : 0, 632
Remote Config [candidate] (631.829 µs) : 0, 632
Telemetry [baseline] (10.104 ms) : 0, 10104
Telemetry [candidate] (10.011 ms) : 0, 10011
section iast
crashtracking [baseline] (1.454 ms) : 0, 1454
crashtracking [candidate] (1.462 ms) : 0, 1462
BytebuddyAgent [baseline] (853.198 ms) : 0, 853198
BytebuddyAgent [candidate] (860.07 ms) : 0, 860070
GlobalTracer [baseline] (233.428 ms) : 0, 233428
GlobalTracer [candidate] (234.271 ms) : 0, 234271
IAST [baseline] (29.317 ms) : 0, 29317
IAST [candidate] (31.036 ms) : 0, 31036
AppSec [baseline] (27.809 ms) : 0, 27809
AppSec [candidate] (26.033 ms) : 0, 26033
Debugger [baseline] (6.635 ms) : 0, 6635
Debugger [candidate] (6.633 ms) : 0, 6633
Remote Config [baseline] (597.107 µs) : 0, 597
Remote Config [candidate] (593.603 µs) : 0, 594
Telemetry [baseline] (8.272 ms) : 0, 8272
Telemetry [candidate] (8.312 ms) : 0, 8312
section profiling
crashtracking [baseline] (1.423 ms) : 0, 1423
crashtracking [candidate] (1.422 ms) : 0, 1422
BytebuddyAgent [baseline] (763.002 ms) : 0, 763002
BytebuddyAgent [candidate] (762.533 ms) : 0, 762533
GlobalTracer [baseline] (223.008 ms) : 0, 223008
GlobalTracer [candidate] (223.059 ms) : 0, 223059
AppSec [baseline] (31.434 ms) : 0, 31434
AppSec [candidate] (30.477 ms) : 0, 30477
Debugger [baseline] (6.264 ms) : 0, 6264
Debugger [candidate] (6.228 ms) : 0, 6228
Remote Config [baseline] (722.27 µs) : 0, 722
Remote Config [candidate] (696.605 µs) : 0, 697
Telemetry [baseline] (15.573 ms) : 0, 15573
Telemetry [candidate] (16.219 ms) : 0, 16219
ProfilingAgent [baseline] (107.791 ms) : 0, 107791
ProfilingAgent [candidate] (107.701 ms) : 0, 107701
Profiling [baseline] (108.462 ms) : 0, 108462
Profiling [candidate] (108.361 ms) : 0, 108361

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.54.0-SNAPSHOT~0e9b72482d, baseline=1.54.0-SNAPSHOT~03d997e2fd

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.048 s) : 0, 1048177
Total [baseline] (8.649 s) : 0, 8648623
Agent [candidate] (1.042 s) : 0, 1042127
Total [candidate] (8.626 s) : 0, 8625792
section iast
Agent [baseline] (1.181 s) : 0, 1180509
Total [baseline] (9.383 s) : 0, 9382754
Agent [candidate] (1.179 s) : 0, 1179441
Total [candidate] (9.331 s) : 0, 9330898

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.048 s	-
Agent	iast	1.181 s	132.332 ms (12.6%)
Total	tracing	8.649 s	-
Total	iast	9.383 s	734.131 ms (8.5%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.042 s	-
Agent	iast	1.179 s	137.314 ms (13.2%)
Total	tracing	8.626 s	-
Total	iast	9.331 s	705.105 ms (8.2%)

gantt
    title insecure-bank - break down per module: candidate=1.54.0-SNAPSHOT~0e9b72482d, baseline=1.54.0-SNAPSHOT~03d997e2fd

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.456 ms) : 0, 1456
crashtracking [candidate] (1.445 ms) : 0, 1445
BytebuddyAgent [baseline] (732.272 ms) : 0, 732272
BytebuddyAgent [candidate] (729.998 ms) : 0, 729998
GlobalTracer [baseline] (242.647 ms) : 0, 242647
GlobalTracer [candidate] (241.548 ms) : 0, 241548
AppSec [baseline] (30.2 ms) : 0, 30200
AppSec [candidate] (29.944 ms) : 0, 29944
Debugger [baseline] (6.115 ms) : 0, 6115
Debugger [candidate] (6.065 ms) : 0, 6065
Remote Config [baseline] (691.928 µs) : 0, 692
Remote Config [candidate] (673.892 µs) : 0, 674
Telemetry [baseline] (13.778 ms) : 0, 13778
Telemetry [candidate] (11.473 ms) : 0, 11473
section iast
crashtracking [baseline] (1.453 ms) : 0, 1453
crashtracking [candidate] (1.447 ms) : 0, 1447
BytebuddyAgent [baseline] (852.122 ms) : 0, 852122
BytebuddyAgent [candidate] (851.541 ms) : 0, 851541
GlobalTracer [baseline] (233.183 ms) : 0, 233183
GlobalTracer [candidate] (232.991 ms) : 0, 232991
IAST [baseline] (30.548 ms) : 0, 30548
IAST [candidate] (28.522 ms) : 0, 28522
AppSec [baseline] (26.723 ms) : 0, 26723
AppSec [candidate] (28.403 ms) : 0, 28403
Debugger [baseline] (6.611 ms) : 0, 6611
Debugger [candidate] (6.603 ms) : 0, 6603
Remote Config [baseline] (604.543 µs) : 0, 605
Remote Config [candidate] (594.347 µs) : 0, 594
Telemetry [baseline] (8.222 ms) : 0, 8222
Telemetry [candidate] (8.282 ms) : 0, 8282

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	pawel.leszczynski/fix-openlineae-instrumentation
git_commit_date	1757338973	1757401832
git_commit_sha	`03d997e`	`0e9b724`
release_version	1.54.0-SNAPSHOT~03d997e2fd	1.54.0-SNAPSHOT~0e9b72482d

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1757403339	1757403339
ci_job_id	1119856392	1119856392
ci_pipeline_id	75916439	75916439
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-t5nwm10r 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-t5nwm10r 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 2 performance improvements and 1 performance regressions! Performance is the same for 9 metrics, 12 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:insecure-bank:profiling:high_load	better [-674.792µs; -381.436µs] or [-7.399%; -4.183%]	unstable [-38.707op/s; +100.207op/s] or [-7.606%; +19.692%]	8.592ms	539.625op/s	9.120ms	508.875op/s
scenario:load:petclinic:no_agent:high_load	worse [+1.058ms; +1.703ms] or [+2.881%; +4.637%]	unstable [-12.566op/s; +3.316op/s] or [-9.872%; +2.605%]	38.111ms	122.662op/s	36.730ms	127.287op/s
scenario:load:petclinic:iast:high_load	better [-1.885ms; -1.032ms] or [-4.056%; -2.221%]	unstable [-3.572op/s; +10.047op/s] or [-3.548%; +9.981%]	45.021ms	103.900op/s	46.480ms	100.662op/s

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~0e9b72482d, baseline=1.54.0-SNAPSHOT~03d997e2fd
    dateFormat X
    axisFormat %s
section baseline
no_agent (4.37 ms) : 4319, 4420
.   : milestone, 4370,
iast (9.461 ms) : 9305, 9616
.   : milestone, 9461,
iast_FULL (14.047 ms) : 13764, 14330
.   : milestone, 14047,
iast_GLOBAL (10.373 ms) : 10189, 10558
.   : milestone, 10373,
profiling (9.12 ms) : 8979, 9260
.   : milestone, 9120,
tracing (7.985 ms) : 7869, 8100
.   : milestone, 7985,
section candidate
no_agent (4.364 ms) : 4313, 4414
.   : milestone, 4364,
iast (9.301 ms) : 9147, 9455
.   : milestone, 9301,
iast_FULL (14.227 ms) : 13937, 14516
.   : milestone, 14227,
iast_GLOBAL (10.175 ms) : 9997, 10353
.   : milestone, 10175,
profiling (8.592 ms) : 8460, 8724
.   : milestone, 8592,
tracing (7.784 ms) : 7674, 7894
.   : milestone, 7784,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.37 ms [4.319 ms, 4.42 ms]	-
iast	9.461 ms [9.305 ms, 9.616 ms]	5.091 ms (116.5%)
iast_FULL	14.047 ms [13.764 ms, 14.33 ms]	9.678 ms (221.5%)
iast_GLOBAL	10.373 ms [10.189 ms, 10.558 ms]	6.004 ms (137.4%)
profiling	9.12 ms [8.979 ms, 9.26 ms]	4.75 ms (108.7%)
tracing	7.985 ms [7.869 ms, 8.1 ms]	3.615 ms (82.7%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.364 ms [4.313 ms, 4.414 ms]	-
iast	9.301 ms [9.147 ms, 9.455 ms]	4.937 ms (113.1%)
iast_FULL	14.227 ms [13.937 ms, 14.516 ms]	9.863 ms (226.0%)
iast_GLOBAL	10.175 ms [9.997 ms, 10.353 ms]	5.812 ms (133.2%)
profiling	8.592 ms [8.46 ms, 8.724 ms]	4.228 ms (96.9%)
tracing	7.784 ms [7.674 ms, 7.894 ms]	3.421 ms (78.4%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~0e9b72482d, baseline=1.54.0-SNAPSHOT~03d997e2fd
    dateFormat X
    axisFormat %s
section baseline
no_agent (36.73 ms) : 36439, 37021
.   : milestone, 36730,
appsec (49.243 ms) : 48807, 49679
.   : milestone, 49243,
code_origins (45.284 ms) : 44900, 45668
.   : milestone, 45284,
iast (46.48 ms) : 46066, 46893
.   : milestone, 46480,
profiling (49.41 ms) : 48957, 49863
.   : milestone, 49410,
tracing (43.371 ms) : 43016, 43727
.   : milestone, 43371,
section candidate
no_agent (38.111 ms) : 37803, 38419
.   : milestone, 38111,
appsec (48.055 ms) : 47620, 48490
.   : milestone, 48055,
code_origins (44.519 ms) : 44150, 44887
.   : milestone, 44519,
iast (45.021 ms) : 44643, 45399
.   : milestone, 45021,
profiling (48.627 ms) : 48168, 49087
.   : milestone, 48627,
tracing (43.753 ms) : 43380, 44125
.   : milestone, 43753,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	36.73 ms [36.439 ms, 37.021 ms]	-
appsec	49.243 ms [48.807 ms, 49.679 ms]	12.513 ms (34.1%)
code_origins	45.284 ms [44.9 ms, 45.668 ms]	8.554 ms (23.3%)
iast	46.48 ms [46.066 ms, 46.893 ms]	9.749 ms (26.5%)
profiling	49.41 ms [48.957 ms, 49.863 ms]	12.679 ms (34.5%)
tracing	43.371 ms [43.016 ms, 43.727 ms]	6.641 ms (18.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	38.111 ms [37.803 ms, 38.419 ms]	-
appsec	48.055 ms [47.62 ms, 48.49 ms]	9.944 ms (26.1%)
code_origins	44.519 ms [44.15 ms, 44.887 ms]	6.408 ms (16.8%)
iast	45.021 ms [44.643 ms, 45.399 ms]	6.91 ms (18.1%)
profiling	48.627 ms [48.168 ms, 49.087 ms]	10.516 ms (27.6%)
tracing	43.753 ms [43.38 ms, 44.125 ms]	5.642 ms (14.8%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	pawel.leszczynski/fix-openlineae-instrumentation
git_commit_date	1757338973	1757401832
git_commit_sha	`03d997e`	`0e9b724`
release_version	1.54.0-SNAPSHOT~03d997e2fd	1.54.0-SNAPSHOT~0e9b72482d

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1757403882	1757403882
ci_job_id	1119856393	1119856393
ci_pipeline_id	75916439	75916439
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-rwn4bt04 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-rwn4bt04 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 2 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~0e9b72482d, baseline=1.54.0-SNAPSHOT~03d997e2fd
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.476 ms) : 1465, 1488
.   : milestone, 1476,
appsec (3.672 ms) : 3454, 3889
.   : milestone, 3672,
iast (2.205 ms) : 2142, 2268
.   : milestone, 2205,
iast_GLOBAL (2.244 ms) : 2182, 2307
.   : milestone, 2244,
profiling (2.489 ms) : 2319, 2660
.   : milestone, 2489,
tracing (2.026 ms) : 1978, 2075
.   : milestone, 2026,
section candidate
no_agent (1.477 ms) : 1465, 1488
.   : milestone, 1477,
appsec (3.663 ms) : 3446, 3880
.   : milestone, 3663,
iast (2.205 ms) : 2142, 2268
.   : milestone, 2205,
iast_GLOBAL (2.248 ms) : 2185, 2311
.   : milestone, 2248,
profiling (2.493 ms) : 2325, 2660
.   : milestone, 2493,
tracing (2.019 ms) : 1970, 2068
.   : milestone, 2019,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.476 ms [1.465 ms, 1.488 ms]	-
appsec	3.672 ms [3.454 ms, 3.889 ms]	2.195 ms (148.7%)
iast	2.205 ms [2.142 ms, 2.268 ms]	728.884 µs (49.4%)
iast_GLOBAL	2.244 ms [2.182 ms, 2.307 ms]	768.216 µs (52.0%)
profiling	2.489 ms [2.319 ms, 2.66 ms]	1.013 ms (68.6%)
tracing	2.026 ms [1.978 ms, 2.075 ms]	550.258 µs (37.3%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.477 ms [1.465 ms, 1.488 ms]	-
appsec	3.663 ms [3.446 ms, 3.88 ms]	2.187 ms (148.1%)
iast	2.205 ms [2.142 ms, 2.268 ms]	728.325 µs (49.3%)
iast_GLOBAL	2.248 ms [2.185 ms, 2.311 ms]	771.921 µs (52.3%)
profiling	2.493 ms [2.325 ms, 2.66 ms]	1.016 ms (68.8%)
tracing	2.019 ms [1.97 ms, 2.068 ms]	542.608 µs (36.7%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~0e9b72482d, baseline=1.54.0-SNAPSHOT~03d997e2fd
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.588 s) : 15588000, 15588000
.   : milestone, 15588000,
appsec (14.9 s) : 14900000, 14900000
.   : milestone, 14900000,
iast (18.141 s) : 18141000, 18141000
.   : milestone, 18141000,
iast_GLOBAL (18.236 s) : 18236000, 18236000
.   : milestone, 18236000,
profiling (15.453 s) : 15453000, 15453000
.   : milestone, 15453000,
tracing (14.935 s) : 14935000, 14935000
.   : milestone, 14935000,
section candidate
no_agent (15.337 s) : 15337000, 15337000
.   : milestone, 15337000,
appsec (15.044 s) : 15044000, 15044000
.   : milestone, 15044000,
iast (18.525 s) : 18525000, 18525000
.   : milestone, 18525000,
iast_GLOBAL (18.263 s) : 18263000, 18263000
.   : milestone, 18263000,
profiling (15.984 s) : 15984000, 15984000
.   : milestone, 15984000,
tracing (15.183 s) : 15183000, 15183000
.   : milestone, 15183000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.588 s [15.588 s, 15.588 s]	-
appsec	14.9 s [14.9 s, 14.9 s]	-688.0 ms (-4.4%)
iast	18.141 s [18.141 s, 18.141 s]	2.553 s (16.4%)
iast_GLOBAL	18.236 s [18.236 s, 18.236 s]	2.648 s (17.0%)
profiling	15.453 s [15.453 s, 15.453 s]	-135.0 ms (-0.9%)
tracing	14.935 s [14.935 s, 14.935 s]	-653.0 ms (-4.2%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.337 s [15.337 s, 15.337 s]	-
appsec	15.044 s [15.044 s, 15.044 s]	-293.0 ms (-1.9%)
iast	18.525 s [18.525 s, 18.525 s]	3.188 s (20.8%)
iast_GLOBAL	18.263 s [18.263 s, 18.263 s]	2.926 s (19.1%)
profiling	15.984 s [15.984 s, 15.984 s]	647.0 ms (4.2%)
tracing	15.183 s [15.183 s, 15.183 s]	-154.0 ms (-1.0%)

Signed-off-by: Pawel Leszczynski <[email protected]>

github-actions · 2025-09-09T07:28:54Z

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

AlexeyKuznetsov-DD · 2025-09-09T17:20:00Z

...on/spark/src/main/java/datadog/trace/instrumentation/spark/AbstractDatadogSparkListener.java

+
+    String sparkAppName = conf.get("spark.app.name", null);
+    if (sparkAppName != null) {
+      log.info(


Should this one be log.debug() too?

AlexeyKuznetsov-DD

Is it possible to cover this change with some test?

raphaelgavache · 2025-09-10T00:50:56Z

...on/spark/src/main/java/datadog/trace/instrumentation/spark/AbstractDatadogSparkListener.java

@@ -180,7 +181,9 @@ public void setupOpenLineage(DDTraceId traceId) {
          "_dd.trace_id:"
              + traceId.toString()
              + ";_dd.ol_intake.emit_spans:false;_dd.ol_service:"
-              + sparkServiceName
+              + getServiceForOpenLineage(sparkConf, isRunningOnDatabricks)
+              + ";_dd.ol_intake.base_hash:"


thanks for adding that, but actually given that you support also container tags, it would be better to skip the base hash and add directly the process tags .
These are currently enabled for jvm 21 and will be enabled by default on next version

pawel-big-lebowski force-pushed the pawel.leszczynski/fix-openlineae-instrumentation branch from 7efaf57 to 07c5e27 Compare September 8, 2025 08:37

Fix tracer OpenLineage instrumentation

0e9b724

Signed-off-by: Pawel Leszczynski <[email protected]>

pawel-big-lebowski force-pushed the pawel.leszczynski/fix-openlineae-instrumentation branch from 07c5e27 to 0e9b724 Compare September 9, 2025 07:10

pawel-big-lebowski marked this pull request as ready for review September 9, 2025 07:28

pawel-big-lebowski requested a review from a team as a code owner September 9, 2025 07:28

pawel-big-lebowski added type: enhancement Enhancements and improvements tag: no release notes Changes to exclude from release notes inst: apache spark Apache Spark instrumentation labels Sep 9, 2025

AlexeyKuznetsov-DD reviewed Sep 9, 2025

View reviewed changes

raphaelgavache reviewed Sep 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix tracer OpenLineage instrumentation #9487

Fix tracer OpenLineage instrumentation #9487

pawel-big-lebowski commented Sep 8, 2025 •

edited

Loading

Uh oh!

datadog-official bot commented Sep 8, 2025 •

edited by datadog-datadog-prod-us1 bot

Loading

Uh oh!

pr-commenter bot commented Sep 8, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Sep 9, 2025

Uh oh!

AlexeyKuznetsov-DD Sep 9, 2025

Uh oh!

AlexeyKuznetsov-DD left a comment

Uh oh!

raphaelgavache Sep 10, 2025

Uh oh!

Uh oh!

Fix tracer OpenLineage instrumentation #9487

Are you sure you want to change the base?

Fix tracer OpenLineage instrumentation #9487

Conversation

pawel-big-lebowski commented Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

datadog-official bot commented Sep 8, 2025 • edited by datadog-datadog-prod-us1 bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pr-commenter bot commented Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

github-actions bot commented Sep 9, 2025

Uh oh!

AlexeyKuznetsov-DD Sep 9, 2025

Choose a reason for hiding this comment

Uh oh!

AlexeyKuznetsov-DD left a comment

Choose a reason for hiding this comment

Uh oh!

raphaelgavache Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pawel-big-lebowski commented Sep 8, 2025 •

edited

Loading

datadog-official bot commented Sep 8, 2025 •

edited by datadog-datadog-prod-us1 bot

Loading

pr-commenter bot commented Sep 8, 2025 •

edited

Loading