LLM Obs SDK Mapper #8372
Merged
LLM Obs SDK Mapper #8372
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![datadog-datadog-prod-us1[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
dd-trace-core/src/main/java/datadog/trace/llmobs/writer/ddintake/LLMObsSpanMapper.java
Outdated
Show resolved
Hide resolved
dd-trace-core/src/main/java/datadog/trace/llmobs/writer/ddintake/LLMObsSpanMapper.java
Outdated
Show resolved
Hide resolved
gary-huang
force-pushed
the
gary/impl-llm-obs-api
branch
2 times, most recently
from
680ec94 to
b1d7aeb
Compare
gary-huang
force-pushed
the
gary/add-llm-obs-writer
branch
from
113bf8c to
9052dc4
Compare
BenchmarksStartupParameters
See matching parameters
SummaryFound 1 performance improvements and 7 performance regressions! Performance is the same for 45 metrics, 18 unstable metrics.
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.50.0-SNAPSHOT~75d85a24a4, baseline=1.51.0-SNAPSHOT~2ee59a972d
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.001 s) : 0, 1000684
Total [baseline] (8.581 s) : 0, 8580781
Agent [candidate] (1.041 s) : 0, 1041164
Total [candidate] (8.627 s) : 0, 8627429
section iast
Agent [baseline] (1.132 s) : 0, 1132474
Total [baseline] (9.296 s) : 0, 9295531
Agent [candidate] (1.152 s) : 0, 1152031
Total [candidate] (9.232 s) : 0, 9232244
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.137 s) : 0, 1136589
Total [baseline] (9.243 s) : 0, 9243239
Agent [candidate] (1.152 s) : 0, 1152177
Total [candidate] (9.188 s) : 0, 9188250
section iast_TELEMETRY_OFF
Agent [baseline] (1.132 s) : 0, 1131586
Total [baseline] (9.243 s) : 0, 9243302
Agent [candidate] (1.153 s) : 0, 1153211
Total [candidate] (9.237 s) : 0, 9236572
gantt
title insecure-bank - break down per module: candidate=1.50.0-SNAPSHOT~75d85a24a4, baseline=1.51.0-SNAPSHOT~2ee59a972d
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (692.578 ms) : 0, 692578
BytebuddyAgent [candidate] (695.179 ms) : 0, 695179
GlobalTracer [baseline] (242.096 ms) : 0, 242096
GlobalTracer [candidate] (245.642 ms) : 0, 245642
AppSec [baseline] (30.206 ms) : 0, 30206
AppSec [candidate] (57.728 ms) : 0, 57728
Debugger [baseline] (6.017 ms) : 0, 6017
Debugger [candidate] (6.248 ms) : 0, 6248
Remote Config [baseline] (670.467 µs) : 0, 670
Remote Config [candidate] (742.984 µs) : 0, 743
Telemetry [baseline] (8.205 ms) : 0, 8205
Telemetry [candidate] (12.015 ms) : 0, 12015
section iast
BytebuddyAgent [baseline] (807.084 ms) : 0, 807084
BytebuddyAgent [candidate] (804.265 ms) : 0, 804265
GlobalTracer [baseline] (232.771 ms) : 0, 232771
GlobalTracer [candidate] (231.301 ms) : 0, 231301
IAST [baseline] (25.264 ms) : 0, 25264
IAST [candidate] (28.479 ms) : 0, 28479
AppSec [baseline] (31.281 ms) : 0, 31281
AppSec [candidate] (50.152 ms) : 0, 50152
Debugger [baseline] (6.711 ms) : 0, 6711
Debugger [candidate] (5.913 ms) : 0, 5913
Remote Config [baseline] (594.193 µs) : 0, 594
Remote Config [candidate] (590.854 µs) : 0, 591
Telemetry [baseline] (7.938 ms) : 0, 7938
Telemetry [candidate] (7.892 ms) : 0, 7892
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (809.88 ms) : 0, 809880
BytebuddyAgent [candidate] (803.06 ms) : 0, 803060
GlobalTracer [baseline] (233.977 ms) : 0, 233977
GlobalTracer [candidate] (232.313 ms) : 0, 232313
IAST [baseline] (27.183 ms) : 0, 27183
IAST [candidate] (26.056 ms) : 0, 26056
AppSec [baseline] (29.44 ms) : 0, 29440
AppSec [candidate] (52.904 ms) : 0, 52904
Debugger [baseline] (6.677 ms) : 0, 6677
Debugger [candidate] (5.904 ms) : 0, 5904
Remote Config [baseline] (575.641 µs) : 0, 576
Remote Config [candidate] (583.864 µs) : 0, 584
Telemetry [baseline] (8.042 ms) : 0, 8042
Telemetry [candidate] (7.869 ms) : 0, 7869
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (806.473 ms) : 0, 806473
BytebuddyAgent [candidate] (805.936 ms) : 0, 805936
GlobalTracer [baseline] (233.495 ms) : 0, 233495
GlobalTracer [candidate] (231.129 ms) : 0, 231129
IAST [baseline] (27.99 ms) : 0, 27990
IAST [candidate] (28.659 ms) : 0, 28659
AppSec [baseline] (28.414 ms) : 0, 28414
AppSec [candidate] (49.485 ms) : 0, 49485
Debugger [baseline] (5.853 ms) : 0, 5853
Debugger [candidate] (6.008 ms) : 0, 6008
Remote Config [baseline] (579.127 µs) : 0, 579
Remote Config [candidate] (607.292 µs) : 0, 607
Telemetry [baseline] (7.904 ms) : 0, 7904
Telemetry [candidate] (7.847 ms) : 0, 7847
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.50.0-SNAPSHOT~75d85a24a4, baseline=1.51.0-SNAPSHOT~2ee59a972d
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (994.874 ms) : 0, 994874
Total [baseline] (10.627 s) : 0, 10626508
Agent [candidate] (1.029 s) : 0, 1028609
Total [candidate] (10.683 s) : 0, 10682883
section appsec
Agent [baseline] (1.164 s) : 0, 1164353
Total [baseline] (10.751 s) : 0, 10751042
Agent [candidate] (1.173 s) : 0, 1173072
Total [candidate] (10.786 s) : 0, 10785737
section iast
Agent [baseline] (1.135 s) : 0, 1135337
Total [baseline] (10.816 s) : 0, 10815689
Agent [candidate] (1.152 s) : 0, 1151875
Total [candidate] (10.889 s) : 0, 10888694
section profiling
Agent [baseline] (1.245 s) : 0, 1244982
Total [baseline] (10.966 s) : 0, 10966147
Agent [candidate] (1.269 s) : 0, 1269296
Total [candidate] (10.921 s) : 0, 10921137
gantt
title petclinic - break down per module: candidate=1.50.0-SNAPSHOT~75d85a24a4, baseline=1.51.0-SNAPSHOT~2ee59a972d
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (686.662 ms) : 0, 686662
BytebuddyAgent [candidate] (684.584 ms) : 0, 684584
GlobalTracer [baseline] (242.062 ms) : 0, 242062
GlobalTracer [candidate] (243.222 ms) : 0, 243222
AppSec [baseline] (30.452 ms) : 0, 30452
AppSec [candidate] (57.236 ms) : 0, 57236
Debugger [baseline] (6.047 ms) : 0, 6047
Debugger [candidate] (6.239 ms) : 0, 6239
Remote Config [baseline] (684.075 µs) : 0, 684
Remote Config [candidate] (754.062 µs) : 0, 754
Telemetry [baseline] (8.18 ms) : 0, 8180
Telemetry [candidate] (12.963 ms) : 0, 12963
section appsec
BytebuddyAgent [baseline] (705.901 ms) : 0, 705901
BytebuddyAgent [candidate] (707.265 ms) : 0, 707265
GlobalTracer [baseline] (239.273 ms) : 0, 239273
GlobalTracer [candidate] (239.962 ms) : 0, 239962
IAST [baseline] (23.174 ms) : 0, 23174
IAST [candidate] (21.919 ms) : 0, 21919
AppSec [baseline] (170.335 ms) : 0, 170335
AppSec [candidate] (177.244 ms) : 0, 177244
Debugger [baseline] (7.305 ms) : 0, 7305
Debugger [candidate] (5.978 ms) : 0, 5978
Remote Config [baseline] (611.885 µs) : 0, 612
Remote Config [candidate] (648.664 µs) : 0, 649
Telemetry [baseline] (8.464 ms) : 0, 8464
Telemetry [candidate] (7.374 ms) : 0, 7374
section iast
BytebuddyAgent [baseline] (809.323 ms) : 0, 809323
BytebuddyAgent [candidate] (803.393 ms) : 0, 803393
GlobalTracer [baseline] (233.45 ms) : 0, 233450
GlobalTracer [candidate] (231.382 ms) : 0, 231382
IAST [baseline] (28.571 ms) : 0, 28571
IAST [candidate] (29.487 ms) : 0, 29487
AppSec [baseline] (28.725 ms) : 0, 28725
AppSec [candidate] (49.659 ms) : 0, 49659
Debugger [baseline] (5.82 ms) : 0, 5820
Debugger [candidate] (5.9 ms) : 0, 5900
Remote Config [baseline] (581.454 µs) : 0, 581
Remote Config [candidate] (596.172 µs) : 0, 596
Telemetry [baseline] (7.993 ms) : 0, 7993
Telemetry [candidate] (7.956 ms) : 0, 7956
section profiling
BytebuddyAgent [baseline] (678.331 ms) : 0, 678331
BytebuddyAgent [candidate] (676.196 ms) : 0, 676196
GlobalTracer [baseline] (362.017 ms) : 0, 362017
GlobalTracer [candidate] (361.013 ms) : 0, 361013
AppSec [baseline] (31.634 ms) : 0, 31634
AppSec [candidate] (61.526 ms) : 0, 61526
Debugger [baseline] (12.049 ms) : 0, 12049
Debugger [candidate] (6.06 ms) : 0, 6060
Remote Config [baseline] (660.085 µs) : 0, 660
Remote Config [candidate] (653.859 µs) : 0, 654
Telemetry [baseline] (8.763 ms) : 0, 8763
Telemetry [candidate] (8.137 ms) : 0, 8137
ProfilingAgent [baseline] (102.937 ms) : 0, 102937
ProfilingAgent [candidate] (104.845 ms) : 0, 104845
Profiling [baseline] (102.96 ms) : 0, 102960
Profiling [candidate] (104.869 ms) : 0, 104869
LoadParameters
See matching parameters
SummaryFound 5 performance improvements and 1 performance regressions! Performance is the same for 6 metrics, 16 unstable metrics.
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics. Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~75d85a24a4, baseline=1.51.0-SNAPSHOT~2ee59a972d
dateFormat X
axisFormat %s
section baseline
no_agent (15.545 s) : 15545000, 15545000
. : milestone, 15545000,
appsec (14.899 s) : 14899000, 14899000
. : milestone, 14899000,
iast (18.704 s) : 18704000, 18704000
. : milestone, 18704000,
iast_GLOBAL (17.93 s) : 17930000, 17930000
. : milestone, 17930000,
profiling (15.782 s) : 15782000, 15782000
. : milestone, 15782000,
tracing (15.077 s) : 15077000, 15077000
. : milestone, 15077000,
section candidate
no_agent (15.26 s) : 15260000, 15260000
. : milestone, 15260000,
appsec (14.69 s) : 14690000, 14690000
. : milestone, 14690000,
iast (18.25 s) : 18250000, 18250000
. : milestone, 18250000,
iast_GLOBAL (17.84 s) : 17840000, 17840000
. : milestone, 17840000,
profiling (15.189 s) : 15189000, 15189000
. : milestone, 15189000,
tracing (14.95 s) : 14950000, 14950000
. : milestone, 14950000,
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~75d85a24a4, baseline=1.51.0-SNAPSHOT~2ee59a972d
dateFormat X
axisFormat %s
section baseline
no_agent (1.474 ms) : 1462, 1485
. : milestone, 1474,
appsec (2.419 ms) : 2370, 2469
. : milestone, 2419,
iast (2.199 ms) : 2137, 2262
. : milestone, 2199,
iast_GLOBAL (2.233 ms) : 2171, 2295
. : milestone, 2233,
profiling (2.048 ms) : 1998, 2098
. : milestone, 2048,
tracing (2.003 ms) : 1955, 2051
. : milestone, 2003,
section candidate
no_agent (1.477 ms) : 1466, 1489
. : milestone, 1477,
appsec (2.408 ms) : 2359, 2457
. : milestone, 2408,
iast (2.188 ms) : 2127, 2249
. : milestone, 2188,
iast_GLOBAL (2.229 ms) : 2167, 2291
. : milestone, 2229,
profiling (2.039 ms) : 1989, 2089
. : milestone, 2039,
tracing (2.013 ms) : 1965, 2061
. : milestone, 2013,
|
gary-huang
force-pushed
the
gary/impl-llm-obs-api
branch
from
b1d7aeb to
3bcbe81
Compare
gary-huang
force-pushed
the
gary/add-llm-obs-writer
branch
from
9052dc4 to
dd8c1ed
Compare
gary-huang
force-pushed
the
gary/impl-llm-obs-api
branch
from
3bcbe81 to
c437b9c
Compare
gary-huang
force-pushed
the
gary/add-llm-obs-writer
branch
2 times, most recently
from
88cbf86 to
ce8a6d3
Compare
gary-huang
force-pushed
the
gary/impl-llm-obs-api
branch
from
c437b9c to
8247b74
Compare
gary-huang
force-pushed
the
gary/add-llm-obs-writer
branch
2 times, most recently
from
fc30660 to
061b18d
Compare
gary-huang
force-pushed
the
gary/impl-llm-obs-api
branch
from
8247b74 to
871d358
Compare
gary-huang
force-pushed
the
gary/add-llm-obs-writer
branch
2 times, most recently
from
5c305b9 to
9ee77d9
Compare
dd-trace-core/src/main/java/datadog/trace/llmobs/writer/ddintake/LLMObsSpanMapper.java
Outdated
Show resolved
Hide resolved
gary-huang
force-pushed
the
gary/impl-llm-obs-api
branch
6 times, most recently
from
94136c5 to
7064f57
Compare
gary-huang
force-pushed
the
gary/add-llm-obs-writer
branch
from
9ee77d9 to
0b4f8a8
Compare
gary-huang
force-pushed
the
gary/impl-llm-obs-api
branch
from
7064f57 to
9cb3e0a
Compare
gary-huang
force-pushed
the
gary/add-llm-obs-writer
branch
from
0b4f8a8 to
738eb77
Compare
dd-trace-core/src/main/java/datadog/trace/llmobs/writer/ddintake/LLMObsSpanMapper.java
Outdated
Show resolved
Hide resolved
gary-huang
force-pushed
the
gary/impl-llm-obs-api
branch
3 times, most recently
from
f9cb23e to
64c94e5
Compare
gary-huang
force-pushed
the
gary/add-llm-obs-writer
branch
from
f858e30 to
fb0af52
Compare
gary-huang
force-pushed
the
gary/add-llm-obs-writer
branch
from
fb0af52 to
e0b4510
Compare
nayeem-kamal
approved these changes
Jul 2, 2025
nayeem-kamal
requested review from
daniel-mohedano and
mcculls
and removed request for
mcculls and
daniel-mohedano
gary-huang
added a commit
that referenced
this pull request
Jul 9, 2025
* add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * impl llmobs agent and llmobs apis * impl llmobs agent * working writer * add support for llm message and tool calls * cleaned up whitespace * resolve merge conflicts * remaining merge conflicts * fix bad method call * fixed llmobs intake creation if llmobs not enabled * removed print statements * added tests for llmobsspanmapper * fixed coverage for tags --------- Co-authored-by: Nayeem Kamal <[email protected]>
nayeem-kamal
added a commit
that referenced
this pull request
Jul 9, 2025
* add APIs for llm obs sdk (#8135) * add APIs for llm obs * add llm message class to support llm spans * follow java convention of naming Id instead of ID * add codeowners * implement LLM Obs SDK spans APIs (#8390) * add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * spotless * add APIs for llm obs sdk (#8135) * add APIs for llm obs * add llm message class to support llm spans * follow java convention of naming Id instead of ID * add codeowners * rename ID to Id according to java naming conventions * Undo change to integrations-core submodule * fix build gradle * rm empty line * fix test * LLM Obs SDK Mapper (#8372) * add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * impl llmobs agent and llmobs apis * impl llmobs agent * working writer * add support for llm message and tool calls * cleaned up whitespace * resolve merge conflicts * remaining merge conflicts * fix bad method call * fixed llmobs intake creation if llmobs not enabled * removed print statements * added tests for llmobsspanmapper * fixed coverage for tags --------- Co-authored-by: Nayeem Kamal <[email protected]> * updated to master submodule * LLM Obs SDK use context API for parent children span linkage (#8711) * add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * impl llmobs agent and llmobs apis * impl llmobs agent * working writer * add support for llm message and tool calls * impl llmobs agent and llmobs apis * use new ctx api to track parent span * cleaned up whitespace * resolve merge conflicts * remaining merge conflicts * fix bad method call * fixed llmobs intake creation if llmobs not enabled * removed print statements * ran spotless * added tests for llmobsspanmapper * fixed coverage for tags --------- Co-authored-by: Nayeem Kamal <[email protected]> Co-authored-by: Nayeem Kamal <[email protected]> * LLM Obs SDK evaluation metrics submission (#8688) * add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * impl llmobs agent and llmobs apis * impl llmobs agent * working writer * add support for llm message and tool calls * impl llmobs agent and llmobs apis * use new ctx api to track parent span * add api for evals * working impl supporting both agentless and agent * handle null tags and default to default ml app if null or empty string provided in the override * cleaned up whitespace * resolve merge conflicts * remaining merge conflicts * fix bad method call * fixed llmobs intake creation if llmobs not enabled * removed print statements * ran spotless * ran spotless * added tests for llmobsspanmapper * fixed coverage for tags --------- Co-authored-by: Nayeem Kamal <[email protected]> Co-authored-by: Nayeem Kamal <[email protected]> * fix CODEOWNERS --------- Co-authored-by: Nayeem Kamal <[email protected]> Co-authored-by: Nayeem Kamal <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
Adds the remote mapper and payload definitions for LLMObs spans as well as adding the remote intake api setup for the LLMObs backend endpoint.
Motivation
Support manual sending of LLMObs spans from within the tracer.
Additional Notes
Additional tests will follow in the feature branch.
Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]