Refactor Severless Gateway Inferred Span #9388
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
PerfectSlayer
added
tag: do not merge
|
🎯 Code Coverage 🔗 Commit SHA: eb6874e | Docs | Was this helpful? Give us feedback! |
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 52 metrics, 7 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.54.0-SNAPSHOT~eb6874e7eb, baseline=1.54.0-SNAPSHOT~44e5822d88
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.062 s) : 0, 1061674
Total [baseline] (10.691 s) : 0, 10691132
Agent [candidate] (1.075 s) : 0, 1075168
Total [candidate] (10.71 s) : 0, 10709901
section appsec
Agent [baseline] (1.236 s) : 0, 1235980
Total [baseline] (10.997 s) : 0, 10997345
Agent [candidate] (1.243 s) : 0, 1243480
Total [candidate] (11.022 s) : 0, 11021617
section iast
Agent [baseline] (1.191 s) : 0, 1191345
Total [baseline] (11.054 s) : 0, 11053753
Agent [candidate] (1.202 s) : 0, 1202416
Total [candidate] (11.008 s) : 0, 11008252
section profiling
Agent [baseline] (1.209 s) : 0, 1209134
Total [baseline] (10.85 s) : 0, 10850495
Agent [candidate] (1.207 s) : 0, 1206648
Total [candidate] (10.837 s) : 0, 10837392
gantt
title petclinic - break down per module: candidate=1.54.0-SNAPSHOT~eb6874e7eb, baseline=1.54.0-SNAPSHOT~44e5822d88
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.452 ms) : 0, 1452
crashtracking [candidate] (1.463 ms) : 0, 1463
BytebuddyAgent [baseline] (732.299 ms) : 0, 732299
BytebuddyAgent [candidate] (740.731 ms) : 0, 740731
GlobalTracer [baseline] (252.881 ms) : 0, 252881
GlobalTracer [candidate] (256.812 ms) : 0, 256812
AppSec [baseline] (30.572 ms) : 0, 30572
AppSec [candidate] (31.282 ms) : 0, 31282
Debugger [baseline] (6.391 ms) : 0, 6391
Debugger [candidate] (6.51 ms) : 0, 6510
Remote Config [baseline] (692.229 µs) : 0, 692
Remote Config [candidate] (705.945 µs) : 0, 706
Telemetry [baseline] (16.343 ms) : 0, 16343
Telemetry [candidate] (16.39 ms) : 0, 16390
section appsec
crashtracking [baseline] (1.46 ms) : 0, 1460
crashtracking [candidate] (1.477 ms) : 0, 1477
BytebuddyAgent [baseline] (756.122 ms) : 0, 756122
BytebuddyAgent [candidate] (761.055 ms) : 0, 761055
GlobalTracer [baseline] (246.406 ms) : 0, 246406
GlobalTracer [candidate] (247.188 ms) : 0, 247188
IAST [baseline] (23.812 ms) : 0, 23812
IAST [candidate] (23.992 ms) : 0, 23992
AppSec [baseline] (171.71 ms) : 0, 171710
AppSec [candidate] (173.021 ms) : 0, 173021
Debugger [baseline] (6.054 ms) : 0, 6054
Debugger [candidate] (6.132 ms) : 0, 6132
Remote Config [baseline] (626.48 µs) : 0, 626
Remote Config [candidate] (649.601 µs) : 0, 650
Telemetry [baseline] (8.558 ms) : 0, 8558
Telemetry [candidate] (8.675 ms) : 0, 8675
section iast
crashtracking [baseline] (1.463 ms) : 0, 1463
crashtracking [candidate] (1.47 ms) : 0, 1470
BytebuddyAgent [baseline] (852.472 ms) : 0, 852472
BytebuddyAgent [candidate] (861.528 ms) : 0, 861528
GlobalTracer [baseline] (245.636 ms) : 0, 245636
GlobalTracer [candidate] (248.185 ms) : 0, 248185
IAST [baseline] (29.46 ms) : 0, 29460
IAST [candidate] (28.888 ms) : 0, 28888
AppSec [baseline] (26.589 ms) : 0, 26589
AppSec [candidate] (26.379 ms) : 0, 26379
Debugger [baseline] (6.093 ms) : 0, 6093
Debugger [candidate] (6.06 ms) : 0, 6060
Remote Config [baseline] (593.125 µs) : 0, 593
Remote Config [candidate] (611.488 µs) : 0, 611
Telemetry [baseline] (8.068 ms) : 0, 8068
Telemetry [candidate] (8.147 ms) : 0, 8147
section profiling
crashtracking [baseline] (1.436 ms) : 0, 1436
crashtracking [candidate] (1.432 ms) : 0, 1432
BytebuddyAgent [baseline] (761.994 ms) : 0, 761994
BytebuddyAgent [candidate] (760.106 ms) : 0, 760106
GlobalTracer [baseline] (232.824 ms) : 0, 232824
GlobalTracer [candidate] (232.195 ms) : 0, 232195
AppSec [baseline] (30.449 ms) : 0, 30449
AppSec [candidate] (30.244 ms) : 0, 30244
Debugger [baseline] (13.738 ms) : 0, 13738
Debugger [candidate] (12.067 ms) : 0, 12067
Remote Config [baseline] (748.048 µs) : 0, 748
Remote Config [candidate] (2.323 ms) : 0, 2323
Telemetry [baseline] (9.252 ms) : 0, 9252
Telemetry [candidate] (9.502 ms) : 0, 9502
ProfilingAgent [baseline] (107.518 ms) : 0, 107518
ProfilingAgent [candidate] (107.608 ms) : 0, 107608
Profiling [baseline] (108.162 ms) : 0, 108162
Profiling [candidate] (108.263 ms) : 0, 108263
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.54.0-SNAPSHOT~eb6874e7eb, baseline=1.54.0-SNAPSHOT~44e5822d88
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.063 s) : 0, 1063105
Total [baseline] (8.616 s) : 0, 8616108
Agent [candidate] (1.07 s) : 0, 1070325
Total [candidate] (8.633 s) : 0, 8633354
section iast
Agent [baseline] (1.197 s) : 0, 1196694
Total [baseline] (9.324 s) : 0, 9323696
Agent [candidate] (1.191 s) : 0, 1190544
Total [candidate] (9.281 s) : 0, 9280566
gantt
title insecure-bank - break down per module: candidate=1.54.0-SNAPSHOT~eb6874e7eb, baseline=1.54.0-SNAPSHOT~44e5822d88
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.46 ms) : 0, 1460
crashtracking [candidate] (1.466 ms) : 0, 1466
BytebuddyAgent [baseline] (734.072 ms) : 0, 734072
BytebuddyAgent [candidate] (738.765 ms) : 0, 738765
GlobalTracer [baseline] (252.551 ms) : 0, 252551
GlobalTracer [candidate] (254.513 ms) : 0, 254513
AppSec [baseline] (30.741 ms) : 0, 30741
AppSec [candidate] (30.822 ms) : 0, 30822
Debugger [baseline] (6.424 ms) : 0, 6424
Debugger [candidate] (6.505 ms) : 0, 6505
Remote Config [baseline] (701.541 µs) : 0, 702
Remote Config [candidate] (701.093 µs) : 0, 701
Telemetry [baseline] (16.153 ms) : 0, 16153
Telemetry [candidate] (16.43 ms) : 0, 16430
section iast
crashtracking [baseline] (1.475 ms) : 0, 1475
crashtracking [candidate] (1.452 ms) : 0, 1452
BytebuddyAgent [baseline] (858.81 ms) : 0, 858810
BytebuddyAgent [candidate] (852.338 ms) : 0, 852338
GlobalTracer [baseline] (248.312 ms) : 0, 248312
GlobalTracer [candidate] (248.441 ms) : 0, 248441
AppSec [baseline] (24.559 ms) : 0, 24559
AppSec [candidate] (25.539 ms) : 0, 25539
Debugger [baseline] (6.006 ms) : 0, 6006
Debugger [candidate] (5.996 ms) : 0, 5996
Remote Config [baseline] (597.051 µs) : 0, 597
Remote Config [candidate] (601.057 µs) : 0, 601
Telemetry [baseline] (8.161 ms) : 0, 8161
Telemetry [candidate] (8.112 ms) : 0, 8112
IAST [baseline] (27.804 ms) : 0, 27804
IAST [candidate] (27.065 ms) : 0, 27065
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 5 performance regressions! Performance is the same for 7 metrics, 12 unstable metrics.
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~eb6874e7eb, baseline=1.54.0-SNAPSHOT~44e5822d88
dateFormat X
axisFormat %s
section baseline
no_agent (36.203 ms) : 35907, 36498
. : milestone, 36203,
appsec (46.883 ms) : 46479, 47287
. : milestone, 46883,
code_origins (45.177 ms) : 44775, 45579
. : milestone, 45177,
iast (44.319 ms) : 43938, 44700
. : milestone, 44319,
profiling (46.544 ms) : 46083, 47005
. : milestone, 46544,
tracing (45.223 ms) : 44848, 45598
. : milestone, 45223,
section candidate
no_agent (36.505 ms) : 36212, 36799
. : milestone, 36505,
appsec (49.693 ms) : 49246, 50140
. : milestone, 49693,
code_origins (44.028 ms) : 43658, 44398
. : milestone, 44028,
iast (44.19 ms) : 43818, 44563
. : milestone, 44190,
profiling (48.464 ms) : 48014, 48915
. : milestone, 48464,
tracing (44.109 ms) : 43720, 44499
. : milestone, 44109,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~eb6874e7eb, baseline=1.54.0-SNAPSHOT~44e5822d88
dateFormat X
axisFormat %s
section baseline
no_agent (4.204 ms) : 4156, 4252
. : milestone, 4204,
iast (9.428 ms) : 9270, 9585
. : milestone, 9428,
iast_FULL (13.841 ms) : 13565, 14116
. : milestone, 13841,
iast_GLOBAL (9.991 ms) : 9816, 10166
. : milestone, 9991,
profiling (8.929 ms) : 8780, 9077
. : milestone, 8929,
tracing (7.729 ms) : 7619, 7838
. : milestone, 7729,
section candidate
no_agent (4.31 ms) : 4263, 4358
. : milestone, 4310,
iast (9.361 ms) : 9206, 9516
. : milestone, 9361,
iast_FULL (14.434 ms) : 14145, 14723
. : milestone, 14434,
iast_GLOBAL (10.616 ms) : 10427, 10805
. : milestone, 10616,
profiling (9.355 ms) : 9211, 9498
. : milestone, 9355,
tracing (7.777 ms) : 7666, 7887
. : milestone, 7777,
DacapoParameters
See matching parameters
SummaryFound 1 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 0 unstable metrics.
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~eb6874e7eb, baseline=1.54.0-SNAPSHOT~44e5822d88
dateFormat X
axisFormat %s
section baseline
no_agent (1.474 ms) : 1463, 1486
. : milestone, 1474,
appsec (3.701 ms) : 3484, 3918
. : milestone, 3701,
iast (2.2 ms) : 2138, 2263
. : milestone, 2200,
iast_GLOBAL (2.238 ms) : 2174, 2301
. : milestone, 2238,
profiling (2.038 ms) : 1988, 2089
. : milestone, 2038,
tracing (2.025 ms) : 1975, 2074
. : milestone, 2025,
section candidate
no_agent (1.476 ms) : 1464, 1487
. : milestone, 1476,
appsec (2.498 ms) : 2445, 2550
. : milestone, 2498,
iast (2.189 ms) : 2126, 2252
. : milestone, 2189,
iast_GLOBAL (2.239 ms) : 2176, 2303
. : milestone, 2239,
profiling (2.034 ms) : 1984, 2085
. : milestone, 2034,
tracing (2.026 ms) : 1977, 2076
. : milestone, 2026,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~eb6874e7eb, baseline=1.54.0-SNAPSHOT~44e5822d88
dateFormat X
axisFormat %s
section baseline
no_agent (14.903 s) : 14903000, 14903000
. : milestone, 14903000,
appsec (15.004 s) : 15004000, 15004000
. : milestone, 15004000,
iast (18.864 s) : 18864000, 18864000
. : milestone, 18864000,
iast_GLOBAL (18.013 s) : 18013000, 18013000
. : milestone, 18013000,
profiling (15.37 s) : 15370000, 15370000
. : milestone, 15370000,
tracing (14.924 s) : 14924000, 14924000
. : milestone, 14924000,
section candidate
no_agent (14.852 s) : 14852000, 14852000
. : milestone, 14852000,
appsec (14.931 s) : 14931000, 14931000
. : milestone, 14931000,
iast (18.424 s) : 18424000, 18424000
. : milestone, 18424000,
iast_GLOBAL (17.956 s) : 17956000, 17956000
. : milestone, 17956000,
profiling (15.959 s) : 15959000, 15959000
. : milestone, 15959000,
tracing (15.105 s) : 15105000, 15105000
. : milestone, 15105000,
|
PerfectSlayer
force-pushed
the
bbujon/gateway-inferred-span2
branch
2 times, most recently
from
ad6356f to
e8c2baf
Compare
PerfectSlayer
requested review from
AlexeyKuznetsov-DD
and removed request for
a team
Comment on lines
+65
to
+70
| long startTime; | ||
| try { | ||
| startTime = Long.parseLong(header(PROXY_START_TIME_MS)) * 1000; // Convert to microseconds | ||
| } catch (NumberFormatException e) { | ||
| return extracted; // Invalid timestamp | ||
| } |
There was a problem hiding this comment.
I have a feeling that we need to have some sort of utility code for such places?
WDYT?
Something like: long startTime = parseLong((header(PROXY_START_TIME_MS), ?some_default? ) * 1000;
There was a problem hiding this comment.
I don't think you can have a default start time in this case.
Gateway inferred spans represent the duration spent in upstream proxy servers (that are not instrumented for distributed tracing, they only inject few Datadog headers).
The start time represents the timestamp the request was intercepted by the proxy server and will be use to create a "proxy span" into the downstream service.
So if you don't have a valid start time, it's most likely a proxy issue, and you can't really "guess" when the proxy intercepted the upstream request.
amarziali
reviewed
Sep 3, 2025
| tracer().getDataStreamsMonitoring().setCheckpoint(span, forHttpServer()); | ||
| } | ||
| // DSM Checkpoint | ||
| tracer().getDataStreamsMonitoring().setCheckpoint(span, forHttpServer()); |
There was a problem hiding this comment.
Is this change related to inferred span?
There was a problem hiding this comment.
No, carrier and getter aren’t used so it’s only an optimization of not checking them. And disabled DSM will call a noop implementation.
| String proxy = SUPPORTED_PROXIES.get(proxySystem); | ||
| AgentSpan span = AgentTracer.get().startSpan(INSTRUMENTATION_NAME, proxy, extracted, startTime); | ||
|
|
||
| span.setTag(COMPONENT, proxySystem); |
There was a problem hiding this comment.
should we have a decorator setting all those properties?
There was a problem hiding this comment.
What kind of decorator do you have in mind?
InferredProxySpan are only created from the HttpServerDecorator and are "product spans" directly created from the serverless product.
PerfectSlayer
force-pushed
the
bbujon/gateway-inferred-span2
branch
from
e8c2baf to
e73fcc3
Compare
* Creates inferred proxy spans for API Gateway calls via presence of http headers --------- Co-authored-by: Zarir Hamza <[email protected]>
Avoid duplicate expensive context extraction Avoid subclassing tracing span for serverless but used serverless context element instead to store / track inferred span while keep tracing feature untouched Improved propagator to not create / capture inferred span context element on invalid data Rework context element to hold the inferred spans and its captured data Release captured data as soon as they span start (never read after this point so reclaiming memory) Refactor context element and propagator into the right package, not context component (product / feature agnostic) Refactor unit tests
PerfectSlayer
force-pushed
the
bbujon/gateway-inferred-span2
branch
from
e73fcc3 to
eb6874e
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
This PR continues @zarirhamza work about Serverless gateway inferred span in #8336 and brings the following improvements:
Motivation
Restore Serverless feature while meeting design and performance requirements
Additional Notes
This requires to complete the instrumentation refactoring first.
I already did a lot of them here:
The last one is pending for review:
beforeFinishMigration #9422Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: LANGPLAT-680