Implement impacted tests detection #8188

nikita-tkachenko-datadog · 2025-01-13T11:29:10Z

What Does This Do

Implements Impacted Tests Detection CI Visibility feature (milestones 1.X as defined in the RFC linked below):

detects which test cases are "impacted" (for now a test is impacted if the body of the test method was changed in a PR)
marks impacted tests with test.is_modified:true tag
if Early Flakiness Detection feature is enabled, applies it to every impacted test

Additional Details

https://docs.google.com/document/d/1oV3qRzfdEAOazjjzsA7z-yAFh8gPOMRQnf7EtIwhsKo/edit?tab=t.0#heading=h.rnd972k0hiye

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: SDTEST-1389

pr-commenter · 2025-01-13T12:00:37Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/impacted-tests
git_commit_date	1737548458	1737549292
git_commit_sha	`523df01`	`9b370b2`
release_version	1.46.0-SNAPSHOT~523df01c66	1.46.0-SNAPSHOT~9b370b2dd7

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1737551637	1737551637
ci_job_id	775687407	775687407
ci_pipeline_id	53672611	53672611
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 60 metrics, 3 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.46.0-SNAPSHOT~9b370b2dd7, baseline=1.46.0-SNAPSHOT~523df01c66

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.051 s) : 0, 1050908
Total [baseline] (10.436 s) : 0, 10436439
Agent [candidate] (1.056 s) : 0, 1056033
Total [candidate] (10.438 s) : 0, 10437507
section appsec
Agent [baseline] (1.19 s) : 0, 1189718
Total [baseline] (10.752 s) : 0, 10752193
Agent [candidate] (1.193 s) : 0, 1192607
Total [candidate] (10.686 s) : 0, 10685910
section iast
Agent [baseline] (1.183 s) : 0, 1183328
Total [baseline] (10.951 s) : 0, 10950940
Agent [candidate] (1.182 s) : 0, 1182325
Total [candidate] (10.998 s) : 0, 10998129
section profiling
Agent [baseline] (1.256 s) : 0, 1256118
Total [baseline] (10.856 s) : 0, 10855851
Agent [candidate] (1.261 s) : 0, 1260653
Total [candidate] (10.785 s) : 0, 10785254

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.051 s	-
Agent	appsec	1.19 s	138.81 ms (13.2%)
Agent	iast	1.183 s	132.42 ms (12.6%)
Agent	profiling	1.256 s	205.21 ms (19.5%)
Total	tracing	10.436 s	-
Total	appsec	10.752 s	315.754 ms (3.0%)
Total	iast	10.951 s	514.501 ms (4.9%)
Total	profiling	10.856 s	419.412 ms (4.0%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.056 s	-
Agent	appsec	1.193 s	136.574 ms (12.9%)
Agent	iast	1.182 s	126.292 ms (12.0%)
Agent	profiling	1.261 s	204.62 ms (19.4%)
Total	tracing	10.438 s	-
Total	appsec	10.686 s	248.403 ms (2.4%)
Total	iast	10.998 s	560.622 ms (5.4%)
Total	profiling	10.785 s	347.748 ms (3.3%)

gantt
    title petclinic - break down per module: candidate=1.46.0-SNAPSHOT~9b370b2dd7, baseline=1.46.0-SNAPSHOT~523df01c66

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (711.966 ms) : 0, 711966
BytebuddyAgent [candidate] (714.835 ms) : 0, 714835
GlobalTracer [baseline] (254.477 ms) : 0, 254477
GlobalTracer [candidate] (255.388 ms) : 0, 255388
AppSec [baseline] (55.381 ms) : 0, 55381
AppSec [candidate] (56.443 ms) : 0, 56443
Remote Config [baseline] (726.826 µs) : 0, 727
Remote Config [candidate] (731.648 µs) : 0, 732
Telemetry [baseline] (13.408 ms) : 0, 13408
Telemetry [candidate] (13.669 ms) : 0, 13669
section appsec
BytebuddyAgent [baseline] (732.333 ms) : 0, 732333
BytebuddyAgent [candidate] (733.578 ms) : 0, 733578
GlobalTracer [baseline] (252.535 ms) : 0, 252535
GlobalTracer [candidate] (254.046 ms) : 0, 254046
AppSec [baseline] (171.365 ms) : 0, 171365
AppSec [candidate] (171.01 ms) : 0, 171010
Remote Config [baseline] (669.746 µs) : 0, 670
Remote Config [candidate] (667.625 µs) : 0, 668
Telemetry [baseline] (8.161 ms) : 0, 8161
Telemetry [candidate] (8.58 ms) : 0, 8580
IAST [baseline] (19.336 ms) : 0, 19336
IAST [candidate] (19.427 ms) : 0, 19427
section iast
BytebuddyAgent [baseline] (832.067 ms) : 0, 832067
BytebuddyAgent [candidate] (831.134 ms) : 0, 831134
GlobalTracer [baseline] (246.926 ms) : 0, 246926
GlobalTracer [candidate] (246.917 ms) : 0, 246917
AppSec [baseline] (58.171 ms) : 0, 58171
AppSec [candidate] (58.165 ms) : 0, 58165
Remote Config [baseline] (669.787 µs) : 0, 670
Remote Config [candidate] (675.185 µs) : 0, 675
Telemetry [baseline] (8.826 ms) : 0, 8826
Telemetry [candidate] (8.782 ms) : 0, 8782
IAST [baseline] (21.714 ms) : 0, 21714
IAST [candidate] (21.633 ms) : 0, 21633
section profiling
BytebuddyAgent [baseline] (704.843 ms) : 0, 704843
BytebuddyAgent [candidate] (709.142 ms) : 0, 709142
GlobalTracer [baseline] (349.581 ms) : 0, 349581
GlobalTracer [candidate] (350.43 ms) : 0, 350430
AppSec [baseline] (54.832 ms) : 0, 54832
AppSec [candidate] (54.044 ms) : 0, 54044
Remote Config [baseline] (709.156 µs) : 0, 709
Remote Config [candidate] (723.74 µs) : 0, 724
Telemetry [baseline] (8.777 ms) : 0, 8777
Telemetry [candidate] (8.677 ms) : 0, 8677
ProfilingAgent [baseline] (95.296 ms) : 0, 95296
ProfilingAgent [candidate] (95.398 ms) : 0, 95398
Profiling [baseline] (95.321 ms) : 0, 95321
Profiling [candidate] (95.423 ms) : 0, 95423

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.46.0-SNAPSHOT~9b370b2dd7, baseline=1.46.0-SNAPSHOT~523df01c66

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.054 s) : 0, 1054088
Total [baseline] (8.602 s) : 0, 8602135
Agent [candidate] (1.051 s) : 0, 1051058
Total [candidate] (8.628 s) : 0, 8627532
section iast
Agent [baseline] (1.193 s) : 0, 1193076
Total [baseline] (9.18 s) : 0, 9179878
Agent [candidate] (1.192 s) : 0, 1192176
Total [candidate] (9.186 s) : 0, 9186139
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.182 s) : 0, 1181608
Total [baseline] (9.156 s) : 0, 9155836
Agent [candidate] (1.185 s) : 0, 1184675
Total [candidate] (9.164 s) : 0, 9163613
section iast_TELEMETRY_OFF
Agent [baseline] (1.181 s) : 0, 1180577
Total [baseline] (9.214 s) : 0, 9213821
Agent [candidate] (1.181 s) : 0, 1181052
Total [candidate] (9.211 s) : 0, 9211045

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.054 s	-
Agent	iast	1.193 s	138.989 ms (13.2%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.182 s	127.52 ms (12.1%)
Agent	iast_TELEMETRY_OFF	1.181 s	126.489 ms (12.0%)
Total	tracing	8.602 s	-
Total	iast	9.18 s	577.743 ms (6.7%)
Total	iast_HARDCODED_SECRET_DISABLED	9.156 s	553.701 ms (6.4%)
Total	iast_TELEMETRY_OFF	9.214 s	611.686 ms (7.1%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.051 s	-
Agent	iast	1.192 s	141.118 ms (13.4%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.185 s	133.618 ms (12.7%)
Agent	iast_TELEMETRY_OFF	1.181 s	129.994 ms (12.4%)
Total	tracing	8.628 s	-
Total	iast	9.186 s	558.607 ms (6.5%)
Total	iast_HARDCODED_SECRET_DISABLED	9.164 s	536.082 ms (6.2%)
Total	iast_TELEMETRY_OFF	9.211 s	583.513 ms (6.8%)

gantt
    title insecure-bank - break down per module: candidate=1.46.0-SNAPSHOT~9b370b2dd7, baseline=1.46.0-SNAPSHOT~523df01c66

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (712.548 ms) : 0, 712548
BytebuddyAgent [candidate] (712.677 ms) : 0, 712677
GlobalTracer [baseline] (255.248 ms) : 0, 255248
GlobalTracer [candidate] (254.681 ms) : 0, 254681
AppSec [baseline] (55.632 ms) : 0, 55632
AppSec [candidate] (55.982 ms) : 0, 55982
Remote Config [baseline] (737.681 µs) : 0, 738
Remote Config [candidate] (743.211 µs) : 0, 743
Telemetry [baseline] (14.991 ms) : 0, 14991
Telemetry [candidate] (12.009 ms) : 0, 12009
section iast
BytebuddyAgent [baseline] (839.604 ms) : 0, 839604
BytebuddyAgent [candidate] (840.008 ms) : 0, 840008
GlobalTracer [baseline] (248.367 ms) : 0, 248367
GlobalTracer [candidate] (247.903 ms) : 0, 247903
AppSec [baseline] (58.671 ms) : 0, 58671
AppSec [candidate] (58.13 ms) : 0, 58130
Remote Config [baseline] (676.43 µs) : 0, 676
Remote Config [candidate] (670.604 µs) : 0, 671
Telemetry [baseline] (8.933 ms) : 0, 8933
Telemetry [candidate] (8.794 ms) : 0, 8794
IAST [baseline] (21.716 ms) : 0, 21716
IAST [candidate] (21.477 ms) : 0, 21477
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (830.766 ms) : 0, 830766
BytebuddyAgent [candidate] (833.261 ms) : 0, 833261
GlobalTracer [baseline] (246.744 ms) : 0, 246744
GlobalTracer [candidate] (246.937 ms) : 0, 246937
AppSec [baseline] (58.052 ms) : 0, 58052
AppSec [candidate] (58.455 ms) : 0, 58455
Remote Config [baseline] (671.626 µs) : 0, 672
Remote Config [candidate] (670.76 µs) : 0, 671
Telemetry [baseline] (8.822 ms) : 0, 8822
Telemetry [candidate] (8.826 ms) : 0, 8826
IAST [baseline] (21.552 ms) : 0, 21552
IAST [candidate] (21.485 ms) : 0, 21485
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (830.545 ms) : 0, 830545
BytebuddyAgent [candidate] (829.969 ms) : 0, 829969
GlobalTracer [baseline] (246.517 ms) : 0, 246517
GlobalTracer [candidate] (247.861 ms) : 0, 247861
AppSec [baseline] (58.036 ms) : 0, 58036
AppSec [candidate] (57.949 ms) : 0, 57949
Remote Config [baseline] (678.262 µs) : 0, 678
Remote Config [candidate] (671.796 µs) : 0, 672
Telemetry [baseline] (8.688 ms) : 0, 8688
Telemetry [candidate] (8.648 ms) : 0, 8648
IAST [baseline] (21.13 ms) : 0, 21130
IAST [candidate] (20.91 ms) : 0, 20910

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-01-22T12:47:15	2025-01-22T12:54:19
git_branch	master	nikita-tkachenko/impacted-tests
git_commit_date	1737548458	1737549292
git_commit_sha	`523df01`	`9b370b2`
release_version	1.46.0-SNAPSHOT~523df01c66	1.46.0-SNAPSHOT~9b370b2dd7
start_time	2025-01-22T12:47:02	2025-01-22T12:54:05

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1737550816	1737550816
ci_job_id	775687408	775687408
ci_pipeline_id	53672611	53672611
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 17 unstable metrics.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.46.0-SNAPSHOT~9b370b2dd7, baseline=1.46.0-SNAPSHOT~523df01c66
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.352 ms) : 1332, 1372
.   : milestone, 1352,
appsec (1.747 ms) : 1723, 1771
.   : milestone, 1747,
appsec_no_iast (1.76 ms) : 1737, 1782
.   : milestone, 1760,
iast (1.518 ms) : 1494, 1541
.   : milestone, 1518,
profiling (1.544 ms) : 1519, 1568
.   : milestone, 1544,
tracing (1.497 ms) : 1473, 1521
.   : milestone, 1497,
section candidate
no_agent (1.362 ms) : 1341, 1383
.   : milestone, 1362,
appsec (1.754 ms) : 1731, 1778
.   : milestone, 1754,
appsec_no_iast (1.748 ms) : 1724, 1772
.   : milestone, 1748,
iast (1.495 ms) : 1471, 1520
.   : milestone, 1495,
profiling (1.565 ms) : 1540, 1591
.   : milestone, 1565,
tracing (1.49 ms) : 1466, 1513
.   : milestone, 1490,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.352 ms [1.332 ms, 1.372 ms]	-
appsec	1.747 ms [1.723 ms, 1.771 ms]	394.746 µs (29.2%)
appsec_no_iast	1.76 ms [1.737 ms, 1.782 ms]	407.634 µs (30.1%)
iast	1.518 ms [1.494 ms, 1.541 ms]	165.773 µs (12.3%)
profiling	1.544 ms [1.519 ms, 1.568 ms]	191.695 µs (14.2%)
tracing	1.497 ms [1.473 ms, 1.521 ms]	144.944 µs (10.7%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.362 ms [1.341 ms, 1.383 ms]	-
appsec	1.754 ms [1.731 ms, 1.778 ms]	392.243 µs (28.8%)
appsec_no_iast	1.748 ms [1.724 ms, 1.772 ms]	385.68 µs (28.3%)
iast	1.495 ms [1.471 ms, 1.52 ms]	133.133 µs (9.8%)
profiling	1.565 ms [1.54 ms, 1.591 ms]	203.224 µs (14.9%)
tracing	1.49 ms [1.466 ms, 1.513 ms]	127.394 µs (9.4%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.46.0-SNAPSHOT~9b370b2dd7, baseline=1.46.0-SNAPSHOT~523df01c66
    dateFormat X
    axisFormat %s
section baseline
no_agent (376.722 µs) : 356, 397
.   : milestone, 377,
iast (499.509 µs) : 478, 521
.   : milestone, 500,
iast_FULL (742.903 µs) : 721, 765
.   : milestone, 743,
iast_GLOBAL (551.328 µs) : 529, 574
.   : milestone, 551,
iast_HARDCODED_SECRET_DISABLED (504.328 µs) : 482, 526
.   : milestone, 504,
iast_INACTIVE (454.876 µs) : 433, 476
.   : milestone, 455,
iast_TELEMETRY_OFF (488.902 µs) : 467, 511
.   : milestone, 489,
tracing (455.112 µs) : 434, 476
.   : milestone, 455,
section candidate
no_agent (370.066 µs) : 350, 391
.   : milestone, 370,
iast (503.217 µs) : 481, 525
.   : milestone, 503,
iast_FULL (737.993 µs) : 716, 760
.   : milestone, 738,
iast_GLOBAL (540.754 µs) : 519, 562
.   : milestone, 541,
iast_HARDCODED_SECRET_DISABLED (501.757 µs) : 480, 523
.   : milestone, 502,
iast_INACTIVE (452.245 µs) : 431, 474
.   : milestone, 452,
iast_TELEMETRY_OFF (487.025 µs) : 466, 508
.   : milestone, 487,
tracing (447.081 µs) : 426, 468
.   : milestone, 447,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	376.722 µs [355.948 µs, 397.497 µs]	-
iast	499.509 µs [477.733 µs, 521.286 µs]	122.787 µs (32.6%)
iast_FULL	742.903 µs [720.924 µs, 764.882 µs]	366.181 µs (97.2%)
iast_GLOBAL	551.328 µs [529.13 µs, 573.526 µs]	174.605 µs (46.3%)
iast_HARDCODED_SECRET_DISABLED	504.328 µs [482.451 µs, 526.205 µs]	127.605 µs (33.9%)
iast_INACTIVE	454.876 µs [433.321 µs, 476.431 µs]	78.153 µs (20.7%)
iast_TELEMETRY_OFF	488.902 µs [466.989 µs, 510.814 µs]	112.179 µs (29.8%)
tracing	455.112 µs [433.933 µs, 476.29 µs]	78.389 µs (20.8%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	370.066 µs [349.574 µs, 390.557 µs]	-
iast	503.217 µs [481.495 µs, 524.94 µs]	133.152 µs (36.0%)
iast_FULL	737.993 µs [716.021 µs, 759.964 µs]	367.927 µs (99.4%)
iast_GLOBAL	540.754 µs [519.131 µs, 562.377 µs]	170.688 µs (46.1%)
iast_HARDCODED_SECRET_DISABLED	501.757 µs [480.43 µs, 523.084 µs]	131.691 µs (35.6%)
iast_INACTIVE	452.245 µs [430.727 µs, 473.763 µs]	82.18 µs (22.2%)
iast_TELEMETRY_OFF	487.025 µs [465.553 µs, 508.496 µs]	116.959 µs (31.6%)
tracing	447.081 µs [425.714 µs, 468.447 µs]	77.015 µs (20.8%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/impacted-tests
git_commit_date	1737548458	1737549292
git_commit_sha	`523df01`	`9b370b2`
release_version	1.46.0-SNAPSHOT~523df01c66	1.46.0-SNAPSHOT~9b370b2dd7

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1737551276	1737551276
ci_job_id	775687409	775687409
ci_pipeline_id	53672611	53672611
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.46.0-SNAPSHOT~9b370b2dd7, baseline=1.46.0-SNAPSHOT~523df01c66
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.962 s) : 14962000, 14962000
.   : milestone, 14962000,
appsec (15.019 s) : 15019000, 15019000
.   : milestone, 15019000,
iast (18.912 s) : 18912000, 18912000
.   : milestone, 18912000,
iast_GLOBAL (17.728 s) : 17728000, 17728000
.   : milestone, 17728000,
profiling (15.033 s) : 15033000, 15033000
.   : milestone, 15033000,
tracing (15.031 s) : 15031000, 15031000
.   : milestone, 15031000,
section candidate
no_agent (15.591 s) : 15591000, 15591000
.   : milestone, 15591000,
appsec (15.028 s) : 15028000, 15028000
.   : milestone, 15028000,
iast (19.055 s) : 19055000, 19055000
.   : milestone, 19055000,
iast_GLOBAL (18.133 s) : 18133000, 18133000
.   : milestone, 18133000,
profiling (15.626 s) : 15626000, 15626000
.   : milestone, 15626000,
tracing (15.049 s) : 15049000, 15049000
.   : milestone, 15049000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.962 s [14.962 s, 14.962 s]	-
appsec	15.019 s [15.019 s, 15.019 s]	57.0 ms (0.4%)
iast	18.912 s [18.912 s, 18.912 s]	3.95 s (26.4%)
iast_GLOBAL	17.728 s [17.728 s, 17.728 s]	2.766 s (18.5%)
profiling	15.033 s [15.033 s, 15.033 s]	71.0 ms (0.5%)
tracing	15.031 s [15.031 s, 15.031 s]	69.0 ms (0.5%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.591 s [15.591 s, 15.591 s]	-
appsec	15.028 s [15.028 s, 15.028 s]	-563.0 ms (-3.6%)
iast	19.055 s [19.055 s, 19.055 s]	3.464 s (22.2%)
iast_GLOBAL	18.133 s [18.133 s, 18.133 s]	2.542 s (16.3%)
profiling	15.626 s [15.626 s, 15.626 s]	35.0 ms (0.2%)
tracing	15.049 s [15.049 s, 15.049 s]	-542.0 ms (-3.5%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.46.0-SNAPSHOT~9b370b2dd7, baseline=1.46.0-SNAPSHOT~523df01c66
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.469 ms) : 1458, 1481
.   : milestone, 1469,
appsec (2.359 ms) : 2316, 2401
.   : milestone, 2359,
iast (2.105 ms) : 2051, 2160
.   : milestone, 2105,
iast_GLOBAL (2.153 ms) : 2099, 2208
.   : milestone, 2153,
profiling (1.957 ms) : 1914, 2001
.   : milestone, 1957,
tracing (1.941 ms) : 1899, 1983
.   : milestone, 1941,
section candidate
no_agent (1.473 ms) : 1462, 1485
.   : milestone, 1473,
appsec (2.362 ms) : 2319, 2404
.   : milestone, 2362,
iast (2.106 ms) : 2052, 2160
.   : milestone, 2106,
iast_GLOBAL (2.148 ms) : 2093, 2202
.   : milestone, 2148,
profiling (1.955 ms) : 1912, 1998
.   : milestone, 1955,
tracing (1.952 ms) : 1910, 1994
.   : milestone, 1952,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.469 ms [1.458 ms, 1.481 ms]	-
appsec	2.359 ms [2.316 ms, 2.401 ms]	889.106 µs (60.5%)
iast	2.105 ms [2.051 ms, 2.16 ms]	636.024 µs (43.3%)
iast_GLOBAL	2.153 ms [2.099 ms, 2.208 ms]	683.763 µs (46.5%)
profiling	1.957 ms [1.914 ms, 2.001 ms]	487.834 µs (33.2%)
tracing	1.941 ms [1.899 ms, 1.983 ms]	471.556 µs (32.1%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.473 ms [1.462 ms, 1.485 ms]	-
appsec	2.362 ms [2.319 ms, 2.404 ms]	888.493 µs (60.3%)
iast	2.106 ms [2.052 ms, 2.16 ms]	632.732 µs (43.0%)
iast_GLOBAL	2.148 ms [2.093 ms, 2.202 ms]	674.383 µs (45.8%)
profiling	1.955 ms [1.912 ms, 1.998 ms]	482.043 µs (32.7%)
tracing	1.952 ms [1.91 ms, 1.994 ms]	479.239 µs (32.5%)

…tection

…tion

…+ extract a no-op git client implementation to avoid spamming logs with exceptions in envs where git executable is not available

| Package | Type | Package file | Manager | Update | Change | |---|---|---|---|---|---| | [com.google.cloud:google-cloud-datastore](https://github.com/googleapis/java-datastore) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `2.25.4` -> `2.26.0` | | [com.datadoghq:dd-trace-api](https://github.com/datadog/dd-trace-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `1.45.2` -> `1.46.0` | | [com.datadoghq:dd-trace-ot](https://github.com/datadog/dd-trace-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `1.45.2` -> `1.46.0` | | [software.amazon.awssdk:sdk-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.9` -> `2.30.10` | | [software.amazon.awssdk:dynamodb-enhanced](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.9` -> `2.30.10` | | [software.amazon.awssdk:dynamodb](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.9` -> `2.30.10` | | [software.amazon.awssdk:aws-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.9` -> `2.30.10` | | [software.amazon.awssdk:bom](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.9` -> `2.30.10` | | [software.amazon.awssdk:auth](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.9` -> `2.30.10` | --- ### Release Notes <details> <summary>googleapis/java-datastore (com.google.cloud:google-cloud-datastore)</summary> ### [`v2.26.0`](https://github.com/googleapis/java-datastore/blob/HEAD/CHANGELOG.md#2260-2025-01-29) ##### Features - Add firestoreInDatastoreMode for datastore emulator ([#1698](googleapis/java-datastore#1698)) ([50f106d](googleapis/java-datastore@50f106d)) ##### Dependencies - Update dependency com.google.cloud:sdk-platform-java-config to v3.42.0 ([#1725](googleapis/java-datastore#1725)) ([1cbaf22](googleapis/java-datastore@1cbaf22)) </details> <details> <summary>datadog/dd-trace-java (com.datadoghq:dd-trace-api)</summary> ### [`v1.46.0`](https://github.com/DataDog/dd-trace-java/releases/tag/v1.46.0): 1.46.0 ##### Breaking Changes > \[!WARNING] > jnr-unixsocket is now an external dependency of dd-trace-ot and must be included when deploying dd-trace-ot. > \[!NOTE] > The API `TracerScope.setAsync(boolean)`, used to manually control asynchronous span propagation, does no more apply to the scope instance but to the active span scope. ##### Components ##### Application Security Management (IAST) - 🐛 Fix String.replace instrumentation for IAST ([#8281](DataDog/dd-trace-java#8281) - [@Mariovido](https://github.com/Mariovido)) - ✨ Apply the standard nomenclature to the stacktrace configs ([#8244](DataDog/dd-trace-java#8244) - [@jandro996](https://github.com/jandro996)) - 🐛 Exclude false positive weak randomness ([#8232](DataDog/dd-trace-java#8232) - [@jandro996](https://github.com/jandro996)) - ✨ Propagation of translateEscapes of String class ([#8186](DataDog/dd-trace-java#8186) - [@sezen-datadog](https://github.com/sezen-datadog)) - ✨ Add security control metrics ([#8175](DataDog/dd-trace-java#8175) - [@jandro996](https://github.com/jandro996)) - ✨ Increase IAST propagation to StringBuffer setLength ([#8128](DataDog/dd-trace-java#8128) - [@Mariovido](https://github.com/Mariovido)) - ✨ Add IAST taint tracking for DB values ([#8072](DataDog/dd-trace-java#8072) - [@Mariovido](https://github.com/Mariovido)) ##### Application Security Management (WAF) - 🐛 Prevents a NPE when there is no subscriber for user events ([#8258](DataDog/dd-trace-java#8258) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Apply the standard nomenclature to the stacktrace configs ([#8244](DataDog/dd-trace-java#8244) - [@jandro996](https://github.com/jandro996)) - 🐛 Ensure cached subscriptions are cleared on reconfiguration via RC ([#8229](DataDog/dd-trace-java#8229) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Add support for session tracking in Vertx ([#8167](DataDog/dd-trace-java#8167) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Create span tag: \_dd.appsec.rasp.timeout ([#8269](DataDog/dd-trace-java#8269) - [@Mariovido](https://github.com/Mariovido)) ##### Build & Tooling - 🐛 Ensure shaded helpers have unique names when injected into class-loaders ([#8192](DataDog/dd-trace-java#8192) - [@mcculls](https://github.com/mcculls)) ##### Configuration at Runtime - 🐛 Remove filtering of `DD_SERVICE` and `DD_ENV` from the tracer ([#8176](DataDog/dd-trace-java#8176) - [@mhlidd](https://github.com/mhlidd)) ##### Continuous Integration Visibility - 🧹 Generalize TestRetryPolicy to TestExecutionPolicy ([#8302](DataDog/dd-trace-java#8302) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🧹 Parallelize CI Visibility settings requests ([#8299](DataDog/dd-trace-java#8299) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🧹 Generalize test retry logic ([#8289](DataDog/dd-trace-java#8289) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🧹 Generalize tests skipping logic ([#8288](DataDog/dd-trace-java#8288) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🧹 Remove skip and shouldBeSkipped methods from TestEventsHandler in favor of isSkippable ([#8286](DataDog/dd-trace-java#8286) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨⚡ Optimize Git repository information computation ([#8270](DataDog/dd-trace-java#8270) - [@dougqh](https://github.com/dougqh)) - ✨ Always request known tests from the backend ([#8268](DataDog/dd-trace-java#8268) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Fix NPE when trying to get retry analyzer in Test NG ([#8253](DataDog/dd-trace-java#8253) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🐛 Set test framework and test framework version tags atomically ([#8252](DataDog/dd-trace-java#8252) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Add debug logging to Android Gradle module layout logic ([#8251](DataDog/dd-trace-java#8251) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🐛 Fix source and destination folders computation for Android Gradle projects ([#8190](DataDog/dd-trace-java#8190) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Add basic Scala Weaver sbt support ([#8189](DataDog/dd-trace-java#8189) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Implement impacted tests detection ([#8188](DataDog/dd-trace-java#8188) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) ##### Data Streams Monitoring - ✨ Change hash computation for protobuf to better represent impacting changes + save proto number in schema ([#8201](DataDog/dd-trace-java#8201) - [@vandonr](https://github.com/vandonr)) ##### Database Monitoring - Add peer service tag in dbm sql commenter ([#7913](DataDog/dd-trace-java#7913) - [@jordan-wong](https://github.com/jordan-wong)) ##### Dynamic Instrumentation - ✨ Add support for SymDB to scan directories ([#8306](DataDog/dd-trace-java#8306) - [@jpbempel](https://github.com/jpbempel)) - ✨ Add SymDB report for any jar scanning failures ([#8300](DataDog/dd-trace-java#8300) - [@jpbempel](https://github.com/jpbempel)) - ✨ Use two budgets depending on type ([#8283](DataDog/dd-trace-java#8283) - [@evanchooly](https://github.com/evanchooly)) - ✨ Institute a 10 snapshot per probe per trace budget ([#8277](DataDog/dd-trace-java#8277) - [@evanchooly](https://github.com/evanchooly)) - 🐛 Avoid double snapshots for Exception Replay ([#8273](DataDog/dd-trace-java#8273) - [@jpbempel](https://github.com/jpbempel)) - ✨ Simplify code origins. Separate out snapshot generation. ([#8263](DataDog/dd-trace-java#8263) - [@evanchooly](https://github.com/evanchooly)) - ✨ Add Exception probe custom instrumentation ([#8230](DataDog/dd-trace-java#8230) - [@jpbempel](https://github.com/jpbempel)) - ✨ Enhance log probes to honor debug session tags ([#8215](DataDog/dd-trace-java#8215) - [@evanchooly](https://github.com/evanchooly)) - 🐛 Don't redact env tokens from debugger probe snapshots ([#8211](DataDog/dd-trace-java#8211) - [@watson](https://github.com/watson)) - ✨⚡ Move Trace/SpanId capture at commit time ([#8184](DataDog/dd-trace-java#8184) - [@jpbempel](https://github.com/jpbempel)) - 🐛 Capture values at entry for method probe ([#8169](DataDog/dd-trace-java#8169) - [@jpbempel](https://github.com/jpbempel)) ##### JMX fetch - 🐛 Mute JMXFetch Shutdown in progress error ([#8068](DataDog/dd-trace-java#8068) - [@ygree](https://github.com/ygree)) ##### OpenTracing - ⚠️🧹 Make jnr-unixsocket an explicit dependency of dd-trace-ot ([#8307](DataDog/dd-trace-java#8307) - [@mcculls](https://github.com/mcculls)) ##### Profiling - 🐛 Avoid unsupported API call for creating folders on windows ([#8304](DataDog/dd-trace-java#8304) - [@jbachorik](https://github.com/jbachorik)) - ✨ Tag profiles for serverless ([#8279](DataDog/dd-trace-java#8279) - [@jbachorik](https://github.com/jbachorik)) - ✨ add queue type and length to queue events ([#8242](DataDog/dd-trace-java#8242) - [@richardstartin](https://github.com/richardstartin)) - 🐛 TempLocationManager Fixes and Improvements ([#8191](DataDog/dd-trace-java#8191) - [@jbachorik](https://github.com/jbachorik)) - ✨ Bump ddprof to 1.18.0 ([#8173](DataDog/dd-trace-java#8173) - [@jbachorik](https://github.com/jbachorik)) - ✨ Report profiler initialization and configuration errors to telemetry ([#8171](DataDog/dd-trace-java#8171) - [@jbachorik](https://github.com/jbachorik)) ##### Telemetry - ✨ Add pending traces report in tracer flares ([#8053](DataDog/dd-trace-java#8053) - [@mhlidd](https://github.com/mhlidd)) ##### Testing - ✨ Test http server requests in parallel ([#8222](DataDog/dd-trace-java#8222) - [@amarziali](https://github.com/amarziali)) ##### Trace context propagation - ✨ Add non default propagator registration ([#8310](DataDog/dd-trace-java#8310) - [@PerfectSlayer](https://github.com/PerfectSlayer)) ##### Tracer core - ✨ Probe for existence of IBMSASL or ACCP security providers ([#8276](DataDog/dd-trace-java#8276) - [@mcculls](https://github.com/mcculls)) - ✨⚡ Overhead improvement to agent feedback based sampling ([#8265](DataDog/dd-trace-java#8265) - [@dougqh](https://github.com/dougqh)) - 🧹 Move async propagation API from scope to tracer ([#8231](DataDog/dd-trace-java#8231) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - ✨ Introduce context propagation API ([#8161](DataDog/dd-trace-java#8161) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - ✨🧪 Use env-entry to add tags per webapp deployment ([#8138](DataDog/dd-trace-java#8138) - [@amarziali](https://github.com/amarziali)) - ✨ Introduce context helpers API ([#8134](DataDog/dd-trace-java#8134) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - ✨ Support IPv6 values for `DD_AGENT_HOST` and `DD_TRACE_AGENT_URL` ([#7984](DataDog/dd-trace-java#7984) - [@mhlidd](https://github.com/mhlidd)) ##### Instrumentations ##### Apache HttpComponents - 🐛 Properly finish spans and support latest apache httpclient5 ([#8272](DataDog/dd-trace-java#8272) - [@amarziali](https://github.com/amarziali)) ##### AWS Lambda instrumentation - 🐛 Properly capture lambda payloads for all handler types. ([#8264](DataDog/dd-trace-java#8264) - [@purple4reina](https://github.com/purple4reina)) ##### AWS S3 instrumentation - 💡 Create S3 instrumentation + add span pointers ([#8075](DataDog/dd-trace-java#8075) - [@nhulston](https://github.com/nhulston)) ##### AWS SDK instrumentation - 🐛 Revert "Add avoid double instrumenting lambda non-streaming handlers." ([#8247](DataDog/dd-trace-java#8247) - [@nhulston](https://github.com/nhulston)) ##### Cassandra - ✨ Allow extracting keyspace from statement result ([#8239](DataDog/dd-trace-java#8239) - [@amarziali](https://github.com/amarziali)) ##### Core Java language instrumentation - ✨ Propagation of translateEscapes of String class ([#8186](DataDog/dd-trace-java#8186) - [@sezen-datadog](https://github.com/sezen-datadog)) ##### Eclipse Vert.x instrumentation - 🐛 Fix vertx worker propagation and error handling ([#8237](DataDog/dd-trace-java#8237) - [@amarziali](https://github.com/amarziali)) - ✨ Support vertx 5 ([#8220](DataDog/dd-trace-java#8220) - [@amarziali](https://github.com/amarziali)) - ✨ Add support for session tracking in Vertx ([#8167](DataDog/dd-trace-java#8167) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) ##### Kafka instrumentation - 🐛 Prevent possible NPE calculating Kafka record header size ([#8292](DataDog/dd-trace-java#8292) - [@ygree](https://github.com/ygree)) ##### Mule instrumentation - 🐛 Fix crash using Mule with JPMS ([#8187](DataDog/dd-trace-java#8187) - [@amarziali](https://github.com/amarziali)) ##### Protocol Buffer instrumentation - ✨ Change hash computation for protobuf to better represent impacting changes + save proto number in schema ([#8201](DataDog/dd-trace-java#8201) - [@vandonr](https://github.com/vandonr)) ##### Spring instrumentation - 🐛 Preserve getQualifier from spring scheduling runnables ([#8293](DataDog/dd-trace-java#8293) - [@amarziali](https://github.com/amarziali)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - "after 6pm every weekday,before 2am every weekday" in timezone Australia/Melbourne, Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). GitOrigin-RevId: bb09d47e4eed77a003f630273b4d0a84003eb899

nikita-tkachenko-datadog added tag: no release notes Changes to exclude from release notes type: refactoring comp: ci visibility Continuous Integration Visibility labels Jan 13, 2025

nikita-tkachenko-datadog force-pushed the nikita-tkachenko/impacted-tests branch from 0cfd616 to 8a25380 Compare January 16, 2025 17:49

nikita-tkachenko-datadog changed the title ~~Pass test source data to retry policy factory~~ Implement impacted tests detection Jan 16, 2025

nikita-tkachenko-datadog added type: enhancement Enhancements and improvements and removed tag: no release notes Changes to exclude from release notes type: refactoring labels Jan 16, 2025

nikita-tkachenko-datadog force-pushed the nikita-tkachenko/impacted-tests branch 2 times, most recently from 5560829 to 3b7cc1e Compare January 20, 2025 18:13

nikita-tkachenko-datadog added 16 commits January 22, 2025 13:34

Pass test source data to retry policy factory

e8eab99

Collect pull request info for GitHub and Gitlab

71a62e0

Implement git diff fetching and parsing

ebc6400

Add PR diff to execution settings

8e492ef

Implement EFD for modified tests

2584775

Implement backend config setting and killswitch for impacted tests de…

6f5c09a

…tection

Implement is_modified telemetry tag

94a681d

Extract Diff interface and implement polymorphic serialization

39acdab

Implement file-granularity diff

9868991

Implement fallback to file-level granularity for impacted tests detec…

8bebe62

…tion

Ensure git repo is not shallow before calculating git diff

d6c4567

Use commit SHA from backend diff response if no PR info is available …

5e0e4ed

…+ extract a no-op git client implementation to avoid spamming logs with exceptions in envs where git executable is not available

Fix spotBugs warning

0588338

Fix Weaver compilation after merge

e767b74

Add instrumentation tests for impacted tests detection

2fda4f6

Add smoke test for impacted tests detection

9b370b2

nikita-tkachenko-datadog force-pushed the nikita-tkachenko/impacted-tests branch from e6a5ff4 to 9b370b2 Compare January 22, 2025 12:35

nikita-tkachenko-datadog marked this pull request as ready for review January 22, 2025 13:51

nikita-tkachenko-datadog requested a review from a team as a code owner January 22, 2025 13:51

nikita-tkachenko-datadog requested review from juan-fernandez, nayeem-kamal and daniel-mohedano and removed request for a team, nayeem-kamal and juan-fernandez January 22, 2025 13:51

daniel-mohedano approved these changes Jan 22, 2025

View reviewed changes

nikita-tkachenko-datadog merged commit ad44687 into master Jan 22, 2025
177 checks passed

nikita-tkachenko-datadog deleted the nikita-tkachenko/impacted-tests branch January 22, 2025 16:03

github-actions bot added this to the 1.46.0 milestone Jan 22, 2025

nikita-tkachenko-datadog mentioned this pull request Feb 14, 2025

Disable backend requests for impacted tests data #8391

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Implement impacted tests detection #8188

Implement impacted tests detection #8188

Uh oh!

nikita-tkachenko-datadog commented Jan 13, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented Jan 13, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Implement impacted tests detection #8188

Implement impacted tests detection #8188

Uh oh!

Conversation

nikita-tkachenko-datadog commented Jan 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Additional Details

Contributor Checklist

Uh oh!

pr-commenter bot commented Jan 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

Uh oh!

Uh oh!

nikita-tkachenko-datadog commented Jan 13, 2025 •

edited

Loading

pr-commenter bot commented Jan 13, 2025 •

edited

Loading