Add possibility to retry every failed test when retries are enabled

[nikita-tkachenko-datadog]

trafficstars

What Does This Do

When flaky test retries are enabled in CI Visibility, by default every failed test will be retried up to N times. A config property is added to switch the tracer to its previous behaviour: only retry the tests that are known to be flaky.

Jira ticket: CIVIS-9432

[pr-commenter[bot]]

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/flaky-test-retry-everything
git_commit_date	1710526664	1710527121
git_commit_sha	36e924eaab	b1072db5bc
release_version	1.32.0-SNAPSHOT~36e924eaab	1.32.0-SNAPSHOT~b1072db5bc

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1710530594	1710530594
ci_job_id	461342839	461342839
ci_pipeline_id	30217672	30217672
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 2 performance improvements and 2 performance regressions! Performance is the same for 45 metrics, 14 unstable metrics.

scenario	Δ mean execution_time	candidate mean execution_time	baseline mean execution_time
scenario:startup:insecure-bank:tracing:Remote Config	better [-367.546µs; -314.292µs] or [-34.043%; -29.110%]	0.739ms	1.080ms
scenario:startup:insecure-bank:tracing:Telemetry	worse [+247.043µs; +773.827µs] or [+3.358%; +10.517%]	7.868ms	7.358ms
scenario:startup:petclinic:tracing:Remote Config	better [-409.854µs; -302.114µs] or [-37.336%; -27.521%]	0.742ms	1.098ms
scenario:startup:petclinic:tracing:Telemetry	worse [+222.075µs; +699.849µs] or [+3.008%; +9.478%]	7.845ms	7.384ms

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.32.0-SNAPSHOT~b1072db5bc, baseline=1.32.0-SNAPSHOT~36e924eaab

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.086 s) : 0, 1085595
Total [baseline] (9.19 s) : 0, 9189862
Agent [candidate] (1.081 s) : 0, 1080819
Total [candidate] (9.157 s) : 0, 9157496
section appsec
Agent [baseline] (1.199 s) : 0, 1199012
Total [baseline] (9.283 s) : 0, 9283389
Agent [candidate] (1.209 s) : 0, 1209040
Total [candidate] (9.294 s) : 0, 9293520
section iast
Agent [baseline] (1.214 s) : 0, 1214417
Total [baseline] (9.497 s) : 0, 9497147
Agent [candidate] (1.205 s) : 0, 1205197
Total [candidate] (9.302 s) : 0, 9301733
section profiling
Agent [baseline] (1.271 s) : 0, 1271317
Total [baseline] (9.347 s) : 0, 9347314
Agent [candidate] (1.272 s) : 0, 1272473
Total [candidate] (9.401 s) : 0, 9401448

• baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.086 s	-
Agent	appsec	1.199 s	113.417 ms (10.4%)
Agent	iast	1.214 s	128.823 ms (11.9%)
Agent	profiling	1.271 s	185.722 ms (17.1%)
Total	tracing	9.19 s	-
Total	appsec	9.283 s	93.527 ms (1.0%)
Total	iast	9.497 s	307.285 ms (3.3%)
Total	profiling	9.347 s	157.452 ms (1.7%)

• candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.081 s	-
Agent	appsec	1.209 s	128.221 ms (11.9%)
Agent	iast	1.205 s	124.378 ms (11.5%)
Agent	profiling	1.272 s	191.655 ms (17.7%)
Total	tracing	9.157 s	-
Total	appsec	9.294 s	136.024 ms (1.5%)
Total	iast	9.302 s	144.237 ms (1.6%)
Total	profiling	9.401 s	243.953 ms (2.7%)

gantt
    title petclinic - break down per module: candidate=1.32.0-SNAPSHOT~b1072db5bc, baseline=1.32.0-SNAPSHOT~36e924eaab

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (700.331 ms) : 0, 700331
BytebuddyAgent [candidate] (696.225 ms) : 0, 696225
GlobalTracer [baseline] (292.786 ms) : 0, 292786
GlobalTracer [candidate] (292.632 ms) : 0, 292632
AppSec [baseline] (49.427 ms) : 0, 49427
AppSec [candidate] (49.038 ms) : 0, 49038
Remote Config [baseline] (1.098 ms) : 0, 1098
Remote Config [candidate] (741.762 µs) : 0, 742
Telemetry [baseline] (7.384 ms) : 0, 7384
Telemetry [candidate] (7.845 ms) : 0, 7845
section appsec
BytebuddyAgent [baseline] (695.308 ms) : 0, 695308
BytebuddyAgent [candidate] (701.461 ms) : 0, 701461
GlobalTracer [baseline] (290.66 ms) : 0, 290660
GlobalTracer [candidate] (293.563 ms) : 0, 293563
AppSec [baseline] (153.515 ms) : 0, 153515
AppSec [candidate] (153.973 ms) : 0, 153973
IAST [baseline] (17.805 ms) : 0, 17805
IAST [candidate] (18.049 ms) : 0, 18049
Remote Config [baseline] (605.599 µs) : 0, 606
Remote Config [candidate] (609.795 µs) : 0, 610
Telemetry [baseline] (6.846 ms) : 0, 6846
Telemetry [candidate] (6.908 ms) : 0, 6908
section iast
BytebuddyAgent [baseline] (807.269 ms) : 0, 807269
BytebuddyAgent [candidate] (801.104 ms) : 0, 801104
GlobalTracer [baseline] (290.404 ms) : 0, 290404
GlobalTracer [candidate] (289.336 ms) : 0, 289336
AppSec [baseline] (52.453 ms) : 0, 52453
AppSec [candidate] (50.191 ms) : 0, 50191
IAST [baseline] (22.445 ms) : 0, 22445
IAST [candidate] (22.905 ms) : 0, 22905
Remote Config [baseline] (574.706 µs) : 0, 575
Remote Config [candidate] (592.816 µs) : 0, 593
Telemetry [baseline] (6.692 ms) : 0, 6692
Telemetry [candidate] (6.629 ms) : 0, 6629
section profiling
BytebuddyAgent [baseline] (687.795 ms) : 0, 687795
BytebuddyAgent [candidate] (688.09 ms) : 0, 688090
GlobalTracer [baseline] (375.156 ms) : 0, 375156
GlobalTracer [candidate] (376.006 ms) : 0, 376006
AppSec [baseline] (49.83 ms) : 0, 49830
AppSec [candidate] (49.548 ms) : 0, 49548
Remote Config [baseline] (895.934 µs) : 0, 896
Remote Config [candidate] (792.563 µs) : 0, 793
Telemetry [baseline] (7.255 ms) : 0, 7255
Telemetry [candidate] (7.347 ms) : 0, 7347
ProfilingAgent [baseline] (94.263 ms) : 0, 94263
ProfilingAgent [candidate] (94.716 ms) : 0, 94716
Profiling [baseline] (94.286 ms) : 0, 94286
Profiling [candidate] (94.74 ms) : 0, 94740

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.32.0-SNAPSHOT~b1072db5bc, baseline=1.32.0-SNAPSHOT~36e924eaab

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.084 s) : 0, 1084186
Total [baseline] (8.559 s) : 0, 8559329
Agent [candidate] (1.083 s) : 0, 1083202
Total [candidate] (8.592 s) : 0, 8591999
section iast
Agent [baseline] (1.202 s) : 0, 1201824
Total [baseline] (9.06 s) : 0, 9059825
Agent [candidate] (1.208 s) : 0, 1207556
Total [candidate] (9.056 s) : 0, 9056304
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.205 s) : 0, 1205141
Total [baseline] (9.018 s) : 0, 9017998
Agent [candidate] (1.213 s) : 0, 1213190
Total [candidate] (9.053 s) : 0, 9052862
section iast_TELEMETRY_OFF
Agent [baseline] (1.198 s) : 0, 1198292
Total [baseline] (9.047 s) : 0, 9046587
Agent [candidate] (1.203 s) : 0, 1203154
Total [candidate] (9.054 s) : 0, 9053899

• baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.084 s	-
Agent	iast	1.202 s	117.638 ms (10.9%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.205 s	120.955 ms (11.2%)
Agent	iast_TELEMETRY_OFF	1.198 s	114.106 ms (10.5%)
Total	tracing	8.559 s	-
Total	iast	9.06 s	500.497 ms (5.8%)
Total	iast_HARDCODED_SECRET_DISABLED	9.018 s	458.669 ms (5.4%)
Total	iast_TELEMETRY_OFF	9.047 s	487.259 ms (5.7%)

• candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.083 s	-
Agent	iast	1.208 s	124.353 ms (11.5%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.213 s	129.988 ms (12.0%)
Agent	iast_TELEMETRY_OFF	1.203 s	119.952 ms (11.1%)
Total	tracing	8.592 s	-
Total	iast	9.056 s	464.305 ms (5.4%)
Total	iast_HARDCODED_SECRET_DISABLED	9.053 s	460.863 ms (5.4%)
Total	iast_TELEMETRY_OFF	9.054 s	461.9 ms (5.4%)

gantt
    title insecure-bank - break down per module: candidate=1.32.0-SNAPSHOT~b1072db5bc, baseline=1.32.0-SNAPSHOT~36e924eaab

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (700.641 ms) : 0, 700641
BytebuddyAgent [candidate] (698.774 ms) : 0, 698774
GlobalTracer [baseline] (291.428 ms) : 0, 291428
GlobalTracer [candidate] (292.315 ms) : 0, 292315
AppSec [baseline] (49.121 ms) : 0, 49121
AppSec [candidate] (49.081 ms) : 0, 49081
Remote Config [baseline] (1.08 ms) : 0, 1080
Remote Config [candidate] (738.744 µs) : 0, 739
Telemetry [baseline] (7.358 ms) : 0, 7358
Telemetry [candidate] (7.868 ms) : 0, 7868
section iast
BytebuddyAgent [baseline] (798.832 ms) : 0, 798832
BytebuddyAgent [candidate] (801.868 ms) : 0, 801868
GlobalTracer [baseline] (287.415 ms) : 0, 287415
GlobalTracer [candidate] (290.069 ms) : 0, 290069
AppSec [baseline] (51.147 ms) : 0, 51147
AppSec [candidate] (52.657 ms) : 0, 52657
IAST [baseline] (22.087 ms) : 0, 22087
IAST [candidate] (21.213 ms) : 0, 21213
Remote Config [baseline] (572.502 µs) : 0, 573
Remote Config [candidate] (628.728 µs) : 0, 629
Telemetry [baseline] (7.402 ms) : 0, 7402
Telemetry [candidate] (6.675 ms) : 0, 6675
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (800.334 ms) : 0, 800334
BytebuddyAgent [candidate] (805.818 ms) : 0, 805818
GlobalTracer [baseline] (288.917 ms) : 0, 288917
GlobalTracer [candidate] (291.091 ms) : 0, 291091
AppSec [baseline] (50.623 ms) : 0, 50623
AppSec [candidate] (51.146 ms) : 0, 51146
IAST [baseline] (23.032 ms) : 0, 23032
IAST [candidate] (22.498 ms) : 0, 22498
Remote Config [baseline] (582.659 µs) : 0, 583
Remote Config [candidate] (613.25 µs) : 0, 613
Telemetry [baseline] (7.29 ms) : 0, 7290
Telemetry [candidate] (7.475 ms) : 0, 7475
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (793.539 ms) : 0, 793539
BytebuddyAgent [candidate] (796.811 ms) : 0, 796811
GlobalTracer [baseline] (289.059 ms) : 0, 289059
GlobalTracer [candidate] (290.958 ms) : 0, 290958
AppSec [baseline] (47.466 ms) : 0, 47466
AppSec [candidate] (51.398 ms) : 0, 51398
IAST [baseline] (25.955 ms) : 0, 25955
IAST [candidate] (22.402 ms) : 0, 22402
Remote Config [baseline] (577.741 µs) : 0, 578
Remote Config [candidate] (566.452 µs) : 0, 566
Telemetry [baseline] (7.325 ms) : 0, 7325
Telemetry [candidate] (6.536 ms) : 0, 6536

Load

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.32.0-SNAPSHOT~b1072db5bc, baseline=1.32.0-SNAPSHOT~36e924eaab
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.361 ms) : 1342, 1380
.   : milestone, 1361,
appsec (1.749 ms) : 1726, 1773
.   : milestone, 1749,
iast (1.52 ms) : 1497, 1543
.   : milestone, 1520,
profiling (1.519 ms) : 1496, 1543
.   : milestone, 1519,
tracing (1.521 ms) : 1498, 1544
.   : milestone, 1521,
section candidate
no_agent (1.358 ms) : 1339, 1377
.   : milestone, 1358,
appsec (1.788 ms) : 1764, 1811
.   : milestone, 1788,
iast (1.499 ms) : 1476, 1522
.   : milestone, 1499,
profiling (1.583 ms) : 1558, 1609
.   : milestone, 1583,
tracing (1.527 ms) : 1505, 1550
.   : milestone, 1527,

• baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.361 ms [1.342 ms, 1.38 ms]	-
appsec	1.749 ms [1.726 ms, 1.773 ms]	388.41 µs (28.5%)
iast	1.52 ms [1.497 ms, 1.543 ms]	159.162 µs (11.7%)
profiling	1.519 ms [1.496 ms, 1.543 ms]	158.166 µs (11.6%)
tracing	1.521 ms [1.498 ms, 1.544 ms]	159.95 µs (11.8%)

• candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.358 ms [1.339 ms, 1.377 ms]	-
appsec	1.788 ms [1.764 ms, 1.811 ms]	429.759 µs (31.7%)
iast	1.499 ms [1.476 ms, 1.522 ms]	141.142 µs (10.4%)
profiling	1.583 ms [1.558 ms, 1.609 ms]	225.557 µs (16.6%)
tracing	1.527 ms [1.505 ms, 1.55 ms]	169.539 µs (12.5%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.32.0-SNAPSHOT~b1072db5bc, baseline=1.32.0-SNAPSHOT~36e924eaab
    dateFormat X
    axisFormat %s
section baseline
no_agent (363.212 µs) : 343, 383
.   : milestone, 363,
iast (469.08 µs) : 449, 489
.   : milestone, 469,
iast_FULL (541.062 µs) : 521, 561
.   : milestone, 541,
iast_GLOBAL (495.0 µs) : 474, 516
.   : milestone, 495,
iast_HARDCODED_SECRET_DISABLED (472.99 µs) : 452, 494
.   : milestone, 473,
iast_INACTIVE (454.503 µs) : 434, 475
.   : milestone, 455,
iast_TELEMETRY_OFF (470.632 µs) : 450, 491
.   : milestone, 471,
tracing (454.708 µs) : 433, 476
.   : milestone, 455,
section candidate
no_agent (366.725 µs) : 347, 386
.   : milestone, 367,
iast (468.521 µs) : 448, 489
.   : milestone, 469,
iast_FULL (533.26 µs) : 513, 554
.   : milestone, 533,
iast_GLOBAL (491.5 µs) : 471, 512
.   : milestone, 492,
iast_HARDCODED_SECRET_DISABLED (476.561 µs) : 456, 497
.   : milestone, 477,
iast_INACTIVE (448.405 µs) : 428, 469
.   : milestone, 448,
iast_TELEMETRY_OFF (472.396 µs) : 452, 493
.   : milestone, 472,
tracing (455.415 µs) : 435, 476
.   : milestone, 455,

• baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	363.212 µs [342.957 µs, 383.467 µs]	-
iast	469.08 µs [448.83 µs, 489.33 µs]	105.868 µs (29.1%)
iast_FULL	541.062 µs [520.762 µs, 561.362 µs]	177.85 µs (49.0%)
iast_GLOBAL	495.0 µs [474.44 µs, 515.56 µs]	131.788 µs (36.3%)
iast_HARDCODED_SECRET_DISABLED	472.99 µs [452.156 µs, 493.824 µs]	109.778 µs (30.2%)
iast_INACTIVE	454.503 µs [433.69 µs, 475.316 µs]	91.291 µs (25.1%)
iast_TELEMETRY_OFF	470.632 µs [449.766 µs, 491.498 µs]	107.42 µs (29.6%)
tracing	454.708 µs [433.299 µs, 476.117 µs]	91.496 µs (25.2%)

• candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	366.725 µs [347.048 µs, 386.402 µs]	-
iast	468.521 µs [447.956 µs, 489.086 µs]	101.796 µs (27.8%)
iast_FULL	533.26 µs [512.816 µs, 553.704 µs]	166.535 µs (45.4%)
iast_GLOBAL	491.5 µs [471.205 µs, 511.795 µs]	124.775 µs (34.0%)
iast_HARDCODED_SECRET_DISABLED	476.561 µs [455.687 µs, 497.435 µs]	109.836 µs (30.0%)
iast_INACTIVE	448.405 µs [427.509 µs, 469.3 µs]	81.679 µs (22.3%)
iast_TELEMETRY_OFF	472.396 µs [451.504 µs, 493.287 µs]	105.671 µs (28.8%)
tracing	455.415 µs [434.6 µs, 476.23 µs]	88.69 µs (24.2%)