Memoize per-test class analysis in a module-wide cache by daniel-mohedano · Pull Request #11969 · DataDog/dd-trace-java

daniel-mohedano · 2026-07-16T09:04:09Z

What Does This Do

Memoizes per-test line-coverage class analysis in a module-wide cache.

When line-level coverage is enabled, JaCoCo's Analyzer re-parses and re-analyzes each covered class's bytecode once per test that covers it, which dominates the report-time cost. This caches the covered lines per (class id, probe set), shared across tests, so a class covered identically by many tests is analyzed only once.

Motivation

Reduce the runtime overhead of line-level per-test code coverage (Test Optimization). Report-time re-analysis was the dominant hotspot; in the test-environment, marginal line-coverage overhead drops meaningfully on suites with headroom (e.g. nebula, spring-boot, okhttp).

The probe recording and instrumentation paths are unchanged, so JaCoCo's aggregate coverage (total %/report uploads, used to back-fill coverage for TIA/ITR-skipped tests) is preserved by its native probe writes.

Additional Notes

Scoped to the report path only: LineCoverageStore.analyzeClass plus two getters on ExecutionDataAdapter. No changes to instrumentation, probe recording, or the coverage API.

The cache is memory-safe: keys bit-pack the probe activations (exact equality), the cache is bounded by approximate retained bytes (not entry count), and each analysis uses a single probe-array snapshot for both the key and the JaCoCo analysis so concurrent late writes can't poison an entry. Coverage stays correct beyond the size bound — the class is simply analyzed each time.

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any other useful labels
Avoid using close, fix, or any linking keywords when referencing an issue
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, migration, or deletion
Update public documentation with any new configuration flags or behaviors
Once approved, use merge queue to merge the PR

Jira ticket: SDTEST-3847

Jacoco's Analyzer re-parsed each covered class once per test, dominating line-coverage report cost. Cache the covered lines per (class id, probe set), shared across tests, so a class covered identically by many tests is analyzed only once. Recording path is unchanged, so Jacoco's aggregate coverage is preserved by its native probe writes (no probe-array swap). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

cit-pr-commenter-54b7da · 2026-07-16T09:24:21Z

Test Environment - nebula-release-plugin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	36.15	36.42	$\color{green}{\blacktriangledown}$ -0.27	36.42	$\color{green}{\blacktriangledown}$ -0.27	32/125
agentless	35.93	36.42	$\color{green}{\blacktriangledown}$ -0.49	36.42	$\color{green}{\blacktriangledown}$ -0.49	32/125
agentlessCodeCoverage	45.21	45.38	$\color{green}{\blacktriangledown}$ -0.17	44.48	$\color{red}{\blacktriangle}$ +0.73	32/125
agentlessLineCoverage	55.57	74.82	$\color{green}{\blacktriangledown}$ -19.25	74.82	$\color{green}{\blacktriangledown}$ -19.25	32/125

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-16T09:26:08Z

Test Environment - netflix-zuul

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	86.66	87.80	$\color{green}{\blacktriangledown}$ -1.14	87.80	$\color{green}{\blacktriangledown}$ -1.14	35/130
agentless	80.30	81.05	$\color{green}{\blacktriangledown}$ -0.75	81.05	$\color{green}{\blacktriangledown}$ -0.75	32/128
agentlessCodeCoverage	95.86	97.04	$\color{green}{\blacktriangledown}$ -1.18	95.12	$\color{red}{\blacktriangle}$ +0.74	32/126
agentlessLineCoverage	111.13	111.62	$\color{green}{\blacktriangledown}$ -0.49	111.62	$\color{green}{\blacktriangledown}$ -0.49	31/125

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-16T09:27:04Z

Test Environment - reactive-streams-jvm

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	23.29	21.65	$\color{red}{\blacktriangle}$ +1.64	21.65	$\color{red}{\blacktriangle}$ +1.64	40/134
agentless	19.42	18.82	$\color{red}{\blacktriangle}$ +0.60	18.82	$\color{red}{\blacktriangle}$ +0.60	36/129
agentlessCodeCoverage	21.44	19.99	$\color{red}{\blacktriangle}$ +1.45	19.99	$\color{red}{\blacktriangle}$ +1.45	34/126
agentlessLineCoverage	27.37	30.42	$\color{green}{\blacktriangledown}$ -3.05	29.82	$\color{green}{\blacktriangledown}$ -2.45	33/125

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-16T09:28:00Z

Test Environment - heliboard

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	9.47	9.73	$\color{green}{\blacktriangledown}$ -0.26	9.54	$\color{green}{\blacktriangledown}$ -0.07	29/60

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

dd-octo-sts · 2026-07-16T09:29:33Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	13.99 s	13.87 s	[+0.3%; +1.5%] (maybe worse)
startup:insecure-bank:tracing:Agent	12.92 s	12.94 s	[-0.9%; +0.6%] (no difference)
startup:petclinic:appsec:Agent	16.99 s	16.77 s	[+0.5%; +2.1%] (maybe worse)
startup:petclinic:iast:Agent	16.92 s	16.95 s	[-0.9%; +0.6%] (no difference)
startup:petclinic:profiling:Agent	16.67 s	16.78 s	[-1.6%; +0.3%] (no difference)
startup:petclinic:sca:Agent	16.93 s	16.82 s	[-0.2%; +1.6%] (no difference)
startup:petclinic:tracing:Agent	16.12 s	16.14 s	[-1.1%; +0.9%] (no difference)

Commit: 682893b4 · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

cit-pr-commenter-54b7da · 2026-07-16T09:32:53Z

Test Environment - sonar-kotlin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	13.78	12.87	$\color{red}{\blacktriangle}$ +0.91	12.87	$\color{red}{\blacktriangle}$ +0.91	31/121
agentless	12.95	12.12	$\color{red}{\blacktriangle}$ +0.83	12.12	$\color{red}{\blacktriangle}$ +0.83	29/118
agentlessCodeCoverage	16.37	14.51	$\color{red}{\blacktriangle}$ +1.86	15.11	$\color{red}{\blacktriangle}$ +1.26	29/118
agentlessLineCoverage	17.91	19.20	$\color{green}{\blacktriangledown}$ -1.29	19.20	$\color{green}{\blacktriangledown}$ -1.29	30/120

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-16T09:35:34Z

Test Environment - jolokia

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	92.29	93.23	$\color{green}{\blacktriangledown}$ -0.94	93.23	$\color{green}{\blacktriangledown}$ -0.94	35/134
agentless	89.33	89.58	$\color{green}{\blacktriangledown}$ -0.25	89.58	$\color{green}{\blacktriangledown}$ -0.25	36/132
agentlessCodeCoverage	99.72	99.00	$\color{red}{\blacktriangle}$ +0.72	99.00	$\color{red}{\blacktriangle}$ +0.72	34/129
agentlessLineCoverage	99.74	99.00	$\color{red}{\blacktriangle}$ +0.74	99.00	$\color{red}{\blacktriangle}$ +0.74	33/126

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-16T09:41:39Z

Test Environment - okhttp

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	19.12	19.20	$\color{green}{\blacktriangledown}$ -0.08	19.20	$\color{green}{\blacktriangledown}$ -0.08	28/129
agentless	17.99	18.82	$\color{green}{\blacktriangledown}$ -0.83	18.82	$\color{green}{\blacktriangledown}$ -0.83	28/126
agentlessCodeCoverage	22.85	22.09	$\color{red}{\blacktriangle}$ +0.76	22.54	$\color{red}{\blacktriangle}$ +0.31	29/125
agentlessLineCoverage	38.80	44.48	$\color{green}{\blacktriangledown}$ -5.68	44.48	$\color{green}{\blacktriangledown}$ -5.68	29/131

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-16T09:48:32Z

Test Environment - spring_boot

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	16.72	16.04	$\color{red}{\blacktriangle}$ +0.68	16.04	$\color{red}{\blacktriangle}$ +0.68	34/115
agentless	10.25	9.73	$\color{red}{\blacktriangle}$ +0.52	9.73	$\color{red}{\blacktriangle}$ +0.52	34/116
agentlessCodeCoverage	14.27	13.67	$\color{red}{\blacktriangle}$ +0.60	13.40	$\color{red}{\blacktriangle}$ +0.87	35/115
agentlessLineCoverage	22.80	32.95	$\color{green}{\blacktriangledown}$ -10.15	32.95	$\color{green}{\blacktriangledown}$ -10.15	33/114

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-16T10:09:50Z

Test Environment - sonar-java

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	11.38	8.13	$\color{red}{\blacktriangle}$ +3.25	13.13	$\color{green}{\blacktriangledown}$ -1.75	34/127
agentless	16.05	8.46	$\color{red}{\blacktriangle}$ +7.59	15.72	$\color{red}{\blacktriangle}$ +0.33	34/126
agentlessCodeCoverage	129.54	82.69	$\color{red}{\blacktriangle}$ +46.85	86.07	$\color{red}{\blacktriangle}$ +43.47	34/126
agentlessLineCoverage	130.37	130.99	$\color{green}{\blacktriangledown}$ -0.62	136.34	$\color{green}{\blacktriangledown}$ -5.97	35/126

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-17T08:47:49Z

Test Environment - pass4s

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	10.44	10.54	$\color{green}{\blacktriangledown}$ -0.10	10.54	$\color{green}{\blacktriangledown}$ -0.10	26/117
agentless	7.21	8.80	$\color{green}{\blacktriangledown}$ -1.59	10.13	$\color{green}{\blacktriangledown}$ -2.92	26/117
agentlessCodeCoverage	12.84	16.69	$\color{green}{\blacktriangledown}$ -3.85	17.03	$\color{green}{\blacktriangledown}$ -4.19	25/115

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

daniel-mohedano · 2026-07-17T09:10:08Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c60b08fac1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Addresses Codex review of the memoization cache: - Bit-pack the probe activations into a BitSet in the key (~8x smaller than the test's boolean[], exact equality). - Cap the cache by approximate retained bytes rather than entry count, so a class with many probes covered by many distinct probe sets can't retain unbounded memory for the module lifetime. - Reserve an entry's weight atomically before insertion (release on over-limit or lost putIfAbsent race) so concurrent inserts can't overshoot the bound. - Count fixed per-entry overhead (key + both BitSets + map node), not just the packed bits, so the byte bound is a real ceiling. - Snapshot the per-test probe array once and use that same snapshot for both the cache key and the JaCoCo analysis, so a late probe recorded by a background thread during report() can't be analyzed into coveredLines yet stored under the earlier key, poisoning the entry for later tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

cit-pr-commenter-54b7da · 2026-07-20T10:58:38Z

Test Environment - sbt-scalatest

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	52.57	55.43	$\color{green}{\blacktriangledown}$ -2.86	55.43	$\color{green}{\blacktriangledown}$ -2.86	60/248
agentEvpProxy	52.47	n/a	n/a	n/a	n/a	-

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

daniel-mohedano · 2026-07-20T12:04:01Z

@codex review

chatgpt-codex-connector · 2026-07-20T12:08:27Z

Codex Review: Didn't find any major issues. 👍

Reviewed commit: 682893b48d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

datadog-official

More details

Memoization is sound. The cache keys correctly identify class+probe pairs using bit-packed probes, concurrent insertion respects the 64MB memory bound through atomic byte counting, and probe snapshotting prevents late writes from poisoning entries. No regressions to coverage reporting or probe recording.

Was this helpful? React 👍 or 👎

_{📊 Validated against 12 scenarios · Open Bits AI session}

_{🤖 Datadog Autotest · Commit 682893b · What is Autotest? · Any feedback? Reach out in #autotest}

daniel-mohedano · 2026-07-20T12:53:48Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-07-20T12:53:51Z

View all feedbacks in Devflow UI.

2026-07-20 12:53:51 UTC ℹ️ Start processing command /merge

2026-07-20 12:53:56 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 2h (p90).

2026-07-20 13:52:11 UTC ℹ️ MergeQueue: This merge request was merged

daniel-mohedano added type: feature Enhancements and improvements comp: ci visibility Continuous Integration Visibility tag: ai generated Largely based on code generated by an AI or LLM labels Jul 16, 2026

This comment has been minimized.

Sign in to view

chatgpt-codex-connector Bot reviewed Jul 17, 2026

View reviewed changes

Comment thread ...-ci-visibility/src/main/java/datadog/trace/civisibility/coverage/line/LineCoverageStore.java Outdated

daniel-mohedano mentioned this pull request Jul 20, 2026

Reduce runtime overhead of line-level per-test coverage #11638

Closed

daniel-mohedano marked this pull request as ready for review July 20, 2026 12:37

daniel-mohedano requested a review from a team as a code owner July 20, 2026 12:37

datadog-official Bot reviewed Jul 20, 2026

View reviewed changes

juan-fernandez approved these changes Jul 20, 2026

View reviewed changes

gh-worker-dd-mergequeue-cf854d Bot merged commit 4719f5a into master Jul 20, 2026
590 of 593 checks passed

gh-worker-dd-mergequeue-cf854d Bot deleted the daniel.mohedano/line-coverage-memoization-only branch July 20, 2026 13:52

github-actions Bot added this to the 1.65.0 milestone Jul 20, 2026

Uh oh!

Conversation

daniel-mohedano commented Jul 16, 2026 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

This comment has been minimized.

cit-pr-commenter-54b7da Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - nebula-release-plugin

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - netflix-zuul

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - reactive-streams-jvm

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - heliboard

Uh oh!

dd-octo-sts Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - sonar-kotlin

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - jolokia

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - okhttp

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 16, 2026

Test Environment - spring_boot

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - sonar-java

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - pass4s

Uh oh!

daniel-mohedano commented Jul 17, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 20, 2026

Test Environment - sbt-scalatest

Uh oh!

daniel-mohedano commented Jul 20, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 20, 2026

Uh oh!

datadog-official Bot left a comment

Choose a reason for hiding this comment

Uh oh!

daniel-mohedano commented Jul 20, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

daniel-mohedano commented Jul 16, 2026 •

edited by atlassian Bot

Loading

cit-pr-commenter-54b7da Bot commented Jul 16, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 16, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 16, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 16, 2026 •

edited

Loading

dd-octo-sts Bot commented Jul 16, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 16, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 16, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 16, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 16, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 17, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Jul 20, 2026 •

edited

Loading