Reduce runtime overhead of line-level per-test coverage by daniel-mohedano · Pull Request #11638 · DataDog/dd-trace-java

daniel-mohedano · 2026-06-12T10:17:00Z

What Does This Do

Reduces the runtime overhead of line-level per-test code coverage (the JaCoCo-based LineCoverageStore path), with two complementary changes:

Per-method probe-array swap — instead of appending a CoveragePerTestBridge.recordCoverage(...) call after every JaCoCo probe, swap JaCoCo's shared probe array for a per-test array once at method entry (ProbeInserterInstrumentation). JaCoCo's own probe writes then record per-test coverage with zero per-probe overhead. JaCoCo's aggregate coverage (total coverage % and report uploads) is preserved by OR-ing the per-test probes back into the shared array at report time.
Report-time analysis memoization — LineCoverageStore.report() ran JaCoCo's Analyzer.analyzeClass once per covered class per test, which is the dominant cost of reporting (~97% of report time, ~0.8 ms/class). Results are now memoized per (classId, probe-set) in a module-wide cache, so a class covered identically across tests is parsed once.

Plus cleanup/fixes:

Removed now-dead per-probe and probe-count machinery (ClassInstrumenterInstrumentation, CoverageStore.Registry / setTotalProbeCount, recordCoverage(Class, long, int), the probeCounts plumbing) — per-test array sizing now comes from JaCoCo's array length.
Fixed an edge case where a test that throws before its first probe fires emitted an empty coverage report (classes with no covered probes are now skipped).

Motivation

Line-level coverage is significantly more expensive than file-level, which discourages enabling the finer-grained data that keeps total-coverage figures accurate when ITR/TIA skips tests. Profiling via test-environment showed the cost is dominated by report-time class re-analysis, not the per-probe recording.

Additional Notes

Results (test-environment, median baseline → this PR). marginal = line − file overhead; file-level coverage is unchanged by this PR (control), so the marginal isolates the line-specific cost:

project	marginal overhead before → after
spring-boot	18.9 → 5.7 (−70%)
nebula-release-plugin	31.2 → 12.3 (−61%)
reactive-streams-jvm	9.8 → 6.4 (−35%)
okhttp	22.4 → 14.7 (−34%)
sonar-kotlin	4.1 → 3.2 (−22%)
netflix-zuul	16.5 → 15.1 (−8%)
jolokia	~0 → ~0 (no line-specific headroom)

Aggregate coverage preserved: total module/session coverage % and uploaded reports are unaffected — the OR-back keeps JaCoCo's shared array populated.
Testing: new JUnit 5 unit tests (LineProbes, ExecutionDataAdapter, LineCoverageStore) plus GradleDaemonSmokeTest (real JaCoCo, fixture bitmaps) are green.

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any other useful labels
Avoid using close, fix, or any linking keywords when referencing an issue
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, migration, or deletion
Update public documentation with any new configuration flags or behaviors
Add your completed PR to the merge queue by commenting /merge. You can also:
- Customize the commit message associated with the merge with /merge --commit-message "..."
- Remove your PR from the merge queue with /merge -c
- Skip all merge queue checks with /merge -f --reason "reason"; please use this judiciously, as some checks do not run at the PR-level (note: the PR still needs to be mergeable, this will only skip the pre-merge build)
- Get more information in this doc

Jira ticket: SDTEST-3847

datadog-official · 2026-06-12T10:19:29Z

🎯 Code Coverage (details)
• Patch Coverage: 45.61%
• Overall Coverage: 56.68% (-0.37%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 826244a | Docs | Datadog PR Page | Give us feedback!}

cit-pr-commenter-54b7da · 2026-06-12T10:34:35Z

Test Environment - netflix-zuul

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	86.82	87.80	$\color{green}{\blacktriangledown}$ -0.98	87.80	$\color{green}{\blacktriangledown}$ -0.98	39/123
agentless	80.00	81.05	$\color{green}{\blacktriangledown}$ -1.05	81.05	$\color{green}{\blacktriangledown}$ -1.05	39/122
agentlessCodeCoverage	96.57	97.04	$\color{green}{\blacktriangledown}$ -0.47	95.12	$\color{red}{\blacktriangle}$ +1.45	39/120
agentlessLineCoverage	110.26	111.62	$\color{green}{\blacktriangledown}$ -1.36	111.62	$\color{green}{\blacktriangledown}$ -1.36	38/119

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-06-12T10:35:12Z

Test Environment - nebula-release-plugin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	38.54	37.15	$\color{red}{\blacktriangle}$ +1.39	36.42	$\color{red}{\blacktriangle}$ +2.12	38/117
agentless	36.10	36.42	$\color{green}{\blacktriangledown}$ -0.32	36.42	$\color{green}{\blacktriangledown}$ -0.32	38/117
agentlessCodeCoverage	44.99	45.38	$\color{green}{\blacktriangledown}$ -0.39	44.48	$\color{red}{\blacktriangle}$ +0.51	38/117
agentlessLineCoverage	55.39	74.82	$\color{green}{\blacktriangledown}$ -19.43	74.82	$\color{green}{\blacktriangledown}$ -19.43	37/116

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-06-12T10:36:55Z

Test Environment - pass4s

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	10.57	9.35	$\color{red}{\blacktriangle}$ +1.22	10.33	$\color{red}{\blacktriangle}$ +0.24	38/117
agentless	11.73	10.13	$\color{red}{\blacktriangle}$ +1.60	10.13	$\color{red}{\blacktriangle}$ +1.60	38/117
agentlessCodeCoverage	18.35	17.03	$\color{red}{\blacktriangle}$ +1.32	17.38	$\color{red}{\blacktriangle}$ +0.97	37/114

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-06-12T10:43:32Z

Test Environment - sonar-kotlin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	13.64	12.87	$\color{red}{\blacktriangle}$ +0.77	13.13	$\color{red}{\blacktriangle}$ +0.51	34/116
agentless	11.15	12.12	$\color{green}{\blacktriangledown}$ -0.97	12.12	$\color{green}{\blacktriangledown}$ -0.97	34/115
agentlessCodeCoverage	14.42	15.11	$\color{green}{\blacktriangledown}$ -0.69	15.11	$\color{green}{\blacktriangledown}$ -0.69	35/115
agentlessLineCoverage	17.33	18.82	$\color{green}{\blacktriangledown}$ -1.49	19.20	$\color{green}{\blacktriangledown}$ -1.87	34/114

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-06-12T10:46:08Z

Test Environment - jolokia

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	95.26	95.12	$\color{red}{\blacktriangle}$ +0.14	93.23	$\color{red}{\blacktriangle}$ +2.03	40/126
agentless	90.86	89.58	$\color{red}{\blacktriangle}$ +1.28	89.58	$\color{red}{\blacktriangle}$ +1.28	39/122
agentlessCodeCoverage	100.52	99.00	$\color{red}{\blacktriangle}$ +1.52	99.00	$\color{red}{\blacktriangle}$ +1.52	41/121
agentlessLineCoverage	101.65	99.00	$\color{red}{\blacktriangle}$ +2.65	99.00	$\color{red}{\blacktriangle}$ +2.65	39/120

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-06-12T10:50:15Z

Test Environment - okhttp

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	19.00	19.20	$\color{green}{\blacktriangledown}$ -0.20	19.20	$\color{green}{\blacktriangledown}$ -0.20	40/128
agentless	20.38	18.82	$\color{red}{\blacktriangle}$ +1.56	19.20	$\color{red}{\blacktriangle}$ +1.18	38/125
agentlessCodeCoverage	20.73	22.54	$\color{green}{\blacktriangledown}$ -1.81	22.09	$\color{green}{\blacktriangledown}$ -1.36	38/123
agentlessLineCoverage	36.87	45.38	$\color{green}{\blacktriangledown}$ -8.51	44.48	$\color{green}{\blacktriangledown}$ -7.61	39/128

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-06-19T10:22:17Z

Test Environment - sbt-scalatest

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	56.95	55.43	$\color{red}{\blacktriangle}$ +1.52	54.33	$\color{red}{\blacktriangle}$ +2.62	76/238
agentEvpProxy	55.32	n/a	n/a	n/a	n/a	-

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-06-19T10:26:55Z

Test Environment - reactive-streams-jvm

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	20.65	21.65	$\color{green}{\blacktriangledown}$ -1.00	21.65	$\color{green}{\blacktriangledown}$ -1.00	39/124
agentless	17.77	18.82	$\color{green}{\blacktriangledown}$ -1.05	18.82	$\color{green}{\blacktriangledown}$ -1.05	37/121
agentlessCodeCoverage	19.89	20.39	$\color{green}{\blacktriangledown}$ -0.50	19.99	$\color{green}{\blacktriangledown}$ -0.10	38/120
agentlessLineCoverage	26.06	30.42	$\color{green}{\blacktriangledown}$ -4.36	29.82	$\color{green}{\blacktriangledown}$ -3.76	37/119

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-06-19T10:54:35Z

Test Environment - spring_boot

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	16.20	16.36	$\color{green}{\blacktriangledown}$ -0.16	16.04	$\color{red}{\blacktriangle}$ +0.16	37/115
agentless	9.92	9.73	$\color{red}{\blacktriangle}$ +0.19	9.73	$\color{red}{\blacktriangle}$ +0.19	37/116
agentlessCodeCoverage	13.94	13.67	$\color{red}{\blacktriangle}$ +0.27	13.40	$\color{red}{\blacktriangle}$ +0.54	38/115
agentlessLineCoverage	19.86	32.95	$\color{green}{\blacktriangledown}$ -13.09	32.95	$\color{green}{\blacktriangledown}$ -13.09	38/114

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-06-19T11:09:24Z

Test Environment - sonar-java

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	9.25	12.37	$\color{green}{\blacktriangledown}$ -3.12	13.94	$\color{green}{\blacktriangledown}$ -4.69	35/121
agentless	42.51	12.62	$\color{red}{\blacktriangle}$ +29.89	16.04	$\color{red}{\blacktriangle}$ +26.47	35/120
agentlessCodeCoverage	94.22	79.45	$\color{red}{\blacktriangle}$ +14.77	86.07	$\color{red}{\blacktriangle}$ +8.15	35/120
agentlessLineCoverage	130.48	125.85	$\color{red}{\blacktriangle}$ +4.63	139.09	$\color{green}{\blacktriangledown}$ -8.61	35/120

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

dd-octo-sts · 2026-07-13T12:27:08Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.73 s	14.63 s	[-0.2%; +1.4%] (no difference)
startup:insecure-bank:tracing:Agent	13.56 s	13.65 s	[-1.5%; +0.1%] (no difference)
startup:petclinic:appsec:Agent	16.73 s	16.60 s	[-0.5%; +2.1%] (no difference)
startup:petclinic:iast:Agent	16.72 s	16.87 s	[-1.9%; +0.0%] (no difference)
startup:petclinic:profiling:Agent	16.70 s	16.16 s	[-1.1%; +7.8%] (no difference)
startup:petclinic:sca:Agent	16.42 s	16.39 s	[-6.1%; +6.4%] (unstable)
startup:petclinic:tracing:Agent	16.06 s	15.75 s	[-2.4%; +6.3%] (no difference)

Commit: 826244aa · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

…-array swap Swap Jacoco's shared probe array for a per-test array once at method entry, so Jacoco's native probes record per-test coverage with no per-probe callback. Aggregate coverage is preserved by OR-ing per-test bits back into the shared array at report time. Excludes the CoverageProbes interface (whose only bytecode is the new fallback default) from the coverage check, consistent with the rest of the coverage-infra classes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Jacoco's Analyzer re-parsed each covered class once per test; cache the covered lines per (class id, probe set) so a class covered identically by many tests is analyzed only once. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

cit-pr-commenter-54b7da · 2026-07-13T14:17:38Z

Test Environment - heliboard

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	11.99	9.54	$\color{red}{\blacktriangle}$ +2.45	9.54	$\color{red}{\blacktriangle}$ +2.45	36/36

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

daniel-mohedano · 2026-07-15T10:01:45Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 826244aaef

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-15T10:16:06Z

+      // Back-fill Jacoco's aggregate coverage (total coverage percentage and report uploads). Skip
+      // classes with no covered probes: the per-test array is allocated on method entry, so a
+      // method that throws before its first probe fires would otherwise yield an empty entry.
+      if (!executionDataAdapter.mergeIntoJacocoProbes()) {


Backfill skipped tests before suppressing reports

When a test is marked skipped after executing code (for example a JUnit assumption/abort after setup or part of the test body), TestImpl.end skips coverageStore.report(...) for TestStatus.skip at dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/TestImpl.java:281-284. Since this new path makes report() the only caller that ORs the swapped per-test probes back into JaCoCo's shared array, those executed probes are dropped from aggregate coverage percentages and report uploads, whereas the previous per-probe instrumentation updated JaCoCo's shared array immediately.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-15T10:16:06Z

+          "resolveProbeArray",
+          "([ZLjava/lang/Class;J)[Z",
          false);
+      methodVisitor.visitVarInsn(Opcodes.ASTORE, variable);


Avoid stranding probes after a method outlives its test

When an instrumented method is entered while a test is active but continues on a background thread past TestImpl.end, this stores the per-test array in the method-local probe slot for the rest of that invocation. Any JaCoCo probe writes that happen after coverageStore.report(...) has already merged the array will keep writing to that stale per-test array, so they are never reflected in JaCoCo's shared aggregate data used for coverage percentages and report uploads; the previous per-probe bridge re-resolved the current store at each probe and could fall back to the shared array after the test ended.

Useful? React with 👍 / 👎.

daniel-mohedano · 2026-07-20T12:37:17Z

closing for now in favor of #11969

daniel-mohedano added type: feature Enhancements and improvements comp: ci visibility Continuous Integration Visibility tag: ai generated Largely based on code generated by an AI or LLM labels Jun 12, 2026

daniel-mohedano changed the title ~~feat: performance improvements to line level coverage~~ Reduce runtime overhead of line-level per-test coverage Jun 24, 2026

daniel-mohedano and others added 2 commits July 13, 2026 15:57

daniel-mohedano force-pushed the daniel.mohedano/improve-line-level-granularity-performance branch from 2ce5fdf to 826244a Compare July 13, 2026 13:59

chatgpt-codex-connector Bot reviewed Jul 15, 2026

View reviewed changes

daniel-mohedano closed this Jul 20, 2026

Uh oh!

Conversation

daniel-mohedano commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

datadog-official Bot commented Jun 12, 2026 • edited by datadog-datadog-prod-us1-2 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cit-pr-commenter-54b7da Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - netflix-zuul

Uh oh!

cit-pr-commenter-54b7da Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - nebula-release-plugin

Uh oh!

cit-pr-commenter-54b7da Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - pass4s

Uh oh!

cit-pr-commenter-54b7da Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - sonar-kotlin

Uh oh!

cit-pr-commenter-54b7da Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - jolokia

Uh oh!

cit-pr-commenter-54b7da Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - okhttp

Uh oh!

cit-pr-commenter-54b7da Bot commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - sbt-scalatest

Uh oh!

cit-pr-commenter-54b7da Bot commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - reactive-streams-jvm

Uh oh!

cit-pr-commenter-54b7da Bot commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - spring_boot

Uh oh!

cit-pr-commenter-54b7da Bot commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - sonar-java

Uh oh!

dd-octo-sts Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 13, 2026

Test Environment - heliboard

Uh oh!

daniel-mohedano commented Jul 15, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

daniel-mohedano commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

daniel-mohedano commented Jun 12, 2026 •

edited

Loading

datadog-official Bot commented Jun 12, 2026 •

edited by datadog-datadog-prod-us1-2 Bot

Loading

cit-pr-commenter-54b7da Bot commented Jun 12, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jun 12, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jun 12, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jun 12, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jun 12, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jun 12, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jun 19, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jun 19, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jun 19, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jun 19, 2026 •

edited

Loading

dd-octo-sts Bot commented Jul 13, 2026 •

edited

Loading