report: audit and validate MongoDB optimizations

[carsontung666]

What

• keep bench/db/report/report.tex / report.pdf as the canonical database report
• audit the MongoDB recommendations against their exact artifacts, implementation status, and adoption gates
• separate MongoDB 7.0.34 storage-harness evidence from a narrow pinned-master source experiment
• publish the final preregistered source-experiment bundle: raw process JSON, logs, analysis, exact patches, binary identities, validation reports, review summaries, and SHA-256 manifest
• explicitly supersede the earlier 10-pair CountScan bundle without modifying it
• keep optimization.tex as superseded working material, not a second final report

Review decisions

• short-read coalescing: promising in the storage harness, not integrated into a MongoDB ConDB path; snapshot/visibility, grouped failure semantics, ordering, tenant isolation, command limits, and plan-cache behavior remain gates
• cursor batch-size override: reject for the measured fully materialized covered-Metadata path
• 256-row DFS buckets: promising only for the measured large/static/single-tree cohort; routing, mutation, publication, and production storage costs remain open
• direct non-deduplicating classic CountStage -> CountScan: narrow source candidate on a pinned master snapshot
• SQLite Beam batching: implemented cross-engine control, not evidence of MongoDB production improvement

No MongoDB optimization in the report is currently integrated into ConDB's public storage path.

MongoDB source candidate

• pinned upstream snapshot: 5d3b36cf3871846fe7894616e964cb520c11d473
• candidate commit: 696f0d5d30f9
• fork branch: carsontung666/mongo:agent/condb-query-hotpath
• draft review: carsontung666/mongo#1

The candidate activates a private resultless protocol only for a direct CountStage -> CountScan edge with !_shouldDedup. Public CountScan::work() retains valid RID_AND_OBJ output. Multikey and scalar compound-wildcard scans continue to materialize and deduplicate. Both entry points share PlanStage::trackWork() accounting.

The report claims only that the direct path avoids per-match WorkingSet-member materialization/lifecycle; it does not claim a zero-allocation query.

Historical redundancy audit

The candidate was checked against MongoDB PR #635, PR #1369, and ancestor commits d71566a55e, dac2f722f8, d8ee635331, 09b89f0986, and 8f52dfc863. Those cover count planning/bounds, shard-filter elision, the valid public WorkingSet contract, cheaper RID_AND_OBJ output, centralized work accounting, and compound-wildcard deduplication.

No equivalent private direct resultless handoff was found in the inspected GitHub PRs or pinned-snapshot ancestry. This is not an exhaustive claim about private Jira work or unpublished branches.

Frozen 400K activation A/B

This is an activation ablation within the final candidate, not a whole-patch comparison against pristine upstream. Both arms contain the shared implementation and benchmark harness; the control removes only the six-line activation block. A pristine-base arm with a neutral harness remains an adoption gate.

The protocol was committed and pushed as 00fd8de before execution:

• 400,000-key warmed, single-thread, in-process scalar count
• exact result and strict COUNT -> COUNT_SCAN preflight
• whole process pinned to CPU 0, performance governor
• 20 fresh process pairs, frozen 10 control-first / 10 candidate-first order
• five technical repetitions per process; inference uses 20 complete pairs
• 100,000-draw order-stratified complete-pair bootstrap with fixed seed

Results:

• benchmark-thread user-space retired instructions: 5.447% reduction, 95% CI [5.445%, 5.448%]
• benchmark-thread CPU time: 6.981% reduction, 95% CI [5.425%, 8.616%]
• wall time (auxiliary): 6.986% reduction, 95% CI [5.428%, 8.622%]

All 20 pairs favor activation on all three metrics; both order strata and all leave-one-pair-out estimates retain the direction.

Validation

• optimized five-target Bazel build: mongod, mongos, mongo, dbtest, count_query_bm
• explicit runtime-dassert dbtest build
• query_stage_count_scan and query_stage_count pass under both builds
• unit checks cover public 0-to-1 backing-slot growth; direct skip/limit/yield with zero backing slots; multikey and scalar compound-wildcard materialization
• forced-classic resmoke:
  • core: 6 JS files / 32 passing report entries
  • aggregation: 2 / 12
  • no_passthrough: 1 / 3
  • sharding: 1 / 3
• latexmk -pdf -interaction=nonstopmode -halt-on-error report.tex: 10-page PDF, no warnings, overflow, unresolved references, or errors
• every new SHA256SUMS entry passes and analyze.py reproduces summary.json byte-for-byte
• independent implementation, history, method, evidence, report, and release-scope reviews found no P0–P3 issue

Report-entry counts include fixture and hook events. These checks are targeted, not a full MongoDB Evergreen or production qualification.

Evidence

Canonical bundle:

bench/db/report/evidence/mongodb_master_countscan_20260805_696f0d5d30f9/

• candidate patch SHA-256: 187c3c5768b66c33f43533b7903d572bc96179ae9160181e4275e00987d9348b
• summary SHA-256: 420e7a6c148b2a2339984012cbbc28a344486f90a3328bcf2fb83f20248d4739
• manifest SHA-256: 5a907c3a98b0b075fc2004d9269187e565d4e0394a495dc111c7b6f79e0201e3

The older mongodb_master_countscan_20260805/ campaign is retained unchanged and explicitly marked superseded. Large storage-harness raw JSON under bench/db/runs remains gitignored and still needs separate publication for full external reproduction.

Scope

The source result covers one pinned host and one warmed in-process scalar-count shape. It does not establish improvement for stock upstream, MongoDB 7.0.34, SBE, indirect or deduplicating plans, server-process/remote-client operation, cold or cache-constrained data, concurrency, any timed ConDB endpoint, or production latency.