@W-23545283 native lib benchmarks

[andres-rad]

What

Adds a language-agnostic benchmark harness for the DataWeave runtime under a new top-level benchmarks/ directory. It runs a shared corpus of DataWeave scripts through three runners — the Node and Python native-lib wrappers plus a JVM engine baseline — each emitting the same JSON schema, and renders a cross-runner comparison. It measures four metrics: cold-start, first-run (compile+exec), warm steady-state, and streaming throughput.

W-23545283.

How it works

• benchmarks/corpus/ — the shared contract every runner consumes: manifest.json (each case has an immutable id, a .dwl script, optional inputs, and an explicit metrics[]), 10 scripts, committed inputs, and a deterministic large-input generator.
• benchmarks/schema/result.schema.json — the JSON schema all runners' output conforms to (env carries weaveVersion/commit/dwlibBuildId for attribution).
• benchmarks/lib/ — dependency-free JS helpers: stats (percentiles + MB/s), manifest (load/validate + fail-fast orphan-id check), env (env stamp). Reference implementation the Python/engine runners claim parity against (parity pinned by always-on tests).
• Runners (benchmarks/runners/):
  • node/ — the reference runner: in-process warm/streaming (warm-bench), a fresh-process spawn harness for cold-start/first-run (coldstart), and emit (orchestrate → validate ids → write result JSON).
  • python/ — stdlib runner mirroring Node file-for-file over the same staged dwlib; bit-exact stats parity, no third-party deps.
  • engine/ — JVM baseline (:benchmarks-engine Scala/Gradle subproject) driving the bare DataWeaveScriptingEngine at the same weaveVersion the native image is built from, so the node-vs-engine comparison isolates the native-image-vs-JVM axis. Includes a JIT-warmup floor.
• benchmarks/report/report.mjs — joins result files by case id, renders a case×metric table with one Δ-vs-baseline column per non-baseline runner (so every runner is represented, not just the first) and a weave-version skew banner; dedupes to the latest result per runner. --emit is a reserved exporter seam.
• Gradle — benchmarkCompare (root) auto-discovers every runner tagged ext.benchmarkRunner, runs them, and renders the report once. Per-runner tasks (native-lib:benchmark/benchmarkNode, native-lib:benchmarkPython, benchmarks-engine:benchmarkEngine) also exist. All opt-in via -Pbenchmark=true, off by default (never in the normal build/test path). Stats/manifest/env parity tests are always-on so cross-runner drift fails the normal build.

Design decisions

• Cold-start is measured by the parent, not the child: each runner spawns a fresh child that flushes a READY marker the instant its runtime is initialized, and the parent records wall-clock from just-before-spawn to that marker. So cold-start includes process launch + library/class load + runtime init on all three runners — which is what makes the native-image-vs-JVM comparison meaningful (the native image has no JVM to boot; the JVM's cold cost is launch + classload). first-run stays measured in-process in the same spawn.
• Timing is a self-contained nanosecond sampling harness (hrtime.bigint() / perf_counter_ns / System.nanoTime()), not a bench library, so all three surfaces share the exact methodology for a fair comparison.
• Corpus covers dimensions the DataWeave engine's own JMH suite exercises: JSON/XML/CSV reads, object transforms, scaled map, groupBy, deep value selectors, a streaming case, and a compile-heavy case.
• Results are local-only/gitignored; durable history (Grafana/exporter) is a deferred follow-up, with env.commit/dwlibBuildId + immutable id + the --emit seam in place so it's a bolt-on.

Testing

• JS harness tests pass (node --test across benchmarks/**/*.test.mjs), including integration tests that drive the real native library; Python parity suite (17 tests) and engine scalatest suite (25 tests) pass.
• Verified the full path end-to-end on GraalVM: Node and Python emit each produce 23 schema-conformant rows over the 10-case corpus against the real dwlib; engine EngineChildTest/EmitTest drive fresh JVM spawns. Numbers are reproducible run-over-run.

Sample results

One local run, all three runners on the same weaveVersion 2.12.0-20260413 (no skew), baseline = engine. Indicative only — single dev machine (Apple M4 Max; node v22, python 3.9, JVM 21), single run, not a dedicated bench box. Each non-baseline runner gets its own signed Δ-vs-engine column, per unit (ms lower-is-better, MB/s higher-is-better).

📊 benchmarks/report/RESULTS.md renders this same run as bar charts grouped by corpus case (one bar per runner, Mermaid, inline on GitHub), stamped with the commit + run date. Regenerate with node report/report.mjs results/*.json --markdown report/RESULTS.md.

case	metric	unit	engine	node-wrapper	python-wrapper	Δ node-wrapper vs engine	Δ python-wrapper vs engine
trivial	cold-start	ms	175.48	38.99	39.79	-77.8%	-77.3%
trivial	first-run	ms	347.90	8.65	8.93	-97.5%	-97.4%
trivial	warm	ms	0.11	0.09	0.08	-23.2%	-28.4%
object-transform	first-run	ms	656.29	16.49	17.65	-97.5%	-97.3%
object-transform	warm	ms	0.34	0.24	0.22	-30.4%	-36.7%
map-scale	first-run	ms	785.94	141.34	144.58	-82.0%	-81.6%
map-scale	warm	ms	33.77	115.52	117.89	+242.1%	+249.1%
map-scale	streaming	MB/s	56.23	29.39	30.17	-47.7%	-46.4%
xml-to-csv	first-run	ms	356.13	8.62	9.01	-97.6%	-97.5%
xml-to-csv	warm	ms	0.09	0.10	0.10	+16.5%	+8.8%
json-stream	first-run	ms	743.44	125.23	126.95	-83.2%	-82.9%
json-stream	warm	ms	25.87	99.79	101.11	+285.8%	+290.9%
json-stream	streaming	MB/s	75.43	37.08	38.31	-50.8%	-49.2%
compile-heavy	first-run	ms	701.18	17.36	17.84	-97.5%	-97.5%
compile-heavy	warm	ms	0.87	0.85	0.84	-2.1%	-3.1%
csv-to-json	first-run	ms	669.11	16.35	17.12	-97.6%	-97.4%
csv-to-json	warm	ms	0.12	0.17	0.17	+44.9%	+41.0%
xml-to-json	first-run	ms	693.56	16.70	16.95	-97.6%	-97.6%
xml-to-json	warm	ms	0.13	0.18	0.17	+42.2%	+37.1%
deep-selector	first-run	ms	348.25	8.61	8.93	-97.5%	-97.4%
deep-selector	warm	ms	0.07	0.12	0.10	+82.7%	+53.9%
group-by	first-run	ms	919.06	196.34	199.28	-78.6%	-78.3%
group-by	warm	ms	57.42	168.43	170.83	+193.3%	+197.5%

Reading it:

• Cold-start / first-run — the native wrappers win decisively (−77% to −98%): no JVM boot, no classload, no cold JIT compile. This is the headline the harness exists to show, and Node ≈ Python throughout.
• Warm steady-state — mixed. On small transforms the wrappers roughly tie or slightly beat the engine. On the large-input cases (map-scale, json-stream, group-by) the JVM engine wins substantially, for two reasons worth noting rather than hiding: (1) the wrappers re-marshal the full input (base64 over the ~1 MB+ buffer) across the FFI inside the timed region every iteration, and (2) after a 2000-iteration warmup the engine's C2 JIT is fully hot. Treat these large-input warm deltas as "wrapper end-to-end call cost" not "runtime exec speed".
• Streaming (MB/s) — now a like-for-like comparison across all three runners: the streaming methodology was aligned (chunked input + deferred output), so the engine's throughput is measured the same way as the wrappers' incremental runTransform, and the Δ is meaningful. The engine leads here (~−47% to −51%), reflecting native-image FFI overhead on the chunked path.

Prerequisites

The Node/Python runners (and benchmarkCompare) benchmark the staged native dwlib, so they require the repo's GraalVM toolchain (Java 21+, graalvm-community, with native-image); the engine runner alone runs on any JDK. Documented in benchmarks/README.md, with a Benchmarks section added to the root README.

Out of scope (follow-ups)

• Durable results datastore / dashboard (the --emit seam is in place).
• Warm-metric compile-cache symmetry between engine and wrappers (whether the engine warm loop is compile-cached vs. the wrappers recompiling per call) — noted in review, deferred.

Design specs and implementation plans are included under docs/superpowers/.