Instrument workflow replay scaling

[NathanColosimo]

Summary

• split workflow.run into VM context, bundle evaluation, argument hydration, execution, serialization, and drain spans
• aggregate replay event-consumer, subscription-wait, step hydration, and cache diagnostics without per-step spans
• split step-result hydration into decrypt, decompress, compression telemetry, and deserialize timings
• add an isolated 20/120/1020-step replay benchmark and route PR benchmark CI through the protected workflow-server preview

Production finding

The production VM itself is not intrinsically hundreds of milliseconds slower than local replay. The original comparison mixed a tiny synthetic local bundle with a 1.322 MB production bundle, encrypted object results, OpenTelemetry, and DEBUG=workflow:*.

Disabling debug logging on the same preview reduced late workflow.run from 604.6 ms to 247.6 ms. The synchronous event-consumer scan fell from 165.3 ms to 11.9 ms because each enabled per-event debug call emitted console output and an active-span event.

The final instrumented, debug-off, non-turbo run was CI 29548488910, workflow run wrun_41KXPWKTB70GJFGJB4Q00XBH2W.

Mean (ms)	K1-K19	K101-K119	K1001-K1019
STSO	232.0	355.3	503.5
prior step_completed durable write	70.9	78.5	71.1
event refresh	0.04	102.8	106.4
workflow.run	72.9	90.8	242.0
next step_started durable write	83.0	79.2	79.3

The refresh cliff is the event pagination issue being handled separately in workflow-server PR #632. It is not the continuous replay slope.

Replay breakdown

Mean (ms)	K1-K19	K101-K119	K1001-K1019
VM context	1.00	0.96	1.08
evaluate 1.322 MB bundle	49.17	44.98	51.45
workflow execute/replay	16.82	42.48	187.21
synchronous event scan	0.22	1.44	11.54
step-result hydration	14.31	36.31	144.70
AES-GCM decrypt	13.57	30.71	101.85
decompress	0.07	0.63	3.68
compression telemetry	0.23	1.55	14.51
deserialize	0.35	2.65	17.73
events / callback invocations	32 / 75	332 / 775	3032 / 7075

The subscription-wait metric is 18.8 / 39.3 / 160.8 ms, but it overlaps hydration and VM continuation and must not be added to the rows above.

Every benchmark result is an object, so all 10 / 110 / 1010 historical step values are non-memoizable by the primitive-only hydration cache. Each replay serially performs one WebCrypto AES-GCM decrypt and one fresh deserialize per historical result. Across 1020 serial steps that is approximately sum(1..1020) = 520,710 decrypt/parse operations in one invocation.

The exact local reproduction used the generated 1.322 MB bundle, encrypted object results, Node 22 Linux, and a 1-vCPU/2-GB container. At K1020 it measured 183.7 ms total workflow.run, including 50.6 ms bundle evaluation, 124.5 ms execution, 86.0 ms hydration, and 9.1 ms synchronous event scan. The preview measured 242.0 / 51.5 / 187.2 / 144.7 / 11.5 ms respectively. The remaining local-cloud gap is therefore hydration/WebCrypto throughput and scheduling, not node:vm context or bundle evaluation.

Evidence

• original debug-on benchmark: CI 29544762077, traces d6ec13404e2537410dac512a521da423 and fec6da2c5114eb50e1eb9a9c31a23902
• debug-off A/B benchmark: CI 29547654943
• hydration split benchmark: CI 29548488910, traces 0d3358d44cc286fec90623114cce3d92 and 80879ec66b238c5727fb3f981d164ea3

Verification

• targeted replay telemetry and serialization tests: 4 passed
• pnpm --filter @workflow/core typecheck
• pnpm --filter @workflow/core build

[changeset-bot]

🦋 Changeset detected

Latest commit: 21397aa

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages

Name	Type
@workflow/core	Patch
@workflow/builders	Patch
@workflow/cli	Patch
@workflow/next	Patch
@workflow/nitro	Patch
@workflow/vitest	Patch
@workflow/web-shared	Patch
@workflow/web	Patch
workflow	Patch
@workflow/world-testing	Patch
@workflow/astro	Patch
@workflow/nest	Patch
@workflow/nuxt	Patch
@workflow/rollup	Patch
@workflow/sveltekit	Patch
@workflow/vite	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[github-actions]

🧪 E2E Test Results

✅ All tests passed

Summary

	Passed	Failed	Skipped	Total
✅ ▲ Vercel Production	1453	0	230	1683
✅ 💻 Local Development	1617	0	219	1836
✅ 📦 Local Production	1617	0	219	1836
✅ 🐘 Local Postgres	1617	0	219	1836
✅ 🪟 Windows	153	0	0	153
✅ 📋 Other	894	0	177	1071
✅ vercel-multi-region	27	0	0	27
Total	7378	0	1064	8442

Details by Category

✅ ▲ Vercel Production

App	Passed	Failed	Skipped
✅ astro	126	0	27
✅ example	126	0	27
✅ express	126	0	27
✅ fastify	126	0	27
✅ hono	126	0	27
✅ nextjs-turbopack	150	0	3
✅ nextjs-webpack	150	0	3
✅ nitro	126	0	27
✅ nuxt	126	0	27
✅ sveltekit	145	0	8
✅ vite	126	0	27

✅ 💻 Local Development

App	Passed	Failed	Skipped
✅ astro-stable	128	0	25
✅ express-stable	128	0	25
✅ fastify-stable	128	0	25
✅ hono-stable	128	0	25
✅ nextjs-turbopack-canary	134	0	19
✅ nextjs-turbopack-stable	153	0	0
✅ nextjs-webpack-canary	134	0	19
✅ nextjs-webpack-stable	153	0	0
✅ nitro-stable	128	0	25
✅ nuxt-stable	128	0	25
✅ sveltekit-stable	147	0	6
✅ vite-stable	128	0	25

✅ 📦 Local Production

App	Passed	Failed	Skipped
✅ astro-stable	128	0	25
✅ express-stable	128	0	25
✅ fastify-stable	128	0	25
✅ hono-stable	128	0	25
✅ nextjs-turbopack-canary	134	0	19
✅ nextjs-turbopack-stable	153	0	0
✅ nextjs-webpack-canary	134	0	19
✅ nextjs-webpack-stable	153	0	0
✅ nitro-stable	128	0	25
✅ nuxt-stable	128	0	25
✅ sveltekit-stable	147	0	6
✅ vite-stable	128	0	25

✅ 🐘 Local Postgres

App	Passed	Failed	Skipped
✅ astro-stable	128	0	25
✅ express-stable	128	0	25
✅ fastify-stable	128	0	25
✅ hono-stable	128	0	25
✅ nextjs-turbopack-canary	134	0	19
✅ nextjs-turbopack-stable	153	0	0
✅ nextjs-webpack-canary	134	0	19
✅ nextjs-webpack-stable	153	0	0
✅ nitro-stable	128	0	25
✅ nuxt-stable	128	0	25
✅ sveltekit-stable	147	0	6
✅ vite-stable	128	0	25

✅ 🪟 Windows

App	Passed	Failed	Skipped
✅ nextjs-turbopack	153	0	0

✅ 📋 Other

App	Passed	Failed	Skipped
✅ e2e-local-dev-nest-stable	128	0	25
✅ e2e-local-dev-tanstack-start-	128	0	25
✅ e2e-local-postgres-nest-stable	128	0	25
✅ e2e-local-postgres-tanstack-start-	128	0	25
✅ e2e-local-prod-nest-stable	128	0	25
✅ e2e-local-prod-tanstack-start-	128	0	25
✅ e2e-vercel-prod-tanstack-start	126	0	27

✅ vercel-multi-region

App	Passed	Failed	Skipped
✅ nextjs-turbopack	27	0	0

---

📋 View full workflow run

[github-actions]

📊 Workflow Benchmarks

⏳ Benchmarks are running for 21397aa... (run logs)

Results below are from a previous run.

commit 81a3155 · Fri, 17 Jul 2026 02:04:18 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric	Scenario	Avg (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	stream	843 (-22%)	834 🔴	1036 🔴	1887 🔴	30
TTFS	hook + stream	776 (-46%)	801 🔴	906 🔴	1044 🔴	30
STSO	1020 steps (1-20)	232 (-23%)	239 🔴	318 🔴	482 🔴	19
STSO	1020 steps (101-120)	355 (-18%)	392 🔴	462 🔴	476 🔴	19
STSO	1020 steps (1001-1020)	504 (-41%)	515 🔴	591 🔴	808 🔴	19
WO	stream	843 (-22%)	834	1036	1887	30
WO	hook + stream	776 (-46%)	801	906	1044	30
SL	stream	1718 (-63%)	1788 🔴	1877 🔴	5819 🔴	30
SL	hook + stream	1615 (-67%)	1670 🔴	1722 🔴	1874 🔴	30

📜 Previous results (3)

0ed0294

Fri, 17 Jul 2026 01:44:11 GMT · run logs

vercel / nextjs-turbopack

Metric	Scenario	Avg (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	stream	842 (-23%)	744 🔴	798 🔴	4537 🔴	30
TTFS	hook + stream	773 (-46%)	833 🔴	850 🔴	1904 🔴	30
STSO	1020 steps (1-20)	259 (-14%)	259 🔴	414 🔴	591 🔴	19
STSO	1020 steps (101-120)	438 (+1.2%)	496 🔴	614 🔴	642 🔴	19
STSO	1020 steps (1001-1020)	633 (-26%)	550 🔴	577 🔴	2949 🔴	19
WO	stream	842 (-23%)	744	798	4537	30
WO	hook + stream	773 (-46%)	833	850	1904	30
SL	stream	1588 (-66%)	1625 🔴	1743 🔴	2798 🔴	30
SL	hook + stream	1589 (-67%)	1560 🔴	1686 🔴	5515 🔴	30

b0ded7f

Fri, 17 Jul 2026 00:47:52 GMT · run logs

vercel / nextjs-turbopack

Metric	Scenario	Avg (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	stream	1499 (+38%)	1890 🔴	1925 🔴	2052 🔴	30
TTFS	hook + stream	1468 (+2.7%)	1988 🔴	2083 🔴	2225 🔴	30
STSO	1020 steps (1-20)	301 (±0%)	326 🔴	351 🔴	510 🔴	19
STSO	1020 steps (101-120)	441 (+1.8%)	496 🔴	549 🔴	590 🔴	19
STSO	1020 steps (1001-1020)	887 (+3.5%)	926 🔴	1030 🔴	1052 🔴	19
WO	stream	1499 (+38%)	1890	1925	2052	30
WO	hook + stream	1468 (+2.7%)	1988	2083	2225	30
SL	stream	4955 (+5.8%)	5589 🔴	5679 🔴	5856 🔴	30
SL	hook + stream	5072 (+4.5%)	5641 🔴	5750 🔴	5857 🔴	30

5f7515a

Fri, 17 Jul 2026 00:18:52 GMT · run logs

vercel / nextjs-turbopack

Metric	Scenario	Avg (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	stream	1191 (+9.6%)	1662 🔴	1710 🔴	1775 🔴	30
TTFS	hook + stream	1630 (+14%)	1878 🔴	1970 🔴	2928 🔴	30
STSO	1020 steps (1-20)	283 (-6.1%)	331 🔴	398 🔴	418 🔴	19
STSO	1020 steps (101-120)	433 (±0%)	473 🔴	564 🔴	605 🔴	19
STSO	1020 steps (1001-1020)	943 (+10%)	1007 🔴	1105 🔴	1318 🔴	19
WO	stream	1191 (+9.6%)	1662	1710	1775	30
WO	hook + stream	1630 (+14%)	1878	1970	2928	30
SL	stream	4605 (-1.6%)	5569 🔴	5617 🔴	5920 🔴	30
SL	hook + stream	4779 (-1.5%)	5063 🔴	5814 🔴	6828 🔴	30

_{Avg deltas compare against the most recent benchmark run on main at the time of this run.}

_{Metrics — TTFS: time to first step body execution · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (time outside step bodies, client start → last step body exit) · SL: stream latency (first chunk write → visible to the reader)}

_{Scenarios — stream: one step that streams chunks back to the client; no hooks, so the run stays in turbo mode · hook + stream: registers a hook before the same streaming step, which exits turbo mode · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges}

_{🟢/🔴 mark percentiles within/above target. Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120}

_{TTFS/WO compare client vs deployment clocks and SL compares the step runner’s clock vs the client’s (NTP-synced in CI). WO ends at the last step body exit, the closest observable proxy for the final step-completion request.}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
example-nextjs-workflow-turbopack	Ready	Preview, Comment	Jul 17, 2026 2:12am
example-nextjs-workflow-webpack	Ready	Preview, Comment	Jul 17, 2026 2:12am
example-workflow	Ready	Preview, Comment	Jul 17, 2026 2:12am
workbench-astro-workflow	Ready	Preview, Comment	Jul 17, 2026 2:12am
workbench-express-workflow	Ready	Preview, Comment	Jul 17, 2026 2:12am
workbench-fastify-workflow	Ready	Preview, Comment	Jul 17, 2026 2:12am
workbench-hono-workflow	Ready	Preview, Comment	Jul 17, 2026 2:12am
workbench-nitro-workflow	Ready	Preview, Comment	Jul 17, 2026 2:12am
workbench-nuxt-workflow	Ready	Preview, Comment	Jul 17, 2026 2:12am
workbench-sveltekit-workflow	Ready	Preview, Comment	Jul 17, 2026 2:12am
workbench-tanstack-start-workflow	Ready	Preview, Comment	Jul 17, 2026 2:12am
workbench-vite-workflow	Ready	Preview, Comment	Jul 17, 2026 2:12am
workflow-swc-playground	Ready	Preview, Comment	Jul 17, 2026 2:12am
workflow-tarballs	Ready	Preview, Comment	Jul 17, 2026 2:12am
workflow-web	Ready	Preview, Comment	Jul 17, 2026 2:12am

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
workflow-docs	Skipped		Jul 17, 2026 2:12am

[NathanColosimo]

The benchmark triggered by the final preview-config restore commit was intentionally canceled because it would no longer run with the debug-off/non-turbo experimental controls. The latest valid diagnostic result is commit 81a3155, run 29548488910; the PR body now contains its full trace breakdown. The final branch has restored the normal preview config.