feat(core): opt-in deferred step_completed write (WORKFLOW_ASYNC_STEP_COMPLETED)#2972
Draft
VaguelySerious wants to merge 8 commits into
Draft
feat(core): opt-in deferred step_completed write (WORKFLOW_ASYNC_STEP_COMPLETED)#2972VaguelySerious wants to merge 8 commits into
VaguelySerious wants to merge 8 commits into
Conversation
…imistic start under sequential replays Two relaxations of conservative gates in the inline replay loop, each tied to the mechanism that makes it safe: 1. The inline-delta fast path (skip one events.list per sequential step) no longer turns off for runs with an open hook when the precondition guard (WORKFLOW_PRECONDITION_GUARD=1) is enabled. A hook_received landing in the delta window is the same read-to-write race the fetch path already has, and with the guard on it is fenced: the marker bump 412s the stale replay's guarded creates, which retry over the reloaded log or exhaust into a fresh-replay re-invocation. Open waits keep the conservative gate (wait_completed does not bump the outside-event marker). 2. Turbo keeps forcing optimistic inline start after the run creates a hook or wait when WORKFLOW_SEQUENTIAL_REPLAYS=1: per-run maxConcurrency: 1 flow topics serialize the resume invocations hooks/waits introduce, so no concurrent orchestrator replay can race the optimistic create-claim, restoring the single-handler guarantee turbo relies on. Adds isSequentialReplaysEnabled() to the core runtime (mirroring the @workflow/builders / @workflow/world-vercel copies) and splits hasOpenHookOrWait into per-kind state so the two gates can differ. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: f7740b9 The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
Contributor
🧪 E2E Test Results✅ All tests passed Summary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
✅ vercel-multi-region
❌ Some E2E test jobs failed:
Check the workflow run for details. |
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Additional Suggestion:
In the WORKFLOW_ASYNC_STEP_COMPLETED opt-in path, the non-suspension failure branch acks the queue message via the ReplayDivergence-recovery re-queue and the run_failed terminal write without first settling the pending deferred step_completed write, permanently failing runs on speculative, already-superseded state.
Member
Author
|
Note on the red |
…; retract the sequential-replays turbo waiver Second review round on #2970: - Guard-enforced batches with an open hook (or hooks created by the same suspension) now take the await-then-run path even when optimistic inline start is enabled: the step_started claim carries the snapshot and is awaited before user code runs, so a claim the backend rejects as stale (412) never executes the step body — the fence covers side effects, not just durable writes. - Retract the WORKFLOW_SEQUENTIAL_REPLAYS waiver of turbo's hook/wait latch: the env var cannot prove the built flow trigger carries maxConcurrency: 1 (it must also be set at build time, and some integrations write their own trigger config), and capabilities.maxConcurrency only declares queue support. The conservative latch stays until the build emits a verifiable serialization signal; the capability remains declared for that follow-up. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…_COMPLETED) When enabled, a single sequential inline step's terminal write is initiated but not awaited: the next replay iteration runs on locally-synthesized events for that step, and (with optimistic start active) the next step's body starts while the write settles in the background. Every subsequent durable write chains behind the deferred write (fail-closed), and every ack path settles it, so the event log can never record downstream events without their upstream terminal and the queue message is never acknowledged with the write in flight. Gated on: no open hook or wait, single clean sequential inline step, and (turbo first delivery || WORKFLOW_SEQUENTIAL_REPLAYS=1), plus optimistic inline start being active. Off by default. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-replays arm of the deferred-completed gate Carries the capability gating from the base branch's review follow-up into the WORKFLOW_ASYNC_STEP_COMPLETED gate and docs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ension failure branch acks the queue message via the ReplayDivergence-recovery re-queue and the `run_failed` terminal write without first settling the pending deferred `step_completed` write, permanently failing runs on speculative, already-superseded state.
This commit fixes the issue reported at packages/core/src/runtime.ts:2822
## The bug
In `packages/core/src/runtime.ts`, the deferred `step_completed` write feature (opt-in via `WORKFLOW_ASYNC_STEP_COMPLETED=1`) establishes an invariant, stated at the `deferredCompleted` declaration (lines ~581-594):
> every ack path settles it (fail-closed) so the queue message is never acknowledged while the terminal write is in flight or failed.
The verified settle-before-ack sites honor it: `reinvoke` (line 756), replay-budget exhausted (1195), NO_INLINE_REPLAY timeout (1218), `run_completed` terminal write (1574), suspension return (2101).
The **non-suspension `else` branch** of the replay `catch` (starting ~line 2677) has two ack paths that did **not** settle first:
1. **ReplayDivergence recovery** — `await queueMessage(...); return;` re-queues a recovery replay (acks the current message).
2. **`run_failed` terminal write** — `await awaitRunReady(); await world.events.create(runId, { eventType: 'run_failed', ... }); return;` (acks the message).
Only the two branches above them settle/avoid-ack correctly: `PreconditionFailedError.is(err)` → `reinvoke(0)` (settles), and `isRetryableWorldError(err)` → `throw err` (leaves message unacked). Everything else — a user error thrown by the workflow body, or a `CorruptedEventLogError` — falls through to `run_failed` with `deferredCompleted` still pending.
## Concrete trigger
Workflow body: `await a(); throw new Error('boom');`, flag on.
- **Iteration N**: step A executes inline and defers its terminal write. `deferredCompleted` (pending) and `pendingSynthesizedEvents` are set.
- **Iteration N+1 (tail replay)**: the top-of-loop settle at line 1183 is skipped because `pendingSynthesizedEvents` is still set. The body replays step A from the synthesized `step_completed`, consumes its result, then throws `boom`.
- `catch` → not a `WorkflowSuspension` → `else` branch → not `PreconditionFailedError`, not retryable → falls through to the `run_failed` write while `deferredCompleted` is **still pending**.
If step A's deferred write was superseded (`EntityConflictError`/`RunExpiredError`, which the reconcile chain converts to `PreconditionFailedError` precisely to force a fresh replay), that rejection is **never observed** because settle is never called. The run is permanently and incorrectly failed on speculative state that another writer already superseded — the exact scenario the `PreconditionFailedError` conversion was built to prevent. Even when the deferred write eventually succeeds, the message is acked while the write is in flight, so a process death before the backgrounded write lands loses `step_completed`.
## The fix
Inserted a fail-closed settle at the top of the fall-through region — after the `PreconditionFailedError.is(err)` and `isRetryableWorldError(err)` checks (both of which already handle the deferred write correctly) and **before** both ack paths (`let terminalError = err;`):
```ts
if (deferredCompleted) {
try {
await settleDeferredCompleted();
} catch (settleErr) {
if (PreconditionFailedError.is(settleErr)) {
runtimeLogger.warn(
'Deferred step_completed superseded during failure handling; re-invoking run for a fresh replay instead of failing',
{ workflowRunId: runId, loopIteration }
);
return await reinvoke(0);
}
throw settleErr;
}
}
```
This routes settle rejections exactly like `run_completed`'s settle-then-catch:
- **`PreconditionFailedError`** (deferred write superseded) → `reinvoke(0)` forces a fresh replay rather than failing the run on superseded state (mirrors the existing `PreconditionFailedError.is(err)` → `reinvoke(0)` branch just above).
- **Anything else (transient)** → rethrow, leaving the message unacked so redelivery's owned recovery re-runs the step.
The guard is `if (deferredCompleted)` and `settleDeferredCompleted()` is a no-op when nothing is pending, so **non-opted-in runs (flag off, `deferredCompleted` always `null`) are entirely unaffected**. Placing it after the retryable check preserves the existing behavior for the original `err` when it is itself a `PreconditionFailedError` or a transient world error.
## Verification notes
- Braces balance across the whole file (checked: net 0).
- All referenced identifiers (`settleDeferredCompleted`, `reinvoke`, `PreconditionFailedError`, `runtimeLogger`, `runId`, `loopIteration`) are in scope and used in adjacent lines. `return await reinvoke(0)` already appears in this same `else` branch, so the return type is consistent.
- `tsc` could not be run in the sandbox (`node_modules` not installed).
Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com>
Co-authored-by: VaguelySerious <mittgfu@gmail.com>
…p_completed Regression test for the failure-branch settle: a workflow that consumes a deferred (speculative) step result and then throws, while the deferred write is superseded, must re-invoke for a fresh replay instead of writing run_failed on state another writer superseded. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Carries the base branch's second review round into the stacked feature: the sequential-replays arm of the deferCompletedWrite gate is retracted for the same reason as the base's turbo waiver — the runtime env var cannot prove the built flow trigger is actually serialized. Deferral now engages only on turbo's first delivery, where no concurrent orchestrator invocation can exist. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
VaguelySerious
force-pushed
the
peter/async-step-completed
branch
from
July 17, 2026 20:20
88f0a58 to
f7740b9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Draft, stacked on #2970 (base branch
peter/inline-delta-turbo-unlocks) — it reuses that PR'sisSequentialReplaysEnabled()runtime helper,openHookAndWaitState, andforceOptimisticStartshape. Retarget tomainafter #2970 merges (GitHub does this automatically).What
Adds an opt-in
WORKFLOW_ASYNC_STEP_COMPLETED=1: for a sequential chain of inline steps, the runtime no longer awaits each step'sstep_completedPOST before continuing. Instead it:executeStepwith the in-flight promise plus locally-synthesizedstep_created/step_started/step_completedevents (content-equal to the canonical ones: same correlationId, stepName, input, result bytes; locally-minted IDs/timestamps).cachedEvents + synthesized tail(the tail is transient — never persisted into the durable cache), so the workflow discovers the next step and — with optimistic start active — starts its body immediately.step_started, all suspension writes (hook_created/wait_created/attr_set, via a new fail-closedpreWriteBarrieronhandleSuspension), andrun_completed. The log can never record downstream events without their upstream terminal.events.listwhen it didn't, and advances the cursor — before any chained write computes itsstateUpdatedAtsnapshot (no self-inflicted 412 underWORKFLOW_PRECONDITION_GUARD).reinvoke(), the run-terminal writes, the V2-timeout / replay-budget bail-outs, and the loop top all await the deferred write first. A failure leaves the message unacked so redelivery's owned recovery (A hook_received while a step attempt is in flight re-dispatches the step; both attempts run concurrently in-process #2780) re-runs the step.Per step this hides one write RTT behind the next step's replay + body start; for chains of short steps on world-vercel that RTT is a large share of step-to-step overhead (STSO).
Gate
Compared to the gate proposed in review discussion (
!hasOpenHookOrWait && (first invocation || WORKFLOW_SEQUENTIAL_REPLAYS)), this adds:Promise.racesplit-brain shape — so parallel batches keep the awaited path;Failure handling
EntityConflictError) or lands on a terminal run (RunExpiredError): another writer owns the outcome and this invocation already speculated on our result, so the awaited path's benignskipped/gonemapping would be wrong. The rejection is converted toPreconditionFailedError, which existing routing turns into a fresh replay (reinvoke(0)) or an unacked redelivery — neverrun_failed.ownerMessageIdunchanged) and re-executes it.step_startedhits the terminal-state conflict and is skipped.Semantics disclosure (why opt-in)
step_completedis durable, step N+1 always sees the same input on any retry. With deferral, a crash while N's write is in flight re-runs N on redelivery — and a nondeterministic N can produce a different result than the one N+1's body already consumed (that body ran, its side effects happened, and no event records it). This is a strictly stronger idempotency requirement on step side effects thanWORKFLOW_OPTIMISTIC_INLINE_START(which only re-runs the same step with the same input).createdAt, soDate.now()inside workflow code can differ by roughly one write RTT + clock skew between the synthesized-tail replay and every later replay. A workflow that branches onDate.now()within that window could take different paths across replays → replay-divergence recovery, and in the worst case a detectedCORRUPTED_EVENT_LOG. (Time-branching workflows this sensitive are already fragile, but the deferral widens the exposure.)Once validated (E2E + benchmarks), folding this into turbo's default is a one-line gate change.
Changes
runtime/constants.ts—isAsyncStepCompletedEnabled().runtime/step-executor.ts—deferCompletedWriteparam; deferred branch initiates the write (threadingsinceCursorfor the delta as before), registers it withsafeWaitUntil, and returnsdeferredCompleted: { write, synthesizedEvents }.runtime/suspension-handler.ts— newpreWriteBarrierparam awaited without swallowing rejections before every suspension write (unlikerunReadyBarrier, whose swallow is intentional forrun_startedordering).runtime.ts— the gate; transient-tail replay path; reconcile chain (delta merge or fetch fallback); barrier composition intoexecuteStep'srunReadyBarrier; settle points on every ack path; the deferred step stays ininFlightOwnedStepsuntil its write settles so the existing ack-invariant assertion covers the window.Tests
pnpm testinpackages/core: 69 files, 1496 passed | 3 expected fail. New:step_completedcreate held open, the second step's body runs (impossible on the awaited path) while the secondstep_startedis not issued; after release,step_completed(A)strictly precedesstep_started(B)and the run completes.WORKFLOW_SEQUENTIAL_REPLAYS=1+WORKFLOW_OPTIMISTIC_INLINE_START=1defers the same way.EntityConflictErrorpropagates asPreconditionFailedErrorout of the unacked handler — norun_completed, norun_failed.constants.test.ts.The World-returned-delta reconcile arm is exercised by the existing #2475 suite (extraction) plus the fetch-fallback tests here; an E2E pass on world-vercel (which returns deltas) is the remaining validation before undrafting.
Updates
capabilities): the sequential-replays arm of thedeferCompletedWritegate now also requiresworld.capabilities.maxConcurrency, matching the base PR's capability gating (env flags alone don't prove backend behavior).run_failedterminal write) acked the message without settling the deferred write — a superseded write could permanently fail the run on speculative state. It now settles fail-closed first: a superseded write re-invokes for a fresh replay; transient failures rethrow (unacked → redelivery). Covered by a new regression test (workflow consumes the speculative result then throws, deferred write superseded → re-invoke, norun_failed).Update: sequential-replays arm retracted
Following the second review round on #2970 (a runtime env var cannot prove the built flow trigger carries
maxConcurrency: 1), theWORKFLOW_SEQUENTIAL_REPLAYSarm of thedeferCompletedWritegate is retracted too: deferral now engages only on turbo's first delivery. The seq arm can return together with the base PR's follow-up once the build emits a verifiable serialization signal.Docs Preview
WORKFLOW_ASYNC_STEP_COMPLETED(Link requires Vercel team access — the preview deployment is behind deployment protection.)