[pull] main from ClickHouse:main - #6
Merged
Merged
Conversation
* feat(sandbox): add geospatial Python packages Adds a geospatial stack to the sandbox Python image: rasterio and rioxarray for raster IO, geopandas/pyogrio/pyproj/shapely for vector data and reprojection, osmnx for OSM extracts, folium for map output, and gpxpy for GPX export. The sandbox runs with clone_newnet, so nothing is installable at runtime -- these have to be baked into the image to be usable at all. All packages resolve to cp314 manylinux wheels on x86_64 and aarch64, so no source builds are introduced. Marginal installed size is ~263 MB, of which rasterio (110 MB), pyogrio (99 MB) and pyproj (32 MB) are vendored GDAL/PROJ copies. Adds test_geospatial to test-sandbox.sh, which reprojects a point to EPSG:26918 -- this exercises pyproj's bundled PROJ data and GDAL loading under NsJail, the parts most likely to break in the sandbox. * test: assert geospatial reprojection actually happened The first version guarded on ^[0-9.-]+, copied from test_statsmodels. That regex matches -74.006, so a to_crs that silently returned unprojected lon/lat would have passed the test. Assert exit code 0, a strictly numeric unsigned easting, and a plausible UTM range instead. * fix: trigger package rebuild on stale /pkgs volumes packages_ready() gates package-init.sh on sentinel site-packages directories. On an upgrade over an existing /pkgs volume, PIL, markitdown, chdb and statsmodels are all already present, so the check passed and the script exited before reaching the pip install. The geospatial packages would never have landed on existing deployments without a manual FORCE_REBUILD=true. Add rasterio as a sentinel so stale volumes rebuild. build-packages.sh needs no equivalent change: it clears .package-installed and reinstalls unconditionally.
* refactor: extract SandboxBackend seam from worker sandbox dispatch
Moves the sandbox execute POST from workers.ts into a pluggable
HttpSandboxBackend behind getSandboxBackend(). Adds
CODEAPI_SANDBOX_BACKEND config (http default) and a startup policy
check that rejects lambda-microvm until that backend lands.
No wire behavior change: the signed request body and headers pass
through byte-identical, and axios errors are rethrown untouched so
the worker's abort/timeout/sandbox-error mapping is unchanged.
* feat: derive runtime session ids and add Redis session registry (dark launch)
Adds runtime_session_hint to /exec (validated, optional) and derives
rt_<sha256(namespace,user,hint)> server-side so a client hint can never
collide across tenants. The id rides JobData into the sandbox backend
context; stateless mode (default) derives nothing and enqueues
byte-identical job data.
The registry maps runtime_session_id -> MicroVM record in Redis with the
replay-state lock discipline: SET NX PX mutex, CAS-delete release,
token-fenced record writes/removals, monotonic generation counter for
launch fencing, and a last-seen zset for the idle sweeper. No consumers
yet - the Lambda backend lands behind CODEAPI_SANDBOX_BACKEND.
* feat: Lambda MicroVM client wrapper, test fake, and op throttle
Adds the LambdaMicrovmClient interface plus AwsLambdaMicrovmClient
(@aws-sdk/client-lambda-microvms, isolated to lambda-client-aws.ts and
absent from http-only bundles), a transport-free in-memory fake for bun
tests, and Redis-backed per-second token buckets with poison backoff
for the account-wide control-plane TPS limits.
Mapping notes from the SDK typings: RunMicrovm takes imageIdentifier +
imageVersion, connector ARN arrays, native idlePolicy (auto-suspend /
auto-terminate / auto-resume), runHookPayload, and a clientToken
idempotency key; auth tokens come back as an X-aws-proxy-auth header
map with minute-granularity expiry (max 60).
* feat: Lambda MicroVM image target, lifecycle hooks, and stateless backend
Adds the AWS Lambda MicroVM execution path (opt-in, config-gated behind
CODEAPI_SANDBOX_BACKEND=lambda-microvm; default http is unchanged):
- api/Dockerfile lambda-microvm-runner target: the existing sandbox-runner
packaged for Lambda MicroVMs (arm64, port 8080, /pkgs baked, no libkrun
launcher since the MicroVM is the VM boundary).
- api lifecycle hooks at /aws/lambda-microvms/runtime/v1/{ready,run,resume,
suspend,terminate}: no-op acks in Phase 1-2, /run captures the per-VM
runHookPayload (Phase 3 checkpoint attachment point).
- LambdaMicrovmSandboxBackend (stateless mode): run -> poll RUNNING ->
mint X-aws-proxy-auth token -> health -> proxy /api/v2/execute ->
terminate (incl. terminate-on-abort); throttle-aware, metrics-wired.
- Startup policy rules (blocking-PTC reject, image-ARN required, hardened
egress-connector required, token-TTL cap, no shell in prod).
- entrypoint raises RLIMIT_NOFILE hard cap to 65536: the AL2023 MicroVM
base caps it at 1024 (below the 2048 sandbox default), which made every
in-guest nsjail job fail setrlimit with EPERM. Verified on live AWS.
Live-verified on AWS Lambda MicroVMs (us-east-1): image builds and
snapshots, nsjail runs with additionalOsCapabilities ALL, bash+python
execute (code 0), and suspend/resume preserves process+memory state
with ~1.2s auto-resume-on-traffic.
* feat: persistent session workspace for stateful MicroVM sessions
Turns the semi-stateless runner stateful when a VM is bound to a runtime
session: instead of a fresh /mnt/data workspace per /execute, the VM
reuses ONE persistent workspace and one pinned UID across calls, so
files, installed packages, and chDB dirs survive between tool calls.
Modular and off by default: gated by TWO independent locks, so the
legacy fresh-per-job path is byte-identical when either is absent.
1. Image-level SANDBOX_SESSION_WORKSPACE_ENABLED — set only in the
lambda-microvm-runner target; the K8s image cannot enter session
mode regardless of any payload.
2. Per-launch /run runHookPayload {runtime_session_id, session_workspace}
— the control plane opts a specific VM in.
Mechanism:
- session-workspace.ts: process singleton bound by the /run hook,
unbound by /terminate; holds the pinned UID + output-diff and
priming-dedup state.
- workspace-isolation.ts: ensureSessionWorkspace (stable id, contents
preserved, reaper-protected) + resetSessionWorkspace teardown.
- Job branches only at three seams — prime (reuse workspace+UID, skip
re-downloading unchanged inputs), walkDir file surfacing (skip prior
outputs unchanged by size+mtime — output diffing), cleanup (keep
workspace+UID for the session).
Verified end-to-end in the arm64 runner container: fresh mode wipes
between calls; session mode persists files across calls (incl. across
languages) and accumulates; /terminate wipes and rebind gives a clean
slate. 274 api tests pass (fresh path unchanged).
* feat: backend session orchestration (find-or-launch + warm-VM reuse)
Connects the proven runner session workspace to the product: when a
runtime session id is present and the mode is affinity/strict, the
Lambda backend finds-or-launches ONE warm VM per runtime_session_id via
the registry, delivers the /run runHookPayload
{runtime_session_id, session_workspace:true} that activates the runner's
persistent workspace, and reuses that VM across executes. AWS idlePolicy
(autoResume) suspends the VM when idle and wakes it on the next request,
so there is no explicit resume in the hot path.
- Serializes per session on the registry lock; strict contention -> 409
(publicExecutionFailure maps RUNTIME_SESSION_BUSY), affinity contention
-> correct stateless one-shot fallback (files always ride the payload).
- Generation-fenced launch: a fenced worker terminates its orphan VM.
- Stateless path unchanged (one VM per exec, terminate after).
- Lifts the stateless-only startup policy; adds session/fallback/lock
metrics. 345 service tests pass, incl. reuse/serialize/fallback/fence.
* feat: session workspace checkpoint/restore for perceived statefulness across expiry
Makes an expiring/evicted MicroVM's state survive a relaunch — the
difference between real statefulness and just re-implementing the
existing file-ref system. The file-ref path only brings back files
surfaced as CodeEnvRefs; checkpoint/restore brings back the WHOLE
workspace: pip-installed packages, venvs, chDB dirs, caches, and files
with unsupported extensions.
Two runner endpoints (session-mode only, 409 otherwise so the legacy
runner exposes nothing new):
- GET /api/v2/session/checkpoint streams tar.gz of the workspace
- POST /api/v2/session/restore replaces the workspace from one,
re-owned to the session's pinned UID
Control-plane driven: the orchestrator pulls the checkpoint over the
authed proxy and owns the S3 write, so the untrusted VM never gets S3
credentials (matches the report's checkpoint-capability security model).
Verified end-to-end with two containers simulating VM expiry: VM1 builds
a python module tree + a 2KB unsupported-extension binary, is
terminated; a fresh VM2 shows the state absent (all file-refs give you),
then after restore imports the module (greet()=42) and reads the binary
(2048 bytes) — full workspace continuity across a VM swap. 276 api
tests pass.
* feat: auto-checkpoint session workspaces to S3 with restore on relaunch
Closes the 8h-rollover / eviction story so perceived statefulness is
automatic instead of control-plane-by-hand. After each successful
session exec (lock still held), pull the workspace tar from the warm VM
and store it to S3 under a deterministic key (rtsx-checkpoints/<id>.tar.gz)
so recovery survives even registry loss; record the pointer under the
same fenced write. On relaunch, a fresh session VM restores its
predecessor's checkpoint before the first exec, making an expired/evicted
VM invisible.
- Coverage is complete and tear-free: the workspace only mutates during
an exec and execs serialize on the session lock, so the post-exec
checkpoint always captures the latest state; a busy lock means a newer
exec will checkpoint instead.
- Never fatal: a missed checkpoint degrades to file-ref recovery, a
failed restore to a fresh workspace ('relaunched must be correct').
- Off => warm reuse still works, cross-VM restore falls back to file
refs. CheckpointStore injected (Minio prod / Memory in tests); byte
cap + timeout bound the transfer; checkpoint/restore/bytes metrics.
354 service tests pass incl. checkpoint-after-exec, restore-before-first
-exec ordering, no-restore-on-reuse, disabled skip, failure non-fatal.
* feat: hookless per-request session binding via X-Runtime-Session-Id header
Deliver session mode per /execute request instead of the /run lifecycle
hook. Lambda image build hooks only route on the snapshot-compatible base
container image, and enabling any runtime hook forces the /ready build hook
(which never reaches a stock container listener), so hookless image builds
are the reliable path. The runner binds its persistent workspace from the
header; the backend stamps it on the proxied execute in session mode and
drops runHookPayload from RunMicrovm. Verified on a real hookless MicroVM.
* docs: Lambda MicroVM stateful sessions runbook + Terraform + image helper
Operator guide for the optional AWS Lambda MicroVM backend: the cross-repo
picture, from-scratch AWS setup, a full config reference, operating modes,
alternative AWS methods (base image, checkpoint store, egress, quota), the
PTC replay/blocking distinction, and the hard-won runbook gotchas.
- docs/lambda-microvm/terraform: prerequisites module (checkpoint + artifact
buckets, build + logging-only execution roles with the sts:TagSession /
logs:* trust the build needs, CloudWatch log groups, checkpoint access
policy). terraform validate + fmt clean.
- service/scripts/create-microvm-image.ts: guaranteed-correct hookless
CreateMicrovmImage helper (ALL caps + cgroupv2 off baked in), the one
provisioning step Terraform can't own.
* fix: address review findings (lock TTL, symlink chown, fence ordering, +)
Fixes from the Macroscope review pass:
- registry: derive RUNTIME_SESSION_LOCK_TTL_MS from the actual launch/health/
execute/checkpoint budgets (was a 60s placeholder, could expire mid-work at
defaults); guard readRuntimeSessionRecord JSON.parse so a corrupt key reads
as missing instead of wedging the session.
- checkpoint: fence (fenced record write) BEFORE the deterministic-key S3 put
so a lock-expired caller can't clobber a newer blob; cap restore size against
maxBytes before buffering.
- session-checkpoint: lchown + never follow symlinks when chowning a restored
(untrusted) checkpoint, so a symlinked entry can't re-own files outside the
workspace.
- lambda-client: throw when a MicroVM response omits microvmId instead of
returning '' (a partial RunMicrovm response would otherwise orphan a billable
VM behind getMicrovm('')/terminateMicrovm('')).
- router: skip runtime_session_hint validation in stateless mode (the field is
ignored there, so a malformed hint must not 400).
- v2/session: only run in the persistent workspace when THIS request carried a
valid X-Runtime-Session-Id, so a headerless request never inherits a prior
session's workspace/UID.
- entrypoint: only ever raise RLIMIT_NOFILE (guard so it never clamps a higher
host default down to 65536).
- secure-startup: warn (don't silently no-op) on affinity+http.
Deferred with reasons in the PR: zset-orphan (no consumer until the sweeper
PR), bindSessionWorkspace rebind race (prevented by the 1-VM-1-session
invariant), releaseLock swallow (TTL self-heals), /run applyRunHook (inert in
the hookless design), startupApiOnly policy placement (split-deploy design Q).
* fix(docs): address review findings in Lambda MicroVM IaC + guide
- terraform: artifact + checkpoint buckets use SSE-S3 (AES256) so the s3:GetObject
build role and the checkpoint access policy need no kms:Decrypt grant (SSE-KMS
would AccessDenied on read).
- terraform: build + runtime log policies grant both the log-group ARN and its
stream form — stream-level actions (CreateLogStream/PutLogEvents) don't
match the group ARN, which would fail builds with an empty stateReason.
- terraform: validate artifact_bucket_name is non-empty when reusing an existing
bucket (else the policy resolves to arn:aws:s3:::/*); bump required_version to
>= 1.9 for cross-variable validation.
- create-microvm-image.ts: cap the poll loop with a deadline (default 30m,
MICROVM_BUILD_DEADLINE_MINUTES) and exit non-zero, so a wedged CREATING build
can't hang a provisioning job forever.
- README: add MINIO_PORT=443 to the S3 example (client defaults to 9000);
scope the teardown sweep to VMs from this image's ARN so it can't terminate
unrelated MicroVMs in a shared account.
* fix(docs): region-unique bucket names + correct checkpoint-cred guidance
- terraform: include region in the artifact + checkpoint bucket names (S3 names
are global, so a same-prefix second-region apply would collide).
- docs: correct the checkpoint credential guidance. The MinIO-compatible client
reads only static MINIO_ACCESS_KEY/SECRET and does not load task-role/IRSA
creds, so attaching checkpoint_access_policy_arn to a task role alone does not
work. Point operators at create_checkpoint_access_user (static keys) and note
IRSA-aware loading as a follow-up. Fixed across variables.tf, outputs.tf, README.
* fix: address Codex review of stateful session mechanics
- Bind the session on checkpoint/restore (F1): the hookless path runs
/session/restore before the first /execute, so the runner had nothing bound
and every real restore 409'd, losing checkpoint state across VM expiry. The
runner now binds from X-Runtime-Session-Id in the checkpoint/restore routes,
and the backend sends that header on both proxied calls.
- Clear/terminate a session VM on reuse failure or abort (F2/F5): on a dead
reused VM (idlePolicy auto-terminated) or an aborted execute (runner keeps
NsJail alive after the socket closes), terminate the VM and drop the record so
the next call relaunches + restores instead of reusing a dead-or-dirty VM. A
plain non-200 from a live runner leaves the warm VM intact.
- Enforce checkpoint size before buffering (F7): CheckpointStore.get takes
maxBytes and stats the S3 object first, so an oversized/stray checkpoint can't
OOM the worker before the (now-removed, too-late) post-buffer guard.
- Mark session outputs surfaced only after upload (F3): a dropped upload no
longer permanently suppresses an unchanged file next turn.
- Bake runner file/egress/manifest env into the image (F4): create-microvm-image
takes --env-json so images can reach FILE_SERVER_URL / EGRESS_GATEWAY_URL
instead of building invalid /sessions/... URLs.
Deferred (P2, design choice): don't count checkpoint time in the client job
timeout — decoupling checkpoint from the response path needs a call on where it
sits relative to job completion; raised with the maintainer.
* fix: address Macroscope review of the Codex-fix commit
- executeOnSessionVm error classification (High, regression from the prior
commit): a healthy runner returning a non-2xx makes axios throw an AxiosError
carrying `.response`, which the old `message === 'Error from sandbox'` check
treated as unreachable and tore the VM down. Now keep the VM when the runner
responded (isAxiosError && response), terminate only on no-response
(connection/timeout) or abort. Regression test added.
- findOrLaunchSession: treat an image_arn/version/port mismatch as non-reusable
so a warm session relaunches on the current config after a deploy instead of
running the old image (or health-checking the wrong port -> UNHEALTHY).
- MinioCheckpointStore.get: cap accumulated bytes during download too, not just
via statObject, so an object that grows between stat and read can't OOM.
- /session/checkpoint + /session/restore: fail closed (409) when the request
carries no valid X-Runtime-Session-Id instead of acting on a stale bound
session, and `.catch(next)` so a rejected handler promise is forwarded to
Express 4's error handler instead of hanging the request.
* fix: address Codex review round 2 of stateful session mechanics
- Lock TTL (P1): budget 2x launch (throttle wait + poll-to-RUNNING) and 2x
checkpoint (restore-before-exec + post-exec) so a slow relaunch+restore can't
expire the lock mid-critical-section.
- Read-only inputs re-download (P1): a read_only prime (skill file) is reported
as not-primed so it always re-downloads, restoring pristine content + the
0444/root protection a sandbox could have stripped by unlink+replace.
- Relaunch idle-expired sessions (P1): treat a record whose last_seen is past
idle+suspended as non-reusable, so the first request after idle expiry
relaunches+restores instead of reusing a dead endpoint and 503ing.
- Content-hash output diffing (P2): suppress a surfaced output only when its
CONTENT hash is unchanged, not size+mtime (spoofable via os.utime); the hash
is computed once per file and reused for input-modification detection.
- RunMicrovm logging (P2): send the CloudWatch logging config (new
LAMBDA_MICROVM_LOG_GROUP) so VM stdout reaches CloudWatch — the role alone
wasn't enough.
- Preserve sessions on non-transport failures (P2): terminate the VM only on
abort or a transport-level axios failure (no response); a throttled
CreateMicrovmAuthToken or a non-2xx runner response keeps the warm VM.
- Skip the optional checkpoint when the job budget is spent (P2), so a run that
already succeeded isn't timed out at the router by checkpoint latency.
* fix: address Codex review round 3 of stateful session mechanics
- Tear down the warm session VM on health-check failure, not just on
transport errors: assertHealthy wraps failures as MICROVM_UNHEALTHY, and
executeOnSessionVm now recycles the VM + drops the registry record for it.
- Restore whenever checkpoints are active on relaunch (drop the record-present
precondition) so a fresh VM still pulls the last checkpoint.
- Bound the checkpoint object-store put with the checkpoint timeout so a
stalled S3/MinIO write can't hold the session lock past JOB_TIMEOUT.
- Wipe the workspace to a clean slate when a restore fails mid-extract, so the
job never runs against a half-applied checkpoint.
- Throttle CreateMicrovmAuthToken under a shared per-second token budget
(LAMBDA_MICROVM_TOKEN_TPS, default 8) so concurrent warm-session executes
queue instead of bursting past the AWS TPS limit, mirroring launch's run
budget.
* fix: address Codex review round 4 + Macroscope of stateful sessions
Codex round 4 (7) and the open Macroscope batch (3):
- Preserve priming/output-diff state across a checkpoint restore (P1). A
relaunched VM started with an empty primed map and re-downloaded every input
ref, overwriting a restored in-place-modified file with its original. The
checkpoint now carries a sidecar (SessionWorkspace.snapshotMeta) that the
restore rebuilds (loadMeta), so warm-VM behavior survives a relaunch.
- Version checkpoint objects by a per-checkpoint monotonic sequence and read the
highest on restore. A put that timed out and lands late writes an older key
and can never overwrite a newer checkpoint; restore still works with no Redis
record (list by session prefix). Best-effort prune of strictly-older keys.
- mintAuthToken maps control-plane failures instead of letting them escape raw:
throttled -> poisonOpBucket('token') + MICROVM_LAUNCH_THROTTLED; not_found
(VM evicted) -> MICROVM_UNHEALTHY so the caller tears down the stale record
and relaunches; other -> MICROVM_LAUNCH_FAILED.
- Route checkpoint/restore token mints through the same token budget via a
mintToken callback, so a burst of concurrent sessions can't bypass the TPS cap.
- Bound the checkpoint fetch (store.get) with the checkpoint timeout, symmetric
with the put, so a stalled S3 read can't hold the session lock through a
relaunch.
- Terminate a superseded (config/version/port drift, deadline-near) session VM
before relaunching, except when AWS already auto-terminated it, so it isn't
left running and billing.
- Restore extracts with --strip-components=1 -C dir so a poisoned archive member
can't escape the workspace into shared runner space (Macroscope corrected: a
plain -C dir double-nests the session/ prefix).
- Fail fast at startup when session checkpoints are enabled without object
storage configured, instead of silently falling back to a localhost/test
bucket and dropping workspace state on the first relaunch.
* fix: address Codex review round 5 of stateful session mechanics
- Refuse a symlinked checkpoint sidecar (P1). The round-4 primed-persistence
sidecar was written with a plain (root) writeFile; sandboxed code could squat
.codeapi-session-meta.json as a symlink and turn the post-exec checkpoint into
an arbitrary-file overwrite. Unlink first (never follows a link) then create a
fresh regular file exclusively (wx); on restore, lstat-guard the read so a
symlinked sidecar in an untrusted archive is never followed.
- Route non-8080 MicroVM ports with X-aws-proxy-port. Health/execute/checkpoint
traffic omitted the port header, so a non-default LAMBDA_MICROVM_PORT would be
routed to 8080 (the token is minted for the configured port). Added
microvmPortHeaders(port) and applied it to all proxied requests.
- Reserve the whole checkpoint path budget before starting it. The skip guard
checked only one checkpoint timeout, but checkpointUnderLock can spend a
token-budget wait + GET + put; a run finishing with barely more than one
timeout left could still blow waitUntilFinished(JOB_TIMEOUT). Require
launchTimeoutMs + 2*checkpoint.timeoutMs of headroom.
- Baseline reused writable inputs against the original upload hash, not a
re-hash of the on-disk copy. A prior turn's in-place mutation was banked as
pristine, letting the walker echo the original ref as unchanged. The primed
map now retains the original hash (round-trips through the checkpoint sidecar)
and reuse baselines against it, so a mutation is reported modified-from-
original without re-downloading (preserves the round-4 persistence fix).
* fix: address Codex review round 6 of stateful session mechanics
- Drop worker-only backend validation from API-only startup. An API pod
authenticates + enqueues and never builds the Lambda backend or checkpoint
store, so validateSandboxBackendPolicy() there forced LAMBDA_MICROVM_* and the
MINIO_* checkpoint creds into API pods just to boot. Worker/combined startup
still validate it.
- Map MicroVM poll (GetMicrovm) failures like runMicrovm errors. A throttle or
transient control-plane error during waitUntilRunning rethrew a raw
LambdaMicrovmApiError -> generic 500; now throttled -> MICROVM_LAUNCH_THROTTLED
(+poison), other -> MICROVM_LAUNCH_FAILED.
- Don't re-surface previously-primed session inputs as generated outputs. A
file primed in an earlier turn persists in the workspace; if a later turn
omits it, it has no inputFileHashes/surfaced entry and was echoed as a brand
new output. Skip paths the session primed as inputs (SessionWorkspace
.isPrimedInput).
- Remove a squatted checkpoint metadata directory before writing. The round-5
no-follow rm was non-recursive, so a sandboxed `.codeapi-session-meta.json`
directory made every checkpoint fail EISDIR; rm now recursive (still no-follow
for symlinks).
- Key checkpoint objects by the wall-clock time they were taken, not a Redis
sequence counter. The counter carried the record TTL, so after a long idle it
reset to 1 while retained S3 objects kept a higher sequence, and restore (which
reads the lexicographically-greatest key) restored stale state. A never-
resetting timestamp sorts correctly across idle gaps and lets prune drop the
stale object; a stalled late put still loses because it carries its earlier
start time.
* fix: address Codex review round 7 of stateful session mechanics
- Re-prime session inputs without following prior-turn symlinks (P1). A
persistent session workspace can contain a symlink (a file path or a parent
dir) planted by earlier sandbox code; priming as root followed it and
wrote/chmod'd outside the workspace. secureAncestors now rejects a symlinked
ancestor (lstat, no follow), and both the inline writeFile and the download
rename clear any squatted symlink/dir/file at the target first, so a prime
always lands a fresh regular file inside the workspace.
- Don't hide CHANGED previously-primed inputs (refines round 6). The unconditional
skip suppressed a primed input even after the sandbox modified it and defeated
the upload-retry path. Only suppress while the on-disk hash matches the primed
baseline (or the input is read-only, whose edits are dropped by contract); a
modified writable input surfaces as an output again.
- Don't time out a normal auto-resume as unhealthy (P2). A suspended session VM
auto-resumes on the first request, and that latency can exceed the 5s health
budget, so the preflight probe misread a valid slow resume as MICROVM_UNHEALTHY
and tore the VM down. Skip the preflight health check for reused session VMs
(the execute carries the resume under the job budget; an evicted VM already
fails token minting with not_found); freshly-launched VMs still get the probe.
* docs(lambda-microvm): guard the build-log path against the wrong "fix"
The AWS docs list the MicroVM build log group as /aws/lambda/microvms/<name>
(slash), but the real path is /aws/lambda-microvms/<name> (hyphen) — re-verified
against a live account (every build's group is the hyphen form; the slash path
does not exist). A passive "not the docs' path" note wasn't enough (a reviewer
still flagged it citing the docs), so make it an explicit guard in both the
terraform comment and the runbook: do not correct it to the slash path or the
build logs are lost and CreateMicrovmImage fails with an empty stateReason.
* fix: address Codex review round 8 of stateful session mechanics
- Advance last_seen_at on checkpointed executes. checkpointUnderLock returned
the record checkpointSession re-read (built from the pre-execute snapshot), so
liveness never moved forward and an actively-used session eventually looked
idle and relaunched needlessly. Re-apply last_seen_at: now.
- Budget the session lock for all checkpoint I/O. Restore (store.get +
pushRestore) and the post-run checkpoint (pullCheckpoint + store.put) each do
TWO timeout-bounded I/Os, so the TTL now reserves 4x CHECKPOINT_TIMEOUT (was
2x) or the lock could expire mid-op and let a second worker run concurrently.
- Include network-connector drift in session reuse (P1). configMatches only
compared image/version/port; connectors apply only at RunMicrovm, so a
hardened deploy that tightened egress kept reusing VMs on the old broader
policy. Record a connector fingerprint at launch and compare it.
- Wait for runner health before restoring. Restore ran as soon as the VM was
RUNNING (control-plane state) — before the app listener was up — so pushRestore
could race the boot, fail non-fatally, and run the first execute on an empty
workspace. Poll readiness first; tear the VM down if it never comes up.
- Treat an empty CODEAPI_CHECKPOINT_BUCKET as unset (|| not ??) so it falls
through to MINIO_BUCKET instead of selecting '' and failing every S3 op.
- Don't clobber a user file at the checkpoint sidecar path. If user code creates
a real .codeapi-session-meta.json, skip metadata persistence that turn rather
than deleting it, and on restore only remove a sidecar we recognize as ours.
* fix: address Codex review round 9 of stateful session mechanics
- Renew the session lock on a heartbeat instead of pinning its TTL to the
worst-case sum. The critical path (launch throttle + readiness/restore +
execute + checkpoint) includes several per-op token-mint throttle waits, so no
fixed TTL reliably covers it (this is the third TTL finding). renewRuntime-
SessionLock extends the lock while the holder runs; a fenced renew stops when
another worker owns it. TTL is now just a comfortable base.
- Key checkpoints by a store-seeded monotonic Redis sequence, not wall-clock.
Wall-clock keys (round 6) mis-order across pods with skewed/backward clocks;
the sequence is a pure INCR seeded above latestSequence() after a TTL reset, so
it's skew-free AND never resets below retained objects.
- Recycle a session VM when push-restore fails. restoreSession now distinguishes
fetch_failed (clean fresh workspace, safe to execute) from push_failed (runner
may be mid-extract/wipe) and tears the VM down on push_failed so the next call
relaunches clean.
- Recycle a reused VM on a proxy 502/503/504 (a suspended VM that failed to
auto-resume) instead of treating it as a live-runner response and reusing the
dead VM until idle expiry.
- Authenticity marker on the checkpoint sidecar: restore only loads/deletes a
file carrying our marker, so a user file that merely shares the reserved name
and happens to hold primed/surfaced arrays is never poisoned/deleted.
- Track inline source files (writeFile) as session inputs so switching entrypoint
(main.py -> script.sh) doesn't re-surface the prior source as a generated file.
- Build session workspace ancestors no-follow BEFORE mkdir. The round-7 symlink
check ran after mkdir -p, which had already followed a symlinked ancestor;
ensureDirNoFollow creates one component at a time, rejecting symlinks.
* fix: address Codex review round 10 of stateful session mechanics
* fix: address Codex review round 11 (tar close-listener races, fresh-runner readiness, session-file preservation, storage-session prime reuse)
* fix(runner): truncate stderr overflow instead of SIGKILL + surface kill reasons in-band
A healthy long-running job whose library spams stderr (chdb dumps a
multi-KB 'Cannot use cgroups reader' stack trace per operation inside
MicroVM guests) was SIGKILLed with status EL the moment stderr crossed
SANDBOX_OUTPUT_MAX_SIZE (default 1024 bytes) — observed as 40-80s
computations dying with exit 137 and zero stdout. stderr overflow now
truncates at the cap and keeps draining; the job runs to completion.
stdout overflow still kills (stdout is the product; runaway producers
should stop paying for compute).
Kill/truncation reasons now also append an in-band '[sandbox] <reason>'
marker to stderr/output: most clients (and the models consuming tool
output) only ever see the streams, so TO/OL/OOM kills previously read
as silent hangs.
create-microvm-image: default memory 2048 -> 4096 MiB (RunMicrovm has no
per-session memory override, so image build time is the only memory
lever and embedded engines OOM inside 2048), poll builds via the ARN
returned by Create/Update (GetMicrovmImage 400s on names), and document
the low runner-limit env defaults that need raising for session
workloads.
* feat(runner): boot-time SANDBOX_WARMUP_COMMAND hook + 30min idle default
Fresh MicroVMs lazy-page their rootfs, so the first touch of a large
read-mostly asset is brutal: chdb's 408MB shared object takes 30-120s to
import cold (it blew the 120s exec timeout in the field, presenting as a
silent hang) but ~250ms warm. Two mitigations:
- SANDBOX_WARMUP_COMMAND: optional command spawned once at sandbox-API
startup, detached and outside any job sandbox, so it pre-faults heavy
assets into the page cache during the boot window instead of the
user's first tool call, without ever holding a session lock or
blocking readiness. Best-effort by design (failures logged, ignored).
Set e.g. SANDBOX_WARMUP_COMMAND="python3 -c 'import chdb'" via the
image env.
- LAMBDA_MICROVM_IDLE_SECONDS default 300 -> 1800: keep a session's VM
fully RUNNING (RAM + page cache intact, ~0.3s follow-ups) across a
realistic conversation gap before suspending; still env-configurable.
* fix(lambda-microvm): retry launch once when a MicroVM dies during boot
Boot-time deaths (VM enters TERMINATED/TERMINATING before becoming
ready) are fast provider-side transients — observed several times a day
in the field — and the backend previously surfaced each one straight to
the caller as MICROVM_LAUNCH_FAILED (HTTP 503) on the user's tool call.
Mark that specific failure transient on SandboxBackendError and have
launch() retry exactly once with a fresh clientToken (RunMicrovm is
idempotent per token, so reusing the original would return the same
dead VM). Throttles, aborts, and poll-deadline timeouts stay
non-retried: a throttle retry adds pressure to a poisoned bucket and a
timeout retry doubles a 60s wait against the job budget. Retries are
visible via the microvmLaunches{outcome=retried} counter.
* docs(runner): correct SANDBOX_OUTPUT_MAX_SIZE semantics
The table claimed both streams truncate at the cap. stdout overflow
actually SIGKILLs the job (status OL) — stdout is the result, so a runaway
producer is cut off — while stderr truncates and the job continues. Also
note that every shipped compose/helm config sets 65536, so the 1024
fallback only applies to a bare runner.
* feat: push-model input file delivery for session VMs
MicroVMs have internet-only egress, so the runner's pull-based input
priming has nothing reachable to pull from — uploads authorized into the
internal file server never reached the sandbox. Deliver them push-model
instead, over the same authed proxy channel as a checkpoint restore:
- runner: additive POST /api/v2/session/files extracts a tar.gz overlay
into the bound session workspace (restore's traversal hardening, but
never a wipe — failure leaves existing session state untouched). A
reserved manifest member registers delivered files as primed, so the
execute's own priming reuses the on-disk copy (same-id reuse) and later
turns suppress unchanged inputs from the output scan.
- service: buildSessionFilesArchive fetches the request's by-ref files
from the internal file server, stages them under session/ with the
primed-files manifest, and pushFiles delivers the archive before the
execute under the held session lock. Build failures throw before any
bytes reach the VM (warm VM survives); a failed push after transfer
began recycles the VM like a failed push-restore.
Known limitations (follow-ups): delivery re-pushes on every exec (no
service-side dedupe against the runner's primed state yet), and the
stateless path still has no delivery leg.
* feat(file-server): optional FILE_SERVER_HOST bind address
Push-model deployments fetch input bytes control-plane-side, so the file
server only needs to be reachable from the co-located service — binding
127.0.0.1 keeps the object routes off the network entirely. Unset
preserves the historical all-interfaces bind.
* fix: harden push delivery, checkpoint restore, and session lifecycle
Addresses the P1 review findings on the push-model delivery and session
machinery:
- delivery dedupe: writable refs already delivered to the live workspace
are recorded on the session (delivered_files + delivered_at) and never
re-pushed — a later turn re-sending the same ref can no longer overwrite
in-place modifications the sandbox made. After a relaunch the list is
trusted only when the restored checkpoint post-dates the last delivery.
Read-only refs (X-Read-Only from the file server) are the deliberate
exception: re-delivered every exec and primed read-only by the runner,
mirroring the pull model's re-download rule.
- checkpoint restore fails closed: a transient fetch failure now recycles
with a retryable 503 instead of executing on an empty workspace whose
post-run checkpoint would prune the last good snapshot (transient S3
blip -> permanent data loss).
- fence semantics: a LOST lock renewal (vs a transient renew error) aborts
the in-flight critical path via AbortSignal instead of merely stopping
the heartbeat; fenced failures never terminate the VM the new lock
holder may be using.
- aborted or failed pushes always recycle the VM (partial extraction must
not be reused); archive construction gains a file-count cap, a
cumulative uncompressed-byte budget, and abort-signal support; fetch
errors log sanitized details instead of raw axios errors that carried
the internal service token.
- hintless requests never land on a session: affinity degrades them to
stateless one-shots, strict rejects them (router 400, worker hard fail)
instead of silently sharing the per-user default workspace.
- runner: rebinding a different runtime session id is refused outright
(fail-closed; no async-wipe race over the shared directory); a failed
workspace wipe retains the pinned UID instead of releasing it over a
quarantined directory; tar spawn errors reject instead of crashing the
process; sha256File opens via fsp.open for typed O_NOFOLLOW.
* fix: close remaining session-delivery and fencing gaps
Second review round on the push-model delivery. Several of these were
introduced by the previous hardening commit:
- rebind rejection is now enforced: bindSessionFromHeader propagates the
refusal (409) and /execute conflicts instead of silently running under
the previously bound session's workspace and UID.
- read-only pushed inputs are usable again: the runner marks control-plane
deliveries as fresh-for-this-exec, so priming reuses the pristine copy
rather than falling through to the pull path a push-model backend cannot
reach. Delivered read-only files also get the pull path's root-owned 0444
protection, and reuse carries the read-only bit so modifications are
dropped by contract.
- delivery keys include the destination path: the same object under a new
filename is a file the workspace does not have yet and is delivered.
- delivery metadata survives relaunch: the record is reconciled even when
every ref is skipped, so a restored session no longer re-pushes originals
over restored in-place edits.
- fencing actually stops concurrent mutation: a fenced worker terminates
the VM (closing the socket does NOT stop the runner's NsJail child), and
persistent renew errors fence once the lease can no longer be proven
held, not just on an explicit lost token.
- affinity lock contention no longer silently drops by-reference inputs:
the stateless fallback runs only for requests without them; otherwise the
call fails retryably as BUSY.
- SandboxBackendError sanitizes its cause at construction, so no logging
path can serialize axios config carrying the internal service token,
MicroVM auth headers, or the pushed archive body.
- manifest failures fail the delivery (500) instead of acknowledging a
push the next execute cannot use; unparseable content under the reserved
name is left alone as user data, and a ref claiming the reserved name is
refused at build time.
- restore metadata replaces rather than merges, so stale primed/surfaced
entries cannot suppress real files after a repeated restore.
- tar 'close' rejections are observed immediately: a spawn failure used to
crash the runner with an unhandled rejection after answering 500.
- typed O_NOFOLLOW reads via fsp.open (clears the remaining src/ tsc
errors in the runner).
* fix(runner): make re-delivery non-destructive independent of control-plane state
Live testing found the overwrite protection was still reachable: a
delivery failure recycles the VM and drops the session record, taking
delivered_files with it, so the retry re-pushed originals over the
checkpoint-restored in-place edits. Redis dedupe cannot be the
correctness mechanism — it dies with the record (recycle, failover,
flush) while the edits live on in the checkpoint.
Enforce the invariant where the data is instead: deliveries now extract
into a 0700 staging dir beside the workspace and merge file-by-file.
A WRITABLE input whose on-disk content differs from its primed baseline
is kept as-is (the sandbox's edit wins over a re-pushed original);
read-only inputs are always restored to pristine bytes, as the pull path
does. The control-plane dedupe stays as a bandwidth optimization.
Staging also tightens the delivery: only regular files found inside
staging are ever moved (symlinks skipped, containment re-checked), and
every manifest entry must correspond to a delivered file, so an
incomplete archive fails the delivery rather than stranding the execute
on the unreachable pull path.
* fix(runner): symlink-safe, validate-then-commit delivery merge
Round 3, runner half. The staging merge added in 82bb551 performed
privileged mkdir/rm/rename against paths validated only lexically, and
committed files before finishing validation.
- Reuse the pull path's proven no-follow ancestor walk instead of
hand-rolling one: Job.ensureDirNoFollow is extracted to
workspace-isolation as a shared helper (identical semantics, plus an
optional identity so directories it creates are owned by the session
rather than left root-owned and unwritable by the sandbox). A prior
turn's sandbox code can plant a symlink in the persistent workspace,
and mkdir -p/rename would follow it and write outside as root.
- Validate everything before touching live state: canonical names via the
same validateFilePath the pull path uses, plus a unique one-to-one
correspondence between manifest entries and staged regular files. A
missing, duplicate, extra, or escaping member now fails the delivery
with the workspace untouched — including a manifest-less archive.
- Preserve a sandbox edit only when the incoming ref is the SAME file
(id + storage session) the edit was made to; a different ref at that
path must land and re-prime, or the execute runs against bytes whose
identity matches nothing it declared.
- Replace directories recursively (plain rm could not, wedging that ref)
and let rename replace files/symlinks atomically without a pre-unlink.
- A failure part-way through the commit phase cannot be rolled back, so
quarantine the workspace: the runner refuses further execution until
the control plane recycles the VM and restores the last checkpoint.
* refactor: deliver inputs through the runner's own priming path
Delivery was a SECOND writer into the session workspace, duplicating what
the pull path already does — name validation, no-follow ancestors,
ownership, read-only protection, priming, modification detection. Every
delivery defect across three review rounds was a bug in one of those
copies. This removes the duplication instead of patching it again.
The push now fills a runner-local input cache keyed by a digest of
(storage session, object id), outside any workspace. Priming resolves a
ref from that cache and otherwise fetches over HTTP; a cache hit is
presented as the Response a fetch would have returned, so every
downstream step is byte-identical for pushed and pulled inputs. The
workspace keeps exactly one writer.
Consequences, all previously open findings:
- No caller-controlled path ever reaches the filesystem during delivery,
so the symlinked-ancestor class cannot recur here at all.
- Re-delivery cannot revert a sandbox edit, because delivery no longer
touches the workspace; reusePrimedInput decides, as it always has.
- Dedupe is a probe answered by the VM, not Redis bookkeeping, so it
stays correct across recycles, failover and flushes — and the
checkpoint-coverage timestamp comparison is gone entirely.
- Stateless one-shots receive by-reference inputs too (the cache is keyed
by object, not session); they previously ran with inputs missing.
- Session lock contention is now always a retryable BUSY: a session-bound
request depends on workspace state a cold VM lacks, whether or not this
particular payload carries refs.
- Read-only, duplicate-name and manifest-shape handling all collapse into
the single existing implementation.
Net effect on the runner: mergeDeliveredFiles, the manifest protocol, the
staging merge, quarantine plumbing and the delivered_files registry
fields are deleted.
* fix(runner): parse the input-probe body
There is no global express.json (see index.ts); parsers are per-route, so
the probe endpoint saw an unparsed body and rejected every ref list with
'refs must be an array'. Only live execution surfaced this — route wiring
is invisible to the unit suites.
* fix(runner): keep the input cache outside the workspace root
The cache was a dot-directory inside SANDBOX_WORKSPACE_ROOT, where the
stale-workspace reaper treats every entry as a workspace — it deleted
pending inputs between the push and the execute they were pushed for, so
a live exec reported 'No such file' for files the control plane had
successfully delivered. Unit and route tests both passed; only the real
sandbox showed it, and reapStaleWorkspaces reproduced it on demand.
Moved to a sibling directory (SANDBOX_INPUT_CACHE_DIR, default
<root>-inputs), which also keeps it off every nsjail mount, and pinned
the invariant with a regression test that runs the reaper over a seeded
cache.
* fix: align the input-cache digest across control plane and runner
The two ends computed the SAME contract differently — the runner joined
(storage session, id) with a literal NUL byte, the control plane with a
space — so every pushed object landed under a name no lookup could ever
produce. Both unit suites passed because each was self-consistent; the
route test and the priming test used one side only. A live execution
reported 'No such file' for files that had been delivered successfully.
Both now use an explicit \u0000 escape (no raw NUL in source, which also
made grep treat the file as binary), and both suites assert the same
golden vector so the contract cannot silently drift again.
* fix: a cached object's destination belongs to the requesting ref
The cache is keyed by OBJECT, but the pushed metadata carried the file
server's filename and the runner emitted it as Content-Disposition. Every
ref for that object therefore resolved to one name, so requesting the same
object at a second path wrote it to the FIRST path instead — overwriting
a file the sandbox had edited in an earlier turn. Live testing caught it;
the unit suites had encoded the wrong behaviour as expected.
Metadata now carries only object-level facts (read-only), and priming
falls back to each ref's requested name. Covered by a prime test that
serves one cached object to two destinations.
* fix: harden lambda microvm sessions and delivery
* ci: make artifact guard shellcheck-safe
* ci: lock terraform provider platforms
* fix: close lambda microvm review gaps
* fix: normalize minio ssl flag
* fix: close late session review findings
* fix: harden checkpoint rollback safety
* fix: consume checkpoint streams through eof
* fix: preserve checkpoint transaction boundaries
* fix: prevent session launch token collisions
* fix: harden runtime session identity and config
* fix: recover runtime session delivery failures
* fix: harden stateful session execution
* fix: close stateful session review gaps
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )