feat(middleware): inspect WebSocket text messages

[pimlock]

Summary

Adds the PR1 supervisor middleware contract and relay path for forward-direction WebSocket text inspection. Operator-run middleware can now preflight a wss upgrade, inspect OpenAI WebSocket mode events before credential resolution, return a replacement payload, or deny the session.

Related Issue

Part of #2428

Changes

• Add the bidirectional EvaluateWebSocket gRPC stream and exact WEBSOCKET_MESSAGE/PRE_CREDENTIALS binding validation.
• Add ordered preflight/session/message handling with fail-open/fail-closed behavior, per-stage deadlines, a total chain budget, shared admission capacity, and bounded envelopes.
• Integrate forward text inspection into the parsed WebSocket relay while leaving server-to-client traffic, binary messages, and no-middleware sessions on their existing paths.
• Preserve built-in policy ordering: inspect unresolved content, apply external transformations, re-evaluate policy, then resolve credential placeholders.
• Add safe OCSF invocation/failure/saturation events and protocol close behavior (1008, 1009, 1002, and 1012).
• Add real tonic sample middleware coverage that redacts an OpenAI response.create event and verifies the upstream sees only [REDACTED]; also verify deny-after-allow closes both peers with 1008.
• Update the RFC contract documentation and leave PRE_RETURN explicitly reserved for PR2.

Testing

• mise run pre-commit passes
• Unit tests added/updated
• E2E tests added/updated (if applicable)
• mise run ci
• mise run e2e:websocket-conformance
• mise run e2e:docker

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[pimlock]

gator-agent

PR Review Status

Validation: This PR is project-valid because it implements the maintainer-authored, review-ready plan in #2428 and stays within the planned PR1 forward-text contract.
Head SHA: 22866e79e76619221db4922193eecc5330870c1c

Review findings:

• High — plaintext WebSocket middleware bypass (CWE-693): handle_forward_proxy runs only the unary HTTP chain, where WebSocket-only bindings are filtered out, and then upgrades ws:// without preflight or a middleware session. Add the same WebSocket preflight/session/subprotocol propagation used by the other relay paths, or reject matching plaintext upgrades until supported. Add a ws:// forward-proxy regression test.
• High — admission does not bound buffered memory (CWE-400): WebSocket payloads are fully read/reassembled/decompressed before waiting for the shared permit, and HTTP bodies are likewise buffered before admission. Concurrent multi-megabyte requests can therefore accumulate while queued. Reserve bounded work or weighted bytes before buffering and bound or shed the waiting queue.
• Medium — termination causes are stringly typed and misclassified: middleware deny and service failure collapse to the same error; close codes are inferred from error text; and session-end reasons conflate policy reload, protocol errors, disconnects, and middleware failures. Use a typed termination cause mapped directly to RFC 6455 close codes and WebSocketSessionEndReason (including 1007 for invalid UTF-8).
• Medium — published operator docs and mapped skills remain HTTP-only: update the existing Fern middleware, policy-schema, gateway-config, sandbox-policy, and logging pages for the WebSocket binding, max_message_bytes, stream/failure behavior, OCSF events, and the valid zero max_body_bytes shape for WebSocket-only services. The existing pages are already in navigation. Also update the affected generate-sandbox-policy, openshell-cli, and debug-openshell-cluster guidance identified by the maintenance map.
• Medium — PR1 fault behavior lacks regression coverage: add deterministic coverage for slow/hanging/closed/missing/duplicate/out-of-order/oversized responses, subsequent messages after fail-open, admission saturation, stream lifetime, plaintext upgrades, and transformed fragmented/compressed text.

Two additional line-specific findings are attached inline.

Docs: Missing for this direct operator-facing middleware contract change.

Next state: gator:in-review pending author changes. No local tests were run as part of this code-only review.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-2477.docs.buildwithfern.com/openshell

[pimlock]

Addressed the initial review in d818907.

• Plaintext forward-proxy WebSocket upgrades now run the WebSocket middleware preflight, preserve the ws scheme and negotiated subprotocol, and carry the middleware session into the frame relay.
• Shared middleware admission is reserved before HTTP body buffering and before WebSocket frame payload buffering, reassembly, or decompression. The queue is bounded at 32 active evaluations plus 32 unbuffered waiters and sheds overflow.
• WebSocket termination is typed and maps directly to close codes and session-end reasons, including 1007 for invalid UTF-8, 1009 for oversized messages, 1008 for policy or middleware denial, and 1012 for policy reload.
• The RPC-wide 61-minute stream deadline is removed; per-event deadlines remain.
• Fail-open stream disablement is now explicit and observable, with regression coverage for later messages. This intentionally preserves the PR1 handoff behavior.
• Operator docs and the generate-sandbox-policy, openshell-cli, and debug-openshell-cluster skills now cover WebSocket middleware configuration, limits, failure behavior, and telemetry.
• Regression coverage was expanded for bounded admission, fail-open continuation, ws preflight metadata, typed termination, OpenAI event redaction and denial, and the existing fragmented, compressed, malformed, and capacity cases. The full-duplex return-path fault fixture remains scoped to PR2 as documented in the handoff.

Validation passed:

• mise run pre-commit
• cargo test -p openshell-supervisor-middleware: 62 passed
• cargo test -p openshell-supervisor-network: 1033 passed, 1 PR2 fixture ignored

[pimlock]

gator-agent

PR Review Status

Validation: This PR remains project-valid because it implements the maintainer-authored, review-ready PR1 plan in #2428.
Head SHA: d81890743a80e74962ab4fdf8a51b22e068f2477

Thanks @pimlock. I checked your update describing the plaintext-upgrade path, bounded admission, typed termination, stream lifetime, fail-open observability, docs, skills, and expanded coverage. The plaintext path, pre-buffer admission, typed close handling, stream-deadline removal, mapped skill updates, and much of the added coverage are present. The independent re-review found three high-severity and four medium-severity issues that remain; they are attached inline.

Review findings:

• High: incomplete WebSocket messages can pin all shared middleware admission indefinitely.
• High: middleware-only parsed WebSockets do not observe policy-generation reloads.
• High: the operator docs claim the built-in regex covers WebSockets, but its manifest and implementation do not.
• Medium: persistent middleware streams have no independent session bound; fully disabled fail-open sessions still require admission; one path performs token-grant work before preflight; and policy denial is reported as middleware denial.

Docs: Fern pages and all three mapped skills were updated, but the built-in WebSocket coverage claim must be corrected or implemented.

Next state: gator:in-review pending author changes. No local tests were run as part of this code-only review.

[pimlock]

gator-agent

High (CWE-400): Admission is reserved before read_masked_payload, but neither that read nor continuation assembly has a deadline. A client can send 32 text-frame headers or first fragments and then stall, pinning every process-wide permit indefinitely so unrelated HTTP/WebSocket middleware work blocks or is shed. Put the complete message-assembly interval under an idle/total deadline and cover stalled initial and fragmented frames.

[pimlock]

gator-agent

High (CWE-693): Policy-generation checking is conditional on inspector. A REST upgrade whose parsed relay is enabled solely by middleware_session has no inspector, so policy or middleware hot reload is never observed and the old chain can remain active indefinitely. Pass the generation guard independently, check it before each client message, and close/end with 1012/POLICY_RELOAD. Please cover middleware-only REST upgrades over both ws:// and wss://.

[pimlock]

gator-agent

High (CWE-693): This promises that openshell/regex inspects WebSocket text, but its manifest advertises only HTTP_REQUEST, and evaluate_web_socket explicitly returns UNIMPLEMENTED. Because HTTP-only bindings are silently excluded from the WebSocket chain, an operator following this guidance receives no WebSocket redaction and no failure. Either implement and advertise the binding or correct every claim that the built-in covers WebSockets, with a regression proving the selected behavior.

[pimlock]

gator-agent

Medium (CWE-400): The preflight admission permit is released when this function returns, while WebSocketSession retains up to ten long-lived gRPC streams. There is no separate session/stream limit, so a sandbox can create unbounded idle upgrades and accumulate streams, channel buffers, and relay tasks. Keep healthy streams deadline-free, but hold a distinct bounded session permit in WebSocketSession and add an idle-session fan-out regression.

[pimlock]

gator-agent

Medium: Once every fail-open stage has been disabled, later messages should bypass middleware. Instead, admission is still mandatory before the loop discovers there are no active stages; the relay likewise reserves unconditionally. Under saturation, a connection documented as bypassing the broken stage can therefore be closed. Return the original payload before admission when no stages remain active, and test a fully disabled session while the semaphore is saturated.

[pimlock]

gator-agent

Medium: Run WebSocket preflight before inject_if_needed. This branch obtains and injects a token grant before preflight, unlike the other TLS and plaintext paths. Although preflight does not expose the header, credential issuance, cache mutation, and token-endpoint traffic still occur before middleware can fail closed. Preflight the unresolved request, then obtain credentials only after it succeeds; add a resolver-call-order regression.

[pimlock]

gator-agent

Medium (CWE-778): PolicyDenial is reported to middleware as MiddlewareDenial, so the typed lifecycle still conflates two distinct enforcement sources. Add a POLICY_DENIAL session-end reason and map it separately, then extend the termination mapping test.