Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions .agents/skills/debug-openshell-cluster/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,9 +102,9 @@ journalctl -u openshell-gateway --no-pager --lines=200
openshell logs <sandbox-name> --tail --source sandbox
```

The middleware service must start before the gateway and be reachable from both the gateway and sandbox supervisors. Gateway startup fails if `Describe` is unavailable, a manifest exposes duplicate `HttpRequest/pre_credentials` bindings, the registration claims the reserved `openshell/` namespace, or body and timeout limits are invalid. Changing a registration requires a gateway restart. A policy update can also fail before persistence if the selected implementation rejects its `network_middlewares` config.
The middleware service must start before the gateway and be reachable from both the gateway and sandbox supervisors. Gateway startup fails if `Describe` is unavailable, a manifest exposes duplicate operation/phase bindings, the registration claims the reserved `openshell/` namespace, or body, message, and timeout limits are invalid. Supported V1 bindings are `HTTP_REQUEST/PRE_CREDENTIALS` and `WEBSOCKET_MESSAGE/PRE_CREDENTIALS`. Changing a registration requires a gateway restart. A policy update can also fail before persistence if the selected implementation rejects its `network_middlewares` config.

At request time, distinguish an explicit `middleware_denied` result from `middleware_failed`. A denial is always enforced. A failure follows the policy-local `on_error`: `fail_closed` blocks the request, while `fail_open` bypasses only that stage and emits a detection finding. If a running supervisor cannot install a new registry, it preserves its last-known-good generation and emits a configuration failure event.
At request time, distinguish an explicit `middleware_denied` result from `middleware_failed`. A denial is always enforced. A failure follows the policy-local `on_error`: `fail_closed` blocks the HTTP request or closes the WebSocket, while `fail_open` bypasses only that stage and emits a detection finding. A broken fail-open WebSocket stream is disabled for later messages on that connection and emits `openshell.middleware.websocket_stage_disabled`. Confirm preflight, session-start, monotonically increasing message sequences, and session-end in service logs. If a running supervisor cannot install a new registry, it preserves its last-known-good generation and emits a configuration failure event.

### Step 4: Check Docker-Backed Gateways

Expand Down Expand Up @@ -400,7 +400,8 @@ openshell logs <sandbox-name>
| Provider profiles disappear after enabling an interceptor catalog | `provider_profile_sources` selected only an authoritative interceptor or returned invalid/duplicate IDs | Inspect source list and interceptor `Describe`/catalog logs; include `builtin` and `user` when intended |
| Gateway fails after registering supervisor middleware | Service unavailable, invalid manifest, duplicate binding, reserved name, or invalid body/timeout limit | Middleware service and gateway logs; `[[openshell.supervisor.middleware]]`; `Describe` response |
| Policy update rejects `network_middlewares` | Unknown middleware name, implementation-owned config invalid, duplicate order, broad/invalid host selector, or fail-closed coverage of `tls: skip` | Policy error, gateway logs, middleware `ValidateConfig`, selector and order fields |
| HTTP request returns `middleware_failed` or `middleware_denied` | Selected stage failed or explicitly denied the admitted request | Sandbox OCSF logs; policy-local middleware config; service availability; `on_error` |
| HTTP request returns `middleware_failed`, or WebSocket closes with `1008` | Selected stage failed or explicitly denied admitted traffic | Sandbox OCSF logs; policy-local middleware config; service availability; binding operation; `on_error` |
| WebSocket messages stop reaching middleware after one failure | A fail-open stage stream was disabled for the rest of the connection | `openshell.middleware.websocket_stage_disabled`; middleware timeout/stream logs; reconnect to create a fresh stream |
| Custom compute driver is unavailable | Driver process/socket missing, inaccessible, or configured with a reserved/mismatched name | Socket ownership/mode, driver service logs, gateway `GetCapabilities` logs |
| Image pull failure | Gateway or sandbox image cannot be pulled | Runtime events and image pull credentials |
| `K8s namespace not ready` with `envoy-gateway-openshell.yaml: the server could not find the requested resource` | Optional Gateway API manifest was applied without Envoy Gateway CRDs, or k3s Helm controller startup exceeded the namespace wait | Apply `deploy/kube/manifests/envoy-gateway-openshell.yaml` manually only after Envoy Gateway is installed and `grpcRoute` is enabled |
Expand Down
8 changes: 5 additions & 3 deletions .agents/skills/generate-sandbox-policy/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ Regardless of tier, extract (or infer) these from the user's description:
| **Paths** | Specific URL paths or patterns | Only for custom/fine-grained |
| **Enforcement** | `enforce` or `audit`? Default to `enforce`. | No — has a default |
| **Binary** | Which binary/process should have access | Yes — ask if not stated |
| **Middleware** | Whether admitted HTTP requests need an ordered built-in or operator-run processing stage | No |
| **Middleware** | Whether admitted HTTP requests or client WebSocket text messages need an ordered built-in or operator-run processing stage | No |

If the host and access level are clear but binaries are not specified, ask the user which binary or process will be making the requests. Suggest common defaults like `/usr/bin/curl`, `/usr/local/bin/claude`, etc.

Expand Down Expand Up @@ -216,10 +216,12 @@ Is L7 inspection needed?

### Middleware Decision

Add `network_middlewares` only when the user asks to inspect, transform, redact, or independently authorize admitted HTTP requests. Middleware runs after network and L7 policy admission and before provider credential injection.
Add `network_middlewares` only when the user asks to inspect, transform, redact, or independently authorize admitted HTTP requests or client WebSocket text messages. Middleware runs after network and L7 policy admission and before provider credential injection.

- Use a built-in name such as `openshell/regex` without gateway registration.
- Use an operator-owned middleware name only when it is already registered under `[[openshell.supervisor.middleware]]` and reachable from both the gateway and sandbox supervisors.
- Confirm that a requested WebSocket implementation exposes a `WEBSOCKET_MESSAGE/PRE_CREDENTIALS` binding. WebSocket middleware runs for both `ws://` and `wss://`, receives complete client text messages only, and does not inspect binary or upstream-to-client messages.
- Treat `fail_open` on WebSocket as a session-scoped bypass: if the stage stream fails, OpenShell disables it for later messages on that connection and emits a state-change finding. Prefer `fail_closed` for required redaction or authorization.
- Default `on_error` to `fail_closed`. Use `fail_open` only when bypassing the stage preserves the user's stated security requirement.
- Assign unique `order` values across the complete policy. Lower values run first, and at most 10 configs may be selected.
- Match the narrowest destination hosts possible with `endpoints.include`; use `exclude` when a broad selector has trusted exceptions.
Expand Down Expand Up @@ -406,7 +408,7 @@ Evaluate the generated policy for overly broad access and **include warnings in
| **Hostless `allowed_ips`** (no `host` field) | "This endpoint has no `host` — any domain resolving to the allowed IP range on this port will be permitted. Consider adding a `host` field to restrict which domains can use this allowlist." |
| **Broad CIDR** in `allowed_ips` (e.g., `10.0.0.0/8`) | "This `allowed_ips` entry covers a very broad range. Consider narrowing to a specific subnet (e.g., `10.0.5.0/24`) to minimize exposure." |
| **`on_error: fail_open`** | "This middleware can be bypassed when it is unavailable, rejects configuration, returns an invalid result, or exceeds its body limit. Use `fail_closed` unless availability is more important than this control." |
| **Broad middleware host selector** | "This middleware applies independently of the admitting network rule to every matching HTTP destination. Narrow `endpoints.include` or add exclusions if the stage is not required for every matching host." |
| **Broad middleware host selector** | "This middleware applies independently of the admitting network rule to every matching HTTP or WebSocket destination. Narrow `endpoints.include` or add exclusions if the stage is not required for every matching host." |

Format breadth warnings clearly in the output, e.g.:

Expand Down
4 changes: 3 additions & 1 deletion .agents/skills/openshell-cli/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -341,10 +341,12 @@ Edit `current-policy.yaml` to allow the blocked actions. **For policy content au
- TLS termination configuration
- Enforcement modes (`audit` vs `enforce`)
- Binary matching patterns
- Ordered `network_middlewares`, host selection, and `fail_open` or `fail_closed` behavior
- Ordered `network_middlewares`, host selection, HTTP and WebSocket bindings, and `fail_open` or `fail_closed` behavior

`network_policies` and `network_middlewares` can be modified at runtime. If `filesystem_policy`, `landlock`, or `process` need changes, the sandbox must be recreated. Built-in middleware such as `openshell/regex` needs no gateway registration. An operator-run middleware must already be registered under `[[openshell.supervisor.middleware]]`; changing that static registration requires a gateway restart.

Middleware can inspect parsed HTTP request bodies and complete client-to-upstream WebSocket text messages over both `ws://` and `wss://` when the implementation advertises the matching binding. Binary and upstream-to-client WebSocket messages remain uninspected. A broken fail-open WebSocket stage is disabled for the rest of that connection; inspect sandbox OCSF logs for `openshell.middleware.websocket_stage_disabled`.

### Step 5: Push the updated policy

```bash
Expand Down
3 changes: 3 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions crates/openshell-core/src/middleware.rs
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,12 @@ pub const DEFAULT_MIDDLEWARE_TIMEOUT: Duration = Duration::from_millis(500);
pub const MIN_MIDDLEWARE_TIMEOUT: Duration = Duration::from_millis(10);
/// Largest operator-configured supervisor middleware RPC timeout.
pub const MAX_MIDDLEWARE_TIMEOUT: Duration = Duration::from_secs(30);
/// Maximum time a complete message may spend in all middleware stages.
pub const MAX_MIDDLEWARE_CHAIN_TIMEOUT: Duration = Duration::from_secs(30);
/// Maximum time WebSocket preflight may delay an upstream handshake.
pub const MAX_MIDDLEWARE_PREFLIGHT_TIMEOUT: Duration = Duration::from_secs(1);
/// Process-wide safety valve for concurrently buffered middleware work.
pub const MAX_CONCURRENT_MIDDLEWARE_WORK: usize = 32;

/// Largest number of middleware configurations accepted in one sandbox policy.
pub const MAX_MIDDLEWARE_CONFIGS: usize = 10;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ prost-types = { workspace = true }
regex = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
tokio-stream = { workspace = true }
tonic = { workspace = true, features = ["server"] }

[dev-dependencies]
Expand Down
24 changes: 22 additions & 2 deletions crates/openshell-supervisor-middleware-builtins/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,20 +5,29 @@

mod regex;

use std::pin::Pin;
use std::sync::Arc;

use miette::{Result, miette};
use openshell_core::proto::middleware::v1::supervisor_middleware_server::SupervisorMiddleware;
use openshell_core::proto::{
HttpRequestEvaluation, HttpRequestResult, MiddlewareManifest, ValidateConfigRequest,
ValidateConfigResponse,
ValidateConfigResponse, WebSocketEvaluationRequest, WebSocketEvaluationResponse,
};
use tonic::{Request, Response, Status};

pub use regex::{NAME as BUILTIN_REGEX, RegexConfig, RegexMode};

/// Return the first-party services that the gateway and supervisor install.
pub fn services() -> Vec<Arc<dyn SupervisorMiddleware>> {
type WebSocketResponseStream = Pin<
Box<
dyn tokio_stream::Stream<Item = std::result::Result<WebSocketEvaluationResponse, Status>>
+ Send,
>,
>;

pub fn services()
-> Vec<Arc<dyn SupervisorMiddleware<EvaluateWebSocketStream = WebSocketResponseStream>>> {
vec![Arc::new(BuiltinMiddlewareService)]
}

Expand Down Expand Up @@ -47,6 +56,8 @@ pub struct BuiltinMiddlewareService;

#[tonic::async_trait]
impl SupervisorMiddleware for BuiltinMiddlewareService {
type EvaluateWebSocketStream = WebSocketResponseStream;

async fn describe(
&self,
_request: Request<()>,
Expand Down Expand Up @@ -86,6 +97,15 @@ impl SupervisorMiddleware for BuiltinMiddlewareService {
.map(Response::new)
.map_err(|error| Status::invalid_argument(error.to_string()))
}

async fn evaluate_web_socket(
&self,
_request: Request<tonic::Streaming<WebSocketEvaluationRequest>>,
) -> Result<Response<Self::EvaluateWebSocketStream>, Status> {
Err(Status::unimplemented(
"built-in middleware does not advertise WebSocket support",
))
}
}

#[cfg(test)]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ pub fn describe() -> MiddlewareBinding {
phase: SupervisorMiddlewarePhase::PreCredentials as i32,
max_body_bytes: MAX_BODY_BYTES,
timeout: String::new(),
max_message_bytes: 0,
}
}

Expand Down
2 changes: 2 additions & 0 deletions crates/openshell-supervisor-middleware/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,11 @@ rust-version.workspace = true
openshell-core = { path = "../openshell-core", default-features = false }

miette = { workspace = true }
futures = { workspace = true }
prost = { workspace = true }
prost-types = { workspace = true }
tokio = { workspace = true }
tokio-stream = { workspace = true }
tonic = { workspace = true, features = ["channel", "server", "tls-native-roots"] }

[dev-dependencies]
Expand Down
Loading
Loading