Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

31 changes: 30 additions & 1 deletion architecture/compute-runtimes.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ delete, reconciliation removes the row; otherwise it can remain `Deleting`.
|---|---|---|---|
| Docker | Local development with Docker available. | Container plus nested sandbox namespace. | Uses host networking so loopback gateway endpoints work from the supervisor. |
| Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API, OCI image volumes, and CDI GPU devices when available. |
| Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, and GPU resources. |
| Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, GPU resources, and optionally Agent Sandbox v1beta1 warm-pool claims. |
| VM | Experimental microVM isolation. | Per-sandbox libkrun VM. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`, waits for its Unix socket, and then consumes it through the same remote `compute_driver.proto` path used by unmanaged endpoint drivers. The VM driver boots a cached bootstrap `rootfs.ext4`, prepares requested OCI images inside a bootstrap VM with `umoci`, attaches the prepared image disk read-only, and gives each sandbox a writable `overlay.ext4` for merged-root changes and runtime material. The driver persists each accepted launch request beside the overlay and restarts those VMs on driver startup without recreating the overlay. |
| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a non-reserved custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. Reserved built-in names such as `vm`, `docker`, `podman`, and `kubernetes` cannot be used as unmanaged socket endpoints. The gateway connects to a UDS the operator already provisioned, runs `GetCapabilities`, logs the advertised `driver_name`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |

Expand Down Expand Up @@ -125,6 +125,35 @@ Driver-controlled environment variables must override sandbox image or template
values for sandbox ID, sandbox name, gateway endpoint, relay socket path, TLS
paths, and command metadata.

Kubernetes supervisors authenticate to the gateway in two stages. They first
call `RegisterSupervisorPod` with the projected ServiceAccount token; the
gateway validates the pod-bound token and live Agent Sandbox owner state, then
activates already-bound cold pods by streaming back a gateway-minted sandbox
JWT. The supervisor installs that JWT in memory and starts the normal
`ConnectSupervisor` session as the activated sandbox. The gateway does not dial
pod IPs or require an inbound activation port; activation is supervisor
initiated over the existing outbound gRPC connection.

When compatible OpenShell-enabled Agent Sandbox warm pools exist, the
Kubernetes driver can create a `SandboxClaim` instead of a direct `Sandbox`.
The driver returns an opaque activation token with the create response for
those delayed warm allocations. The gateway stores only a hash of that token
against the sandbox ID and later requires the claim activation controller to
present the same token before minting a sandbox-scoped supervisor JWT. The
token is derived by the driver from immutable live claim identity, including
the claim UID; Kubernetes labels remain useful for diagnostics and cross-checks
but are not sufficient activation proof. The driver watches warm pools and
templates into a local cache so create requests do not list or fetch extension
objects on the hot path. This remains behind the compute-driver boundary:
other drivers and the driver-agnostic gateway lifecycle continue to operate on
OpenShell sandbox identity and status.
The Kubernetes driver can optionally run a separate ConfigMap profile
reconciler that turns admin-authored TOML profiles into generated
`SandboxTemplate` and `SandboxWarmPool` resources. That reconciler is
independent of the create-path matcher: admins can use it, Helm, kubectl, or
another controller to create warm pools, and the matcher only considers the
resulting enabled warm-pool resources and their fingerprints.

Kubernetes can run the supervisor in the default combined topology or in a
sidecar topology. Combined mode keeps network and process supervision in the
agent container. Sidecar mode runs network enforcement, the proxy, and gateway
Expand Down
53 changes: 34 additions & 19 deletions architecture/gateway.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,24 +155,36 @@ token is reported as connected but unauthenticated.
Sandbox supervisor RPCs authenticate with explicit sandbox credentials; mTLS
does not grant sandbox identity. Kubernetes deployments use the
gateway-minted JWT bootstrap path: the supervisor starts with a projected
ServiceAccount token, exchanges it for a gateway-minted sandbox JWT, and uses
that JWT on subsequent gateway RPCs.
ServiceAccount token, registers the pod with the gateway, receives a
gateway-minted sandbox JWT after activation, and uses that JWT on subsequent
gateway RPCs.
User-facing mutations are authorized by role policy when OIDC or edge identity
is enabled.

Sandbox secrets are gateway-signed JWTs bound to a single sandbox ID. Docker,
Podman, and VM drivers deliver the initial token through supervisor-only
runtime material; Kubernetes supervisors exchange a projected ServiceAccount
token through `IssueSandboxToken`. The gateway validates that projected token
with Kubernetes `TokenReview`, requires the configured sandbox service account,
checks the returned pod binding against the live pod UID, and verifies the pod's
controlling `Sandbox` ownerReference against the live Sandbox CR UID and
sandbox-id label before minting the gateway JWT. The bootstrap path accepts
both `agents.x-k8s.io/v1beta1` ownerReferences from newer Agent Sandbox
controllers and `agents.x-k8s.io/v1alpha1` ownerReferences from existing
deployments. Supervisors renew gateway JWTs in memory before expiry only while
the sandbox record still exists. Older tokens are not server-revoked; shared
deployments bound replay exposure with short `gateway_jwt.ttl_secs` lifetimes.
token through `RegisterSupervisorPod`. The gateway validates that projected
token with Kubernetes `TokenReview`, requires the configured sandbox service
account, and checks the returned pod binding against the live pod UID. Warm
pods that are not yet bound receive only a pending bootstrap identity and keep
the registration stream open. Later warm activation must match that pending
pod identity, the live Sandbox owner UID, and a gateway-stored opaque
activation token hash returned by the driver during sandbox creation. A missing
token mapping is retryable and leaves the pending stream intact; an empty or
mismatched token is fatal. Kubernetes sandbox-id labels are cross-checks and
diagnostics only, not authority to choose which sandbox receives the gateway
JWT. Already-bound pods still verify the pod's controlling `Sandbox`
ownerReference against the live Sandbox CR UID and sandbox-id label before
sending an activation message with the gateway JWT.
The bootstrap path accepts both `agents.x-k8s.io/v1beta1` ownerReferences from
newer Agent Sandbox controllers and `agents.x-k8s.io/v1alpha1` ownerReferences
from existing deployments. `IssueSandboxToken` remains as a compatibility shim
for older supervisor images and mints through the same already-bound pod
activation logic. Supervisors renew gateway JWTs in memory before expiry only
while the sandbox record still exists. Older tokens are not server-revoked;
shared deployments bound replay exposure with short `gateway_jwt.ttl_secs`
lifetimes.
The config default is
`gateway_jwt.ttl_secs = 0` for local single-player Docker, Podman, and VM
gateways; those tokens carry `exp = 0` and do not expire. Kubernetes and other
Expand Down Expand Up @@ -271,13 +283,16 @@ default WAL journal mode), which mirror the same sensitive contents.

Persisted state includes sandboxes, providers, provider credential refresh
state, SSH sessions, policy revisions, settings, inference configuration, and
deployment records. Provider refresh material is stored as a separate object
scoped to the provider instance through `objects.scope`; the provider record
keeps only the current injectable credential values and optional per-credential
expiry timestamps. A refresh normally mints one credential, but a strategy may
co-mint several (AWS STS mints the access key, secret key, and session token in
one call); the refresh state pins the resolved set of env keys it owns so
collision checks reserve all of them before the first mint.
deployment records. Warm-pool activation token mappings are stored as
generic objects keyed by sandbox ID with only the token hash in the payload;
sandbox deletion removes the mapping opportunistically after the sandbox row is
gone. Provider refresh material is stored as a separate object scoped to the
provider instance through `objects.scope`; the provider record keeps only the
current injectable credential values and optional per-credential expiry
timestamps. A refresh normally mints one credential, but a strategy may co-mint
several (AWS STS mints the access key, secret key, and session token in one
call); the refresh state pins the resolved set of env keys it owns so collision
checks reserve all of them before the first mint.

### Optimistic Concurrency (CAS)

Expand Down
11 changes: 11 additions & 0 deletions crates/openshell-cli/tests/ensure_providers_integration.rs
Original file line number Diff line number Diff line change
Expand Up @@ -564,6 +564,17 @@ impl OpenShell for TestOpenShell {
Err(Status::unimplemented("not implemented in test"))
}

type RegisterSupervisorPodStream = tokio_stream::wrappers::ReceiverStream<
Result<openshell_core::proto::PodActivationMessage, Status>,
>;

async fn register_supervisor_pod(
&self,
_request: tonic::Request<openshell_core::proto::RegisterSupervisorPodRequest>,
) -> Result<Response<Self::RegisterSupervisorPodStream>, Status> {
Err(Status::unimplemented("not implemented in test"))
}

async fn refresh_sandbox_token(
&self,
_request: tonic::Request<openshell_core::proto::RefreshSandboxTokenRequest>,
Expand Down
11 changes: 11 additions & 0 deletions crates/openshell-cli/tests/mtls_integration.rs
Original file line number Diff line number Diff line change
Expand Up @@ -449,6 +449,17 @@ impl OpenShell for TestOpenShell {
Err(Status::unimplemented("not implemented in test"))
}

type RegisterSupervisorPodStream = tokio_stream::wrappers::ReceiverStream<
Result<openshell_core::proto::PodActivationMessage, Status>,
>;

async fn register_supervisor_pod(
&self,
_request: tonic::Request<openshell_core::proto::RegisterSupervisorPodRequest>,
) -> Result<Response<Self::RegisterSupervisorPodStream>, Status> {
Err(Status::unimplemented("not implemented in test"))
}

async fn refresh_sandbox_token(
&self,
_request: tonic::Request<openshell_core::proto::RefreshSandboxTokenRequest>,
Expand Down
11 changes: 11 additions & 0 deletions crates/openshell-cli/tests/provider_commands_integration.rs
Original file line number Diff line number Diff line change
Expand Up @@ -967,6 +967,17 @@ impl OpenShell for TestOpenShell {
Err(Status::unimplemented("not implemented in test"))
}

type RegisterSupervisorPodStream = tokio_stream::wrappers::ReceiverStream<
Result<openshell_core::proto::PodActivationMessage, Status>,
>;

async fn register_supervisor_pod(
&self,
_request: tonic::Request<openshell_core::proto::RegisterSupervisorPodRequest>,
) -> Result<Response<Self::RegisterSupervisorPodStream>, Status> {
Err(Status::unimplemented("not implemented in test"))
}

async fn refresh_sandbox_token(
&self,
_request: tonic::Request<openshell_core::proto::RefreshSandboxTokenRequest>,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -637,6 +637,17 @@ impl OpenShell for TestOpenShell {
Err(Status::unimplemented("not implemented in test"))
}

type RegisterSupervisorPodStream = tokio_stream::wrappers::ReceiverStream<
Result<openshell_core::proto::PodActivationMessage, Status>,
>;

async fn register_supervisor_pod(
&self,
_request: tonic::Request<openshell_core::proto::RegisterSupervisorPodRequest>,
) -> Result<Response<Self::RegisterSupervisorPodStream>, Status> {
Err(Status::unimplemented("not implemented in test"))
}

async fn refresh_sandbox_token(
&self,
_request: tonic::Request<openshell_core::proto::RefreshSandboxTokenRequest>,
Expand Down
11 changes: 11 additions & 0 deletions crates/openshell-cli/tests/sandbox_name_fallback_integration.rs
Original file line number Diff line number Diff line change
Expand Up @@ -537,6 +537,17 @@ impl OpenShell for TestOpenShell {
Err(Status::unimplemented("not implemented in test"))
}

type RegisterSupervisorPodStream = tokio_stream::wrappers::ReceiverStream<
Result<openshell_core::proto::PodActivationMessage, Status>,
>;

async fn register_supervisor_pod(
&self,
_request: tonic::Request<openshell_core::proto::RegisterSupervisorPodRequest>,
) -> Result<Response<Self::RegisterSupervisorPodStream>, Status> {
Err(Status::unimplemented("not implemented in test"))
}

async fn refresh_sandbox_token(
&self,
_request: tonic::Request<openshell_core::proto::RefreshSandboxTokenRequest>,
Expand Down
Loading
Loading