Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .agents/skills/openshell-cli/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -422,6 +422,8 @@ The `--from` flag accepts a Dockerfile path, a directory containing a Dockerfile

Local Dockerfile and directory builds require a local gateway because the CLI builds through the local Docker daemon. Use a registry image reference for remote gateways. Bare community names resolve under `ghcr.io/nvidia/openshell-community/sandboxes` unless `OPENSHELL_COMMUNITY_REGISTRY` overrides the prefix.

For Docker and Podman, local `image` identity mode requires the image to declare a non-root OCI `USER`. Passwd-less images must use an explicit numeric pair such as `USER 1234:1235`. External bind mounts and named volumes require operator-configured `fixed` identity mode. Kubernetes and VM gateways retain their driver-specific identity behavior.

### Forward ports

```bash
Expand Down
121 changes: 112 additions & 9 deletions architecture/compute-runtimes.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,15 +73,118 @@ template resource limits. Docker and Podman apply them as runtime limits.
Kubernetes mirrors each limit into the matching request. VM accepts the fields
but currently ignores them.

Docker and Podman also accept per-sandbox driver-config mounts for existing
runtime-managed named volumes and tmpfs mounts. Podman additionally accepts
image mounts through its image-volume API. User-supplied bind and volume mounts
default to read-only. Direct host bind mounts, and Docker or Podman local-driver
bind-backed named volumes, are available only when explicitly enabled in the
active local driver table of `gateway.toml`. Host bind mounts are an unsafe
operator override because they place gateway-host filesystem state inside the
sandbox and can negate OpenShell workspace isolation and filesystem-policy
controls. Driver-owned supervisor, token, and TLS bind mounts stay reserved.
## Docker and Podman Agent Identity

By default, Docker and Podman preserve the selected sandbox image's non-root OCI
`USER` for agent processes. This local-container contract does not run the
supervisor as that user: the supervisor starts as root, prepares the isolation
boundary, and then drops privileges for direct and SSH-launched agent children.

### Image Selection and Startup

The local drivers bind identity inspection to the root filesystem that the
container engine starts. Docker implements this flow in
`crates/openshell-driver-docker/src/lib.rs`; Podman implements it in
`crates/openshell-driver-podman/src/driver.rs` and
`crates/openshell-driver-podman/src/container.rs`. Supervisor resolution and
persistence live in `crates/openshell-supervisor-process/src/identity.rs`.

```mermaid
sequenceDiagram
participant Gateway
participant Driver as Docker/Podman driver
participant Engine as Container engine
participant Supervisor
participant Agent as Agent child

Gateway->>Driver: CreateSandbox(spec)
Driver->>Engine: Resolve or pull requested image
Driver->>Engine: Inspect selected image
Engine-->>Driver: Immutable image ID + OCI Config.User
Driver->>Engine: Create by immutable ID with root supervisor
Engine->>Supervisor: Start in selected image rootfs
Supervisor->>Supervisor: Resolve and persist UID/GID
Supervisor->>Gateway: ConnectSupervisor(resolved identity)
Gateway-->>Supervisor: Persisted identity accepted
Supervisor->>Agent: Clear groups, setgid, setuid, exec
```

Docker inspects a local hit or the result of a pull and passes the returned image
ID, rather than the mutable reference, to container creation. Podman pulls
according to policy, inspects the selected result, and creates with
`image_pull_policy = "never"` and the inspected ID. Podman applies the same
inspection and pinning to its supervisor image and requested image mounts. A tag
can therefore move after inspection without changing the rootfs selected for
that sandbox.

Both drivers insert the identity source, immutable image ID, and required
identity inputs after image and request environment, so the sandbox image cannot
override them. Docker sets the container user to `0`; Podman sets it to `0:0`.
Those values apply only to the supervisor, which needs root for namespace,
mount, network, filesystem, and privilege-drop setup.

### Identity Modes

The active Docker or Podman driver configuration selects one identity mode.

| Mode | Agent identity | Storage boundary |
|---|---|---|
| `image` (default) | Resolve the inspected image's raw OCI `Config.User` to one non-root UID/GID. | Reject creator-selected bind and named-volume mounts. The driver-owned workspace and tmpfs mounts remain available; Podman also permits read-only image mounts pinned to inspected image IDs. |
| `fixed` | Use the operator-configured `fixed_uid` and `fixed_gid`; both must be present and within the process-policy numeric range. | Permit external or shared storage subject to the normal mount and bind-enable checks. |

Image mode accepts named users and groups from the image's `/etc/passwd` and
`/etc/group`, or an explicit numeric `USER <uid>:<gid>` without account entries.
A numeric UID without a group requires a matching passwd entry from which to
obtain the primary GID. Missing users or groups and any identity resolving to
UID or GID 0 fail sandbox startup. The supervisor uses bounded, file-only
account parsing rather than NSS and does not create or rewrite image account
files.

Before the first agent starts, the supervisor stores the resolved identity in
`/var/lib/openshell/agent-identity.json`, bound to the immutable image ID and,
for image mode, the raw OCI user declaration. It reports the same versioned
record to the gateway. The gateway persists the source, image ID, UID, GID, and
empty supplementary-group set and rejects a missing or changed record on later
connections. Restarts reuse the persisted identity rather than resolving a
mutable tag again.

The gateway marks new Docker and Podman sandboxes as requiring this managed
identity and rejects creator-supplied `process.run_as_user` or
`process.run_as_group`; those policy fields cannot override the runtime's
selection. Existing containers without the creation marker retain their legacy
policy-derived identity and are not reinterpreted during upgrade.

### External Mount Boundary

Docker and Podman accept per-sandbox driver-config mounts for existing named
volumes and tmpfs mounts; Podman also accepts OCI image mounts. User-supplied
bind and volume mounts default to read-only, but read-only access does not make
external storage safe for an image-selected principal. Image mode therefore
rejects every creator-selected host bind and named volume even when the global
bind-mount switch is enabled. Operators must select fixed mode to bind external
or shared storage to an operator-controlled UID/GID.

Fixed mode is necessary but not sufficient for direct host access. Host binds,
including local-driver bind-backed named volumes, still require
`enable_bind_mounts` in the active local driver table of `gateway.toml`.
Requested named volumes must already exist, and the driver inspects them to
detect bind-backed storage. User mounts cannot replace the driver-owned
workspace root, supervisor, token, or TLS paths, and cannot overlap
identity-sensitive account, identity-state, or UID/GID-map paths.

Host bind mounts remain an unsafe operator override because they place gateway
host filesystem state inside the sandbox and can negate workspace isolation and
filesystem-policy controls.

### Kubernetes, OpenShift, and VM

The managed image-identity flow applies only to Docker and Podman. Kubernetes
continues to use an explicitly configured sandbox UID/GID, an OpenShift SCC
namespace-assigned range when available, or its existing numeric defaults. The
VM driver continues to use its configured numeric guest UID/GID, defaulting to
`10001` and using the UID as the GID when no GID is configured. These runtimes
do not derive agent identity from OCI `Config.User` and do not participate in
the Docker/Podman resolved-identity handshake.

Kubernetes deployments may set an AppArmor profile on sandbox agent containers
through the driver configuration. The Helm chart defaults sandbox agents to
Expand Down
93 changes: 79 additions & 14 deletions architecture/sandbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ Each sandbox workload has two trust levels:

| Process | Role |
|---|---|
| Supervisor | Starts as root inside the workload, prepares isolation, runs the proxy, fetches config, injects credentials, serves the relay socket, and launches child processes. |
| Agent child | Runs as an unprivileged user with filesystem, process, and network restrictions applied. |
| Supervisor | Runs as root inside the workload, prepares isolation, runs the proxy, fetches config, injects credentials, serves the relay socket, and launches child processes. |
| Agent child | Runs as the selected unprivileged identity with filesystem, process, and network restrictions applied. Managed local children use a resolved numeric UID and GID. |

The supervisor keeps enough privilege to manage the sandbox, but the agent child
loses that privilege before user code runs. On Linux, child setup clears the
Expand All @@ -21,18 +21,82 @@ container-granted capabilities. This is fail-closed: the supervisor retains
aborts unless the bounding set ends up empty. A `setpcap` `EPERM` is tolerated
only when the set is already empty; any other outcome fails the spawn.

### Managed local identity

Docker and Podman select the agent identity; the gateway rejects creator-set
`process.run_as_user` or `process.run_as_group` values for these runtimes. Their
default `image` mode inspects the selected image's immutable ID and OCI
`Config.User`. Operators can instead configure `fixed` mode with both a numeric
UID and GID in the policy identity range. Fixed mode is explicit and is the mode
used when external storage requires an operator-controlled identity; image mode
rejects bind and named-volume mounts that could have incompatible ownership.

Both local drivers override the image entrypoint and OCI user so the supervisor
starts as `0:0`. They pass the inspected image metadata or fixed IDs through
driver-owned environment entries that template, request, and image environment
values cannot replace. The supervisor then resolves the identity once and
normalizes the process policy to its numeric UID and GID before launching any
agent child.

Image mode accepts only a non-empty OCI `USER` in `user`, `uid`, `user:group`,
or `uid:gid` form. Named fields and an implicit primary group must resolve from
the image's `/etc/passwd` and `/etc/group`; missing, ambiguous, malformed, or
root-resolving identities fail startup. Resolution parses bounded, regular
account files directly without following symlinks or invoking NSS. OpenShell
does not add accounts or modify the image's passwd or group files. Fixed mode
does not require account-file entries, but still rejects UID or GID 0.

The first successful resolution is persisted at
`/var/lib/openshell/agent-identity.json` and bound to the immutable image ID,
identity source, and source-specific inputs. A restart fails if those inputs do
not match the persisted record. Managed children receive no supplementary
groups: child setup clears and verifies the group list before `setgid` and
`setuid`, and aborts the launch if it cannot prove the list is empty.

The gateway persists `SandboxStatus.managed_identity_required = true` when it
creates a Docker- or Podman-managed sandbox. This creation-time marker is the
only authorization for the supervisor to resolve and report a managed local
identity. Unmarked legacy, offline, Kubernetes, and VM sandboxes do not
interpret Docker/Podman source and image metadata. Kubernetes masks and VM
strips those environment keys as defense in depth, so an image cannot opt either
runtime into this protocol.

Before exposing SSH or launching the direct entrypoint, a marked supervisor
sends the resolved identity in `SupervisorHello`. The gateway validates the
schema version, source, non-empty immutable image ID, non-root UID/GID, and
empty supplementary-group list. It atomically stores the first identity in
`SandboxStatus.resolved_identity`; later sessions must match it exactly. The
supervisor proceeds only after `SessionAccepted`, and fails closed if the
gateway rejects the identity or does not accept it before the startup timeout.
The supervisor also emits an informational OCSF configuration-state event with
the resolved source, image ID, UID, GID, and empty supplementary-group list.

Filesystem preparation changes ownership only where required. For a
driver-selected identity, it changes the `/sandbox` directory itself but never
recursively traverses image content or nested mounts. Existing policy
`read_write` paths retain their ownership; a missing final directory is created
and owned through descriptor-relative, no-follow operations that reject symlink
races.
The implementation lives in
`crates/openshell-supervisor-process/src/identity.rs` and
`crates/openshell-supervisor-process/src/process.rs`.

## Startup Flow

1. The compute runtime starts the workload with sandbox identity, callback
endpoint, TLS or secret material, image metadata, and initial command.
1. The compute runtime starts the root supervisor with sandbox identity,
callback endpoint, TLS or secret material, image metadata, and initial
command.
2. The supervisor loads policy and runtime settings from local files or the
gateway, depending on mode.
3. It prepares filesystem access, process restrictions, network namespace
routing, trust stores, provider credential resolution, and inference routes.
4. It starts the policy proxy and local SSH server.
5. It opens a supervisor session back to the gateway for connect, exec, file
sync, config polling, and log push.
6. It launches the agent command as the restricted sandbox user.
3. For a marked local sandbox, it resolves and persists the numeric agent
identity. It then prepares filesystem access, process restrictions, network
namespace routing, trust stores, provider credentials, and inference routes.
4. It starts the policy proxy and opens the outbound supervisor session. A
marked sandbox waits for gateway identity acceptance.
5. It starts the local SSH server and launches the agent command. Direct and SSH
children drop to the same resolved identity before executing user code.
6. The supervisor session carries connect, exec, file sync, config polling, and
log traffic for the lifetime of the sandbox.

## Isolation Layers

Expand All @@ -41,7 +105,7 @@ OpenShell uses overlapping controls rather than a single sandbox primitive:
| Layer | Purpose |
|---|---|
| Filesystem policy | Landlock restricts the paths the agent can read or write. |
| Process policy | The child process runs as a non-root user with reduced privileges. |
| Process policy | The child process runs as a non-root identity without retained capabilities. Managed local children use numeric UID/GID and no supplementary groups. |
| Seccomp | Blocks dangerous syscalls, including raw socket paths that bypass the proxy. |
| Network namespace | Forces ordinary agent egress through the local CONNECT proxy. |
| Policy proxy | Evaluates destination, binary identity, TLS/L7 rules, SSRF checks, and inference interception. |
Expand Down Expand Up @@ -174,9 +238,10 @@ operator decision.

Provider credentials are stored at the gateway and fetched by the supervisor at
runtime. The supervisor injects resolved environment variables into the initial
agent process and SSH child processes. Driver-controlled environment variables
override template values so sandbox images cannot spoof identity, callback, or
relay settings.
agent process and SSH child processes. Driver-controlled bootstrap values
override template values so sandbox images cannot spoof callback or relay
settings; managed local identity uses the creation marker and acceptance
protocol described above rather than trusting environment presence alone.

Supervisor bootstrap identity is not inherited by agent child processes. When
provider token grants mount a SPIFFE Workload API socket, the socket path must
Expand Down
66 changes: 66 additions & 0 deletions crates/openshell-core/src/driver_mounts.rs
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,15 @@ const RESERVED_MOUNT_TARGETS: &[&str] = &[
"/run/netns",
];

const IDENTITY_SENSITIVE_MOUNT_TARGETS: &[&str] = &[
"/etc/passwd",
"/etc/group",
"/var/lib/openshell",
"/proc/self/uid_map",
"/proc/self/gid_map",
"/proc/self/setgroups",
];

/// Validate a non-empty driver mount source.
pub fn validate_mount_source(source: &str, field: &str) -> Result<(), String> {
if source.is_empty() {
Expand Down Expand Up @@ -124,6 +133,15 @@ pub fn validate_container_mount_target(target: &str) -> Result<(), String> {
));
}
}
for sensitive in IDENTITY_SENSITIVE_MOUNT_TARGETS {
let sensitive = Path::new(sensitive);
if path_is_or_under(path, sensitive) || path_is_or_under(sensitive, path) {
return Err(format!(
"mount target '{target}' overlaps identity-sensitive path '{}'",
sensitive.display()
));
}
}
Ok(())
}

Expand Down Expand Up @@ -176,6 +194,54 @@ mod tests {
validate_container_mount_target("/etc/openshell-tools").unwrap();
}

#[test]
fn container_target_rejects_identity_sensitive_paths_and_descendants() {
for &sensitive in IDENTITY_SENSITIVE_MOUNT_TARGETS {
for target in [sensitive.to_string(), format!("{sensitive}/shadowed")] {
let err = validate_container_mount_target(&target).unwrap_err();

assert!(
err.contains("overlaps identity-sensitive path"),
"unexpected error for {target}: {err}"
);
assert!(
err.contains(sensitive),
"error for {target} does not identify {sensitive}: {err}"
);
}
}
}

#[test]
fn container_target_rejects_identity_sensitive_path_ancestors() {
for target in ["/etc", "/var", "/var/lib", "/proc", "/proc/self"] {
let err = validate_container_mount_target(target).unwrap_err();

assert!(
err.contains("overlaps identity-sensitive path"),
"unexpected error for {target}: {err}"
);
}
}

#[test]
fn container_target_allows_identity_path_prefix_near_misses() {
for target in [
"/etc-agent",
"/etc/passwd-cache",
"/etc/group.d",
"/var/lib/openshell-data",
"/proc-agent",
"/proc/selfish",
"/proc/self/uid_map-backup",
"/proc/self/gid_map-backup",
"/proc/self/setgroups-backup",
] {
validate_container_mount_target(target)
.unwrap_or_else(|err| panic!("unexpected error for {target}: {err}"));
}
}

#[test]
fn mount_subpath_must_be_relative_without_parent_dirs() {
assert!(validate_mount_subpath("project/a").is_ok());
Expand Down
17 changes: 17 additions & 0 deletions crates/openshell-core/src/grpc_client.rs
Original file line number Diff line number Diff line change
Expand Up @@ -780,6 +780,8 @@ pub struct SettingsPollResult {
pub supervisor_middleware_services: Vec<crate::proto::SupervisorMiddlewareService>,
/// Workspace the sandbox belongs to.
pub workspace: String,
/// Whether the gateway requires managed runtime identity for this sandbox.
pub managed_identity_required: bool,
}

fn settings_poll_result(inner: crate::proto::GetSandboxConfigResponse) -> SettingsPollResult {
Expand All @@ -795,6 +797,21 @@ fn settings_poll_result(inner: crate::proto::GetSandboxConfigResponse) -> Settin
provider_env_revision: inner.provider_env_revision,
supervisor_middleware_services: inner.supervisor_middleware_services,
workspace: inner.workspace,
managed_identity_required: inner.managed_identity_required,
}
}

#[cfg(test)]
mod settings_snapshot_tests {
use super::*;

#[test]
fn settings_snapshot_propagates_managed_identity_requirement() {
let snapshot = settings_poll_result(crate::proto::GetSandboxConfigResponse {
managed_identity_required: true,
..Default::default()
});
assert!(snapshot.managed_identity_required);
}
}

Expand Down
Loading
Loading