diff --git a/.agents/skills/debug-openshell-cluster/SKILL.md b/.agents/skills/debug-openshell-cluster/SKILL.md index 07de687103..1e3058c853 100644 --- a/.agents/skills/debug-openshell-cluster/SKILL.md +++ b/.agents/skills/debug-openshell-cluster/SKILL.md @@ -227,6 +227,37 @@ If the gateway exits with `failed to read sandbox JWT signing key from `sandbox-jwt` secret at `/etc/openshell-jwt`. The sandbox JWT mount is required even when local Helm values disable TLS. +If `certManager.serverIssuerRef` points the server certificate at an external +Issuer or ClusterIssuer (for example an ACME issuer, for a publicly-trusted +cert on an OpenShift `Route` with TLS passthrough — see +`openshiftRoute.enabled`), check the `Certificate`/`CertificateRequest`/ +`Challenge` resources directly when the secret never becomes Ready: + +```bash +kubectl -n openshell get certificate,certificaterequest,challenge +kubectl -n openshell describe certificate openshell-server +oc -n openshell get route +``` + +ACME issuers reject certificate requests that include internal-only names +(`*.svc.cluster.local`, `localhost`, loopback IPs) and require the +`commonName` to also be a SAN — the server `Certificate` only requests the +hostnames in `certManager.serverDnsNames` when `serverIssuerRef` is set, for +exactly this reason. + +If sandbox supervisors fail their TLS handshake to the gateway with +`UnknownCA` after configuring `serverIssuerRef`, the client (mTLS) certificate +and the server certificate now come from different CAs, and the gateway's +client-verification CA is misconfigured. Set +`certManager.clientCaFromServerTlsSecret=false` and +`server.tls.clientCaSecretName` to a secret that actually contains the CA that +signs the client certificate (`certManager.caSecretName`'s value by default): + +```bash +helm -n openshell get values openshell | grep -E 'clientCaFromServerTlsSecret|clientCaSecretName|serverIssuerRef|caSecretName' +kubectl -n openshell get secret openshell-ca-tls -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -subject +``` + If `server.providerTokenGrants.spiffe.enabled=true`, the gateway should still render `[openshell.gateway.gateway_jwt]` and mount the `sandbox-jwt` Secret. SPIRE is used only by sandbox pods for dynamic provider token grants. Verify @@ -406,6 +437,8 @@ openshell logs | `K8s namespace not ready` with `envoy-gateway-openshell.yaml: the server could not find the requested resource` | Optional Gateway API manifest was applied without Envoy Gateway CRDs, or k3s Helm controller startup exceeded the namespace wait | Apply `deploy/kube/manifests/envoy-gateway-openshell.yaml` manually only after Envoy Gateway is installed and `grpcRoute` is enabled | | HTTPS ingress (`grpcRoute.gateway.listener.protocol=HTTPS`) connection resets or TLS handshake hangs | Envoy terminates TLS but the gateway pod still expects TLS, so the plaintext backend hop fails | Set `server.disableTls=true` so Envoy forwards plaintext to the pod; verify the listener `certificateRefs` Secret exists in the release namespace and `openshell status` over `https://` | | HTTPS ingress returns `Unauthenticated` after connecting | TLS terminates at Envoy, so the gateway never sees a client cert; no OIDC issuer is configured for identity | Configure `server.oidc.issuer` and register with `openshell gateway add https:// --oidc-issuer `, or set `server.auth.allowUnauthenticatedUsers=true` for a trusted-proxy/dev cluster | +| Server `Certificate` never becomes Ready with `certManager.serverIssuerRef` set | ACME issuer rejected internal-only SANs, a loopback IP, or a `commonName` absent from the SANs | `kubectl -n openshell describe certificate openshell-server`; confirm `certManager.serverDnsNames` lists only real, externally-resolvable hostnames | +| Sandbox supervisors fail TLS handshake with `UnknownCA` after configuring `certManager.serverIssuerRef` | Client (mTLS) cert and server cert now come from different CAs, and `server.tls.clientCaSecretName` still points at the wrong (or default, unpopulated) secret | Set `certManager.clientCaFromServerTlsSecret=false` and `server.tls.clientCaSecretName` to the secret named by `certManager.caSecretName` | ## Reporting diff --git a/Cargo.lock b/Cargo.lock index 268da3982d..3b1477d1c1 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -6948,6 +6948,7 @@ dependencies = [ "tower-layer", "tower-service", "tracing", + "webpki-roots 1.0.7", ] [[package]] diff --git a/crates/openshell-core/Cargo.toml b/crates/openshell-core/Cargo.toml index 53735a6bb8..0cbaf46eb6 100644 --- a/crates/openshell-core/Cargo.toml +++ b/crates/openshell-core/Cargo.toml @@ -14,7 +14,7 @@ repository.workspace = true glob = { workspace = true } prost = { workspace = true } prost-types = { workspace = true } -tonic = { workspace = true, features = ["channel", "tls-native-roots"] } +tonic = { workspace = true, features = ["channel", "tls-native-roots", "tls-webpki-roots"] } tonic-prost = { workspace = true } tokio = { workspace = true } thiserror = { workspace = true } diff --git a/crates/openshell-core/src/grpc_client.rs b/crates/openshell-core/src/grpc_client.rs index 070704fb0a..ce311fa9e6 100644 --- a/crates/openshell-core/src/grpc_client.rs +++ b/crates/openshell-core/src/grpc_client.rs @@ -167,8 +167,16 @@ async fn build_plain_channel(endpoint: &str) -> Result { .into_diagnostic() .wrap_err_with(|| format!("failed to read client key from {key_path}"))?; + // Trust the configured CA (self-signed deployments sign the gateway's + // cert with it) *in addition to* the public root stores, rather than + // instead of them — tonic's root store is a union of all configured + // sources, so this also covers deployments where the gateway's server + // cert comes from a public CA (e.g. an ACME issuer) instead of the + // same private CA that signs this client's identity. let mut tls_config = ClientTlsConfig::new() .ca_certificate(Certificate::from_pem(ca_pem)) + .with_native_roots() + .with_webpki_roots() .identity(Identity::from_pem(cert_pem, key_pem)); if let Ok(server_name) = std::env::var(sandbox_env::GATEWAY_TLS_SERVER_NAME) && !server_name.is_empty() diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 8723535d1a..1052497c50 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -144,10 +144,12 @@ add `ci/values-spire.yaml` to the OpenShell release values files. | certManager.caSecretName | string | `"openshell-ca-tls"` | Secret created for the intermediate CA (Certificate with isCA: true). | | certManager.certificateDuration | string | `"8760h"` | Duration for cert-manager-issued certificates. | | certManager.certificateRenewBefore | string | `"720h"` | Renewal window for cert-manager-issued certificates. | -| certManager.clientCaFromServerTlsSecret | bool | `true` | Mount gateway client CA from the server TLS secret's ca.crt (populated by cert-manager for certs issued by a CA Issuer). Avoids a separate openshell-server-client-ca Secret. | +| certManager.clientCaFromServerTlsSecret | bool | `true` | Mount gateway client CA from the server TLS secret's ca.crt (populated by cert-manager for certs issued by a CA Issuer). Set to false when serverIssuerRef points at an external issuer (its secret's ca.crt would be that issuer's chain, not the CA that signs the client cert). When false, also set server.tls.clientCaSecretName to a secret containing the actual client CA — for example caSecretName's value, since that's the CA the client certificate above is issued from by default. | +| certManager.clientIssuerRef | object | `{"group":"","kind":"","name":""}` | Override the issuerRef for the client (mTLS) Certificate. Client certs don't need public trust; leave empty to use the chart's own CA issuer unless an internal PKI issuer is preferred. | | certManager.enabled | bool | `false` | Create cert-manager Issuer and Certificate resources. When enabled, cert-manager owns TLS and the chart runs a JWT-only certgen hook to create the sandbox JWT signing Secret that cert-manager does not manage. | | certManager.serverDnsNames | list | `["openshell","openshell.openshell.svc","openshell.openshell.svc.cluster.local","localhost","openshell.localhost","*.openshell.localhost","host.docker.internal"]` | DNS SANs on the cert-manager-issued server certificate. | | certManager.serverIpAddresses | list | `["127.0.0.1"]` | IP SANs on the cert-manager-issued server certificate. | +| certManager.serverIssuerRef | object | `{"group":"","kind":"","name":""}` | Override the issuerRef for the server Certificate (e.g. a real ACME ClusterIssuer for a publicly-trusted cert on an external hostname). Leave name empty to use the chart's own self-signed CA issuer (default). | | fullnameOverride | string | `""` | Override the full generated resource name. | | grpcRoute.enabled | bool | `false` | Create a Gateway API GRPCRoute for the gateway service. | | grpcRoute.gateway.className | string | `"eg"` | GatewayClass to reference. Envoy Gateway installs one named "eg". | @@ -166,6 +168,9 @@ add `ci/values-spire.yaml` to the OpenShell release values files. | nameOverride | string | `"openshell"` | Override the chart name used in generated resource names. | | networkPolicy.enabled | bool | `true` | Create a NetworkPolicy restricting SSH ingress on sandbox pods to the gateway. | | nodeSelector | object | `{}` | Node selector for the gateway pod. | +| openshiftRoute.annotations | object | `{}` | Extra annotations on the Route (e.g. haproxy.router.openshift.io/*). | +| openshiftRoute.enabled | bool | `false` | Create an OpenShift Route with TLS passthrough. | +| openshiftRoute.host | string | `""` | Hostname for the Route. Must match a SAN on the gateway's server cert. | | pkiInitJob.enabled | bool | `true` | Run a pre-install/pre-upgrade Job that creates gateway and client mTLS Secrets. When certManager.enabled=true, cert-manager owns TLS and this same hook runs in JWT-only mode even if pkiInitJob.enabled remains true. | | pkiInitJob.serverDnsNames | list | `[]` | Extra DNS SANs to append to the server certificate. | | pkiInitJob.serverIpAddresses | list | `[]` | Extra IP SANs to append to the server certificate. | diff --git a/deploy/helm/openshell/ci/values-openshift-route-cert-manager.yaml b/deploy/helm/openshell/ci/values-openshift-route-cert-manager.yaml new file mode 100644 index 0000000000..d06283ee15 --- /dev/null +++ b/deploy/helm/openshell/ci/values-openshift-route-cert-manager.yaml @@ -0,0 +1,33 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Render-coverage overlay for cert-manager issuing the server certificate from +# an external Issuer/ClusterIssuer (e.g. a real ACME issuer), plus an +# OpenShift Route with TLS passthrough. Merge after values.yaml: +# helm lint deploy/helm/openshell -f ci/values-openshift-route-cert-manager.yaml +# +# The ClusterIssuer name below is a placeholder for render coverage; a real +# deployment must reference an Issuer/ClusterIssuer that's actually installed +# and Ready in the target cluster. See docs/kubernetes/managing-certificates.mdx. + +server: + disableTls: false + grpcEndpoint: "https://openshell.example.com:443" + tls: + # Must name a secret containing the actual client CA when + # clientCaFromServerTlsSecret is false — here, the same CA secret + # cert-manager creates for signing the client (mTLS) certificate. + clientCaSecretName: openshell-ca-tls + +certManager: + enabled: true + clientCaFromServerTlsSecret: false + serverIssuerRef: + name: letsencrypt-prod + kind: ClusterIssuer + serverDnsNames: + - openshell.example.com + +openshiftRoute: + enabled: true + host: openshell.example.com diff --git a/deploy/helm/openshell/templates/cert-manager-pki.yaml b/deploy/helm/openshell/templates/cert-manager-pki.yaml index fdd702a305..ae9723b5da 100644 --- a/deploy/helm/openshell/templates/cert-manager-pki.yaml +++ b/deploy/helm/openshell/templates/cert-manager-pki.yaml @@ -42,6 +42,7 @@ spec: ca: secretName: {{ .Values.certManager.caSecretName | quote }} --- +{{- $externalServerIssuer := .Values.certManager.serverIssuerRef.name }} apiVersion: cert-manager.io/v1 kind: Certificate metadata: @@ -53,7 +54,30 @@ spec: secretName: {{ .Values.server.tls.certSecretName | quote }} duration: {{ .Values.certManager.certificateDuration | quote }} renewBefore: {{ .Values.certManager.certificateRenewBefore | quote }} - commonName: openshell-server + {{- if $externalServerIssuer }} + # External issuers (e.g. ACME) reject internal-only names (cluster-local, + # localhost, loopback IPs) per CA/Browser Forum baseline requirements, and + # require any commonName to also appear in dnsNames. Only request the + # externally-resolvable SANs the operator configured — no defaults, no IPs. + {{- if .Values.certManager.serverDnsNames }} + commonName: {{ first .Values.certManager.serverDnsNames | quote }} + {{- end }} + dnsNames: + {{- range .Values.certManager.serverDnsNames }} + - {{ . | quote }} + {{- end }} + {{- else }} + # The chart fullname is always the first entry of defaultServerDnsNames + # below, so it's always valid as a commonName. + # + # Upgrade note: this replaces the previously-hardcoded "openshell-server" + # commonName. The ACME commonName-must-be-a-SAN constraint this exists for + # only applies to the external-issuer branch above, but this branch was + # changed too for consistency. Existing self-signed/internal-CA deployments + # will get a new server cert with a different Subject CN on next reissue + # after upgrading — SAN-based hostname verification (what TLS clients + # actually check) is unaffected. + commonName: {{ include "openshell.fullname" . }} dnsNames: {{- range (include "openshell.defaultServerDnsNames" . | fromYamlArray) }} - {{ . | quote }} @@ -65,6 +89,7 @@ spec: ipAddresses: {{- toYaml .Values.certManager.serverIpAddresses | nindent 4 }} {{- end }} + {{- end }} privateKey: algorithm: ECDSA size: 256 @@ -73,9 +98,15 @@ spec: - digital signature - key encipherment issuerRef: + {{- if .Values.certManager.serverIssuerRef.name }} + name: {{ .Values.certManager.serverIssuerRef.name }} + kind: {{ .Values.certManager.serverIssuerRef.kind | default "ClusterIssuer" }} + group: {{ .Values.certManager.serverIssuerRef.group | default "cert-manager.io" }} + {{- else }} name: {{ include "openshell.fullname" . }}-ca-issuer kind: Issuer group: cert-manager.io + {{- end }} --- apiVersion: cert-manager.io/v1 kind: Certificate @@ -97,7 +128,13 @@ spec: - digital signature - key encipherment issuerRef: + {{- if .Values.certManager.clientIssuerRef.name }} + name: {{ .Values.certManager.clientIssuerRef.name }} + kind: {{ .Values.certManager.clientIssuerRef.kind | default "ClusterIssuer" }} + group: {{ .Values.certManager.clientIssuerRef.group | default "cert-manager.io" }} + {{- else }} name: {{ include "openshell.fullname" . }}-ca-issuer kind: Issuer group: cert-manager.io + {{- end }} {{- end }} diff --git a/deploy/helm/openshell/templates/route.yaml b/deploy/helm/openshell/templates/route.yaml new file mode 100644 index 0000000000..26262ee7c6 --- /dev/null +++ b/deploy/helm/openshell/templates/route.yaml @@ -0,0 +1,28 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +{{- if .Values.openshiftRoute.enabled }} +apiVersion: route.openshift.io/v1 +kind: Route +metadata: + name: {{ include "openshell.fullname" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "openshell.labels" . | nindent 4 }} + {{- with .Values.openshiftRoute.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + {{- if .Values.openshiftRoute.host }} + host: {{ .Values.openshiftRoute.host }} + {{- end }} + to: + kind: Service + name: {{ include "openshell.fullname" . }} + port: + targetPort: grpc + tls: + termination: passthrough + wildcardPolicy: None +{{- end }} diff --git a/deploy/helm/openshell/tests/cert_manager_pki_test.yaml b/deploy/helm/openshell/tests/cert_manager_pki_test.yaml new file mode 100644 index 0000000000..2847e2d00b --- /dev/null +++ b/deploy/helm/openshell/tests/cert_manager_pki_test.yaml @@ -0,0 +1,140 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: cert-manager PKI issuerRef overrides +templates: + - templates/cert-manager-pki.yaml +release: + name: openshell + namespace: my-namespace + +tests: + - it: defaults both server and client Certificates to the chart's own CA issuer + template: templates/cert-manager-pki.yaml + set: + certManager.enabled: true + asserts: + - equal: + path: spec.issuerRef.name + value: openshell-ca-issuer + documentIndex: 3 + - equal: + path: spec.issuerRef.kind + value: Issuer + documentIndex: 3 + - equal: + path: spec.issuerRef.name + value: openshell-ca-issuer + documentIndex: 4 + - equal: + path: spec.issuerRef.kind + value: Issuer + documentIndex: 4 + + - it: default server Certificate includes internal SANs, IPs, and a fixed commonName + template: templates/cert-manager-pki.yaml + set: + certManager.enabled: true + asserts: + - equal: + path: spec.commonName + value: openshell + documentIndex: 3 + - contains: + path: spec.dnsNames + content: openshell.my-namespace.svc.cluster.local + documentIndex: 3 + - contains: + path: spec.dnsNames + content: localhost + documentIndex: 3 + - equal: + path: spec.ipAddresses[0] + value: 127.0.0.1 + documentIndex: 3 + + - it: overrides the server Certificate issuerRef when serverIssuerRef is set + template: templates/cert-manager-pki.yaml + set: + certManager.enabled: true + certManager.serverIssuerRef.name: letsencrypt-prod + certManager.serverIssuerRef.kind: ClusterIssuer + certManager.serverDnsNames: + - openshell.example.com + asserts: + - equal: + path: spec.issuerRef.name + value: letsencrypt-prod + documentIndex: 3 + - equal: + path: spec.issuerRef.kind + value: ClusterIssuer + documentIndex: 3 + - equal: + path: spec.issuerRef.group + value: cert-manager.io + documentIndex: 3 + + - it: scopes server Certificate SANs to only the configured external hostnames when serverIssuerRef is set + template: templates/cert-manager-pki.yaml + set: + certManager.enabled: true + certManager.serverIssuerRef.name: letsencrypt-prod + certManager.serverIssuerRef.kind: ClusterIssuer + certManager.serverDnsNames: + - openshell.example.com + asserts: + - equal: + path: spec.commonName + value: openshell.example.com + documentIndex: 3 + - equal: + path: spec.dnsNames + value: + - openshell.example.com + documentIndex: 3 + - notExists: + path: spec.ipAddresses + documentIndex: 3 + - notContains: + path: spec.dnsNames + content: localhost + documentIndex: 3 + - notContains: + path: spec.dnsNames + content: openshell.my-namespace.svc.cluster.local + documentIndex: 3 + + - it: overrides the client Certificate issuerRef when clientIssuerRef is set + template: templates/cert-manager-pki.yaml + set: + certManager.enabled: true + certManager.clientIssuerRef.name: internal-pki-issuer + certManager.clientIssuerRef.kind: ClusterIssuer + asserts: + - equal: + path: spec.issuerRef.name + value: internal-pki-issuer + documentIndex: 4 + - equal: + path: spec.issuerRef.kind + value: ClusterIssuer + documentIndex: 4 + + - it: client Certificate issuerRef is unaffected by serverIssuerRef + template: templates/cert-manager-pki.yaml + set: + certManager.enabled: true + certManager.serverIssuerRef.name: letsencrypt-prod + certManager.serverIssuerRef.kind: ClusterIssuer + certManager.serverDnsNames: + - openshell.example.com + asserts: + - equal: + path: spec.issuerRef.name + value: openshell-ca-issuer + documentIndex: 4 + - equal: + path: spec.issuerRef.kind + value: Issuer + documentIndex: 4 diff --git a/deploy/helm/openshell/tests/route_test.yaml b/deploy/helm/openshell/tests/route_test.yaml new file mode 100644 index 0000000000..b218bf3264 --- /dev/null +++ b/deploy/helm/openshell/tests/route_test.yaml @@ -0,0 +1,62 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: OpenShift Route +templates: + - templates/route.yaml +release: + name: openshell + namespace: my-namespace + +tests: + - it: renders nothing by default + asserts: + - hasDocuments: + count: 0 + + - it: renders a passthrough Route when enabled + set: + openshiftRoute.enabled: true + openshiftRoute.host: openshell.apps.example.com + asserts: + - isKind: + of: Route + - equal: + path: apiVersion + value: route.openshift.io/v1 + - equal: + path: spec.host + value: openshell.apps.example.com + - equal: + path: spec.to.kind + value: Service + - equal: + path: spec.to.name + value: openshell + - equal: + path: spec.port.targetPort + value: grpc + - equal: + path: spec.tls.termination + value: passthrough + - equal: + path: spec.wildcardPolicy + value: None + + - it: omits host when not set + set: + openshiftRoute.enabled: true + asserts: + - notExists: + path: spec.host + + - it: renders custom annotations + set: + openshiftRoute.enabled: true + openshiftRoute.host: openshell.apps.example.com + openshiftRoute.annotations: + haproxy.router.openshift.io/balance: roundrobin + asserts: + - equal: + path: metadata.annotations["haproxy.router.openshift.io/balance"] + value: roundrobin diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index e89a234912..2032df6037 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -337,7 +337,7 @@ pkiInitJob: serverIpAddresses: [] # cert-manager Certificate/Issuer resources (requires cert-manager CRDs in-cluster). -# Uses namespaced Issuers only (no ClusterIssuer). Does not install cert-manager itself. +# Does not install cert-manager itself. certManager: # -- Create cert-manager Issuer and Certificate resources. When enabled, # cert-manager owns TLS and the chart runs a JWT-only certgen hook to create @@ -345,9 +345,27 @@ certManager: enabled: false # -- Secret created for the intermediate CA (Certificate with isCA: true). caSecretName: openshell-ca-tls + # -- Override the issuerRef for the server Certificate (e.g. a real ACME + # ClusterIssuer for a publicly-trusted cert on an external hostname). Leave + # name empty to use the chart's own self-signed CA issuer (default). + serverIssuerRef: + name: "" + kind: "" + group: "" + # -- Override the issuerRef for the client (mTLS) Certificate. Client certs + # don't need public trust; leave empty to use the chart's own CA issuer + # unless an internal PKI issuer is preferred. + clientIssuerRef: + name: "" + kind: "" + group: "" # -- Mount gateway client CA from the server TLS secret's ca.crt (populated by - # cert-manager for certs issued by a CA Issuer). Avoids a separate - # openshell-server-client-ca Secret. + # cert-manager for certs issued by a CA Issuer). Set to false when + # serverIssuerRef points at an external issuer (its secret's ca.crt would be + # that issuer's chain, not the CA that signs the client cert). When false, + # also set server.tls.clientCaSecretName to a secret containing the actual + # client CA — for example caSecretName's value, since that's the CA the + # client certificate above is issued from by default. clientCaFromServerTlsSecret: true # -- Duration for cert-manager-issued certificates. certificateDuration: 8760h @@ -407,3 +425,15 @@ grpcRoute: # or the existing openshell-server-tls Secret (its SANs must include the # external hostname). certificateRefs: [] + +# OpenShift Route with TLS passthrough. The gateway terminates its own +# TLS/mTLS; the router only forwards based on SNI, so it never sees plaintext +# or the client certificate. Requires server.disableTls=false and a server +# cert whose SANs include the Route host (see certManager.serverIssuerRef). +openshiftRoute: + # -- Create an OpenShift Route with TLS passthrough. + enabled: false + # -- Hostname for the Route. Must match a SAN on the gateway's server cert. + host: "" + # -- Extra annotations on the Route (e.g. haproxy.router.openshift.io/*). + annotations: {} diff --git a/docs/kubernetes/managing-certificates.mdx b/docs/kubernetes/managing-certificates.mdx index b66419b505..91012be91a 100644 --- a/docs/kubernetes/managing-certificates.mdx +++ b/docs/kubernetes/managing-certificates.mdx @@ -60,6 +60,56 @@ The chart also runs a pre-install hook in JWT-only mode to create the gateway's sandbox JWT signing Secret. That Secret is separate from the cert-manager TLS certificate Secrets and is mounted at `/etc/openshell-jwt`. +## Using a real Issuer for the server certificate + +By default, cert-manager issues both the server and client certificates from +a self-signed CA the chart creates — this rotates automatically, but the +server certificate is still not publicly trusted. `certManager.serverIssuerRef` +overrides the `issuerRef` on the server `Certificate` resource to point at a +real `Issuer` or `ClusterIssuer` instead, for example an ACME issuer: + +```shell +helm upgrade --install openshell \ + oci://ghcr.io/nvidia/openshell/helm-chart \ + --version \ + --namespace openshell \ + --set certManager.enabled=true \ + --set certManager.clientCaFromServerTlsSecret=false \ + --set server.tls.clientCaSecretName=openshell-ca-tls \ + --set certManager.serverIssuerRef.name=letsencrypt-prod \ + --set certManager.serverIssuerRef.kind=ClusterIssuer \ + --set certManager.serverDnsNames[0]=openshell.example.com +``` + + +Public CAs such as Let's Encrypt reject certificate requests that include +internal-only names (`*.svc.cluster.local`, `localhost`, loopback IPs) per +CA/Browser Forum baseline requirements. When `serverIssuerRef` is set, the +server certificate's SANs are limited to the hostnames in +`certManager.serverDnsNames` — the chart's usual internal cluster-local SANs +are omitted. Sandboxes calling back into the gateway must then use the same +external hostname; see `server.grpcEndpoint` in +[Gateway Configuration](/reference/gateway-config). + + +Set `certManager.clientCaFromServerTlsSecret=false` whenever `serverIssuerRef` +is set, and set `server.tls.clientCaSecretName` to name a secret that actually +contains the client CA — by default that's the same secret named by +`certManager.caSecretName` (`openshell-ca-tls`), since that's the CA the chart +issues the client certificate from unless you've also overridden +`clientIssuerRef`. The client (mTLS) certificate used by sandbox supervisors +stays on the chart's own internal CA — it doesn't need public trust, and +getting a public CA to sign it isn't practical: ACME only validates domain +identifiers, not arbitrary workload identity, and issuing one certificate per +sandbox would put a public CA's issuance rate limits directly in the +sandbox-creation path. + +`certManager.clientIssuerRef` is also available if you'd rather sign the +client certificate from an internal PKI issuer of your own instead of the +chart's self-signed CA — most deployments can leave it unset. + ## Next Steps -Return to [Setup](/kubernetes/setup) to complete the installation. +Return to [Setup](/kubernetes/setup) to complete the installation. For +exposing the gateway externally on OpenShift with a real certificate, see +[OpenShift](/kubernetes/openshift). diff --git a/docs/kubernetes/openshift.mdx b/docs/kubernetes/openshift.mdx index 7512eaa65e..d947ea3efc 100644 --- a/docs/kubernetes/openshift.mdx +++ b/docs/kubernetes/openshift.mdx @@ -87,8 +87,55 @@ openshell gateway add http://127.0.0.1:8080 --local --name openshift openshell status ``` +## Production: expose externally with a real certificate + +The steps above run the gateway over plaintext HTTP for quick evaluation. For +a real deployment, cert-manager can issue the gateway's server certificate +from a real Issuer or ClusterIssuer (for example, an ACME issuer), and an +OpenShift Route with TLS passthrough exposes it externally while the gateway +keeps terminating its own TLS and mTLS. + +Install cert-manager and configure a working `ClusterIssuer` first, then see +[Managing Certificates](/kubernetes/managing-certificates) for the +`certManager.serverIssuerRef` details. Install the chart with: + +```shell +helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \ + --version \ + --namespace openshell \ + --set podSecurityContext.fsGroup=null \ + --set securityContext.runAsUser=null \ + --set server.disableTls=false \ + --set server.grpcEndpoint=https://:443 \ + --set certManager.enabled=true \ + --set certManager.clientCaFromServerTlsSecret=false \ + --set server.tls.clientCaSecretName=openshell-ca-tls \ + --set certManager.serverIssuerRef.name= \ + --set certManager.serverIssuerRef.kind=ClusterIssuer \ + --set certManager.serverDnsNames[0]= \ + --set openshiftRoute.enabled=true \ + --set openshiftRoute.host= +``` + +| Override | Reason | +|---|---| +| `server.grpcEndpoint` | Sandboxes call back into the gateway using this hostname. A cert from an external issuer only carries the external SANs you configure, not the internal cluster-local ones, so sandboxes must use the same externally-valid hostname to pass TLS verification. | +| `certManager.clientCaFromServerTlsSecret=false` + `server.tls.clientCaSecretName` | The server cert's issuer no longer shares a CA with the client (mTLS) cert, so this points the gateway's client-verification CA at `certManager.caSecretName` (the chart's own CA secret, `openshell-ca-tls` by default) instead of the server secret. | +| `certManager.serverIssuerRef` | Points the server certificate at your real Issuer or ClusterIssuer instead of the chart's built-in self-signed CA. | +| `openshiftRoute.enabled` / `openshiftRoute.host` | Creates an OpenShift Route with TLS passthrough — the router forwards the encrypted connection by SNI without decrypting, so the gateway keeps terminating its own TLS and mTLS. | + +Register the gateway with the CLI over OIDC. Remote gateways authenticate CLI +users via OIDC, not mTLS — see [Access Control](/kubernetes/access-control): + +```shell +openshell gateway add https:// \ + --name openshift \ + --oidc-issuer +openshell gateway login openshift +``` + ## Next Steps -- For TLS-enabled deployments, refer to [Managing Certificates](/kubernetes/managing-certificates). -- To expose the gateway externally, refer to [Ingress](/kubernetes/ingress). +- For more on certificate provisioning modes, refer to [Managing Certificates](/kubernetes/managing-certificates). +- To expose the gateway externally through the Kubernetes Gateway API instead of a Route, refer to [Ingress](/kubernetes/ingress). - To configure OIDC authentication, refer to [Access Control](/kubernetes/access-control).