feat(helm): cert-manager external issuer + OpenShift passthrough Route

.agents/skills/debug-openshell-cluster/SKILL.md

@@ -227,6 +227,37 @@ If the gateway exits with `failed to read sandbox JWT signing key from
 `sandbox-jwt` secret at `/etc/openshell-jwt`. The sandbox JWT mount is required
 even when local Helm values disable TLS.
 
+If `certManager.serverIssuerRef` points the server certificate at an external
+Issuer or ClusterIssuer (for example an ACME issuer, for a publicly-trusted
+cert on an OpenShift `Route` with TLS passthrough — see
+`openshiftRoute.enabled`), check the `Certificate`/`CertificateRequest`/
+`Challenge` resources directly when the secret never becomes Ready:
+
+```bash
+kubectl -n openshell get certificate,certificaterequest,challenge
+kubectl -n openshell describe certificate openshell-server
+oc -n openshell get route
+```
+
+ACME issuers reject certificate requests that include internal-only names
+(`*.svc.cluster.local`, `localhost`, loopback IPs) and require the
+`commonName` to also be a SAN — the server `Certificate` only requests the
+hostnames in `certManager.serverDnsNames` when `serverIssuerRef` is set, for
+exactly this reason.
+
+If sandbox supervisors fail their TLS handshake to the gateway with
+`UnknownCA` after configuring `serverIssuerRef`, the client (mTLS) certificate
+and the server certificate now come from different CAs, and the gateway's
+client-verification CA is misconfigured. Set
+`certManager.clientCaFromServerTlsSecret=false` and
+`server.tls.clientCaSecretName` to a secret that actually contains the CA that
+signs the client certificate (`certManager.caSecretName`'s value by default):
+
+```bash
+helm -n openshell get values openshell | grep -E 'clientCaFromServerTlsSecret|clientCaSecretName|serverIssuerRef|caSecretName'
+kubectl -n openshell get secret openshell-ca-tls -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -subject
+```
+
 If `server.providerTokenGrants.spiffe.enabled=true`, the gateway should still
 render `[openshell.gateway.gateway_jwt]` and mount the `sandbox-jwt` Secret.
 SPIRE is used only by sandbox pods for dynamic provider token grants. Verify
@@ -406,6 +437,8 @@ openshell logs <sandbox-name>
 | `K8s namespace not ready` with `envoy-gateway-openshell.yaml: the server could not find the requested resource` | Optional Gateway API manifest was applied without Envoy Gateway CRDs, or k3s Helm controller startup exceeded the namespace wait | Apply `deploy/kube/manifests/envoy-gateway-openshell.yaml` manually only after Envoy Gateway is installed and `grpcRoute` is enabled |
 | HTTPS ingress (`grpcRoute.gateway.listener.protocol=HTTPS`) connection resets or TLS handshake hangs | Envoy terminates TLS but the gateway pod still expects TLS, so the plaintext backend hop fails | Set `server.disableTls=true` so Envoy forwards plaintext to the pod; verify the listener `certificateRefs` Secret exists in the release namespace and `openshell status` over `https://<host>` |
 | HTTPS ingress returns `Unauthenticated` after connecting | TLS terminates at Envoy, so the gateway never sees a client cert; no OIDC issuer is configured for identity | Configure `server.oidc.issuer` and register with `openshell gateway add https://<host> --oidc-issuer <url>`, or set `server.auth.allowUnauthenticatedUsers=true` for a trusted-proxy/dev cluster |
+| Server `Certificate` never becomes Ready with `certManager.serverIssuerRef` set | ACME issuer rejected internal-only SANs, a loopback IP, or a `commonName` absent from the SANs | `kubectl -n openshell describe certificate openshell-server`; confirm `certManager.serverDnsNames` lists only real, externally-resolvable hostnames |
+| Sandbox supervisors fail TLS handshake with `UnknownCA` after configuring `certManager.serverIssuerRef` | Client (mTLS) cert and server cert now come from different CAs, and `server.tls.clientCaSecretName` still points at the wrong (or default, unpopulated) secret | Set `certManager.clientCaFromServerTlsSecret=false` and `server.tls.clientCaSecretName` to the secret named by `certManager.caSecretName` |
 
 ## Reporting
 

crates/openshell-core/Cargo.toml

@@ -14,7 +14,7 @@ repository.workspace = true
 glob = { workspace = true }
 prost = { workspace = true }
 prost-types = { workspace = true }
-tonic = { workspace = true, features = ["channel", "tls-native-roots"] }
+tonic = { workspace = true, features = ["channel", "tls-native-roots", "tls-webpki-roots"] }
 tonic-prost = { workspace = true }
 tokio = { workspace = true }
 thiserror = { workspace = true }

crates/openshell-core/src/grpc_client.rs

@@ -167,8 +167,16 @@ async fn build_plain_channel(endpoint: &str) -> Result<Channel> {
             .into_diagnostic()
             .wrap_err_with(|| format!("failed to read client key from {key_path}"))?;
 
+        // Trust the configured CA (self-signed deployments sign the gateway's
+        // cert with it) *in addition to* the public root stores, rather than
+        // instead of them — tonic's root store is a union of all configured
+        // sources, so this also covers deployments where the gateway's server
+        // cert comes from a public CA (e.g. an ACME issuer) instead of the
+        // same private CA that signs this client's identity.
         let mut tls_config = ClientTlsConfig::new()
             .ca_certificate(Certificate::from_pem(ca_pem))
+            .with_native_roots()
+            .with_webpki_roots()
             .identity(Identity::from_pem(cert_pem, key_pem));
         if let Ok(server_name) = std::env::var(sandbox_env::GATEWAY_TLS_SERVER_NAME)
             && !server_name.is_empty()

deploy/helm/openshell/README.md

@@ -144,10 +144,12 @@ add `ci/values-spire.yaml` to the OpenShell release values files.
 | certManager.caSecretName | string | `"openshell-ca-tls"` | Secret created for the intermediate CA (Certificate with isCA: true). |
 | certManager.certificateDuration | string | `"8760h"` | Duration for cert-manager-issued certificates. |
 | certManager.certificateRenewBefore | string | `"720h"` | Renewal window for cert-manager-issued certificates. |
-| certManager.clientCaFromServerTlsSecret | bool | `true` | Mount gateway client CA from the server TLS secret's ca.crt (populated by cert-manager for certs issued by a CA Issuer). Avoids a separate openshell-server-client-ca Secret. |
+| certManager.clientCaFromServerTlsSecret | bool | `true` | Mount gateway client CA from the server TLS secret's ca.crt (populated by cert-manager for certs issued by a CA Issuer). Set to false when serverIssuerRef points at an external issuer (its secret's ca.crt would be that issuer's chain, not the CA that signs the client cert). When false, also set server.tls.clientCaSecretName to a secret containing the actual client CA — for example caSecretName's value, since that's the CA the client certificate above is issued from by default. |
+| certManager.clientIssuerRef | object | `{"group":"","kind":"","name":""}` | Override the issuerRef for the client (mTLS) Certificate. Client certs don't need public trust; leave empty to use the chart's own CA issuer unless an internal PKI issuer is preferred. |
 | certManager.enabled | bool | `false` | Create cert-manager Issuer and Certificate resources. When enabled, cert-manager owns TLS and the chart runs a JWT-only certgen hook to create the sandbox JWT signing Secret that cert-manager does not manage. |
 | certManager.serverDnsNames | list | `["openshell","openshell.openshell.svc","openshell.openshell.svc.cluster.local","localhost","openshell.localhost","*.openshell.localhost","host.docker.internal"]` | DNS SANs on the cert-manager-issued server certificate. |
 | certManager.serverIpAddresses | list | `["127.0.0.1"]` | IP SANs on the cert-manager-issued server certificate. |
+| certManager.serverIssuerRef | object | `{"group":"","kind":"","name":""}` | Override the issuerRef for the server Certificate (e.g. a real ACME ClusterIssuer for a publicly-trusted cert on an external hostname). Leave name empty to use the chart's own self-signed CA issuer (default). |
 | fullnameOverride | string | `""` | Override the full generated resource name. |
 | grpcRoute.enabled | bool | `false` | Create a Gateway API GRPCRoute for the gateway service. |
 | grpcRoute.gateway.className | string | `"eg"` | GatewayClass to reference. Envoy Gateway installs one named "eg". |
@@ -166,6 +168,9 @@ add `ci/values-spire.yaml` to the OpenShell release values files.
 | nameOverride | string | `"openshell"` | Override the chart name used in generated resource names. |
 | networkPolicy.enabled | bool | `true` | Create a NetworkPolicy restricting SSH ingress on sandbox pods to the gateway. |
 | nodeSelector | object | `{}` | Node selector for the gateway pod. |
+| openshiftRoute.annotations | object | `{}` | Extra annotations on the Route (e.g. haproxy.router.openshift.io/*). |
+| openshiftRoute.enabled | bool | `false` | Create an OpenShift Route with TLS passthrough. |
+| openshiftRoute.host | string | `""` | Hostname for the Route. Must match a SAN on the gateway's server cert. |
 | pkiInitJob.enabled | bool | `true` | Run a pre-install/pre-upgrade Job that creates gateway and client mTLS Secrets. When certManager.enabled=true, cert-manager owns TLS and this same hook runs in JWT-only mode even if pkiInitJob.enabled remains true. |
 | pkiInitJob.serverDnsNames | list | `[]` | Extra DNS SANs to append to the server certificate. |
 | pkiInitJob.serverIpAddresses | list | `[]` | Extra IP SANs to append to the server certificate. |

deploy/helm/openshell/ci/values-openshift-route-cert-manager.yaml

@@ -0,0 +1,33 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+# Render-coverage overlay for cert-manager issuing the server certificate from
+# an external Issuer/ClusterIssuer (e.g. a real ACME issuer), plus an
+# OpenShift Route with TLS passthrough. Merge after values.yaml:
+#   helm lint deploy/helm/openshell -f ci/values-openshift-route-cert-manager.yaml
+#
+# The ClusterIssuer name below is a placeholder for render coverage; a real
+# deployment must reference an Issuer/ClusterIssuer that's actually installed
+# and Ready in the target cluster. See docs/kubernetes/managing-certificates.mdx.
+
+server:
+  disableTls: false
+  grpcEndpoint: "https://openshell.example.com:443"
+  tls:
+    # Must name a secret containing the actual client CA when
+    # clientCaFromServerTlsSecret is false — here, the same CA secret
+    # cert-manager creates for signing the client (mTLS) certificate.
+    clientCaSecretName: openshell-ca-tls
+
+certManager:
+  enabled: true
+  clientCaFromServerTlsSecret: false
+  serverIssuerRef:
+    name: letsencrypt-prod
+    kind: ClusterIssuer
+  serverDnsNames:
+    - openshell.example.com
+
+openshiftRoute:
+  enabled: true
+  host: openshell.example.com

deploy/helm/openshell/templates/cert-manager-pki.yaml

@@ -42,6 +42,7 @@ spec:
   ca:
     secretName: {{ .Values.certManager.caSecretName | quote }}
 ---
+{{- $externalServerIssuer := .Values.certManager.serverIssuerRef.name }}
 apiVersion: cert-manager.io/v1
 kind: Certificate
 metadata:
@@ -53,7 +54,30 @@ spec:
   secretName: {{ .Values.server.tls.certSecretName | quote }}
   duration: {{ .Values.certManager.certificateDuration | quote }}
   renewBefore: {{ .Values.certManager.certificateRenewBefore | quote }}
-  commonName: openshell-server
+  {{- if $externalServerIssuer }}
+  # External issuers (e.g. ACME) reject internal-only names (cluster-local,
+  # localhost, loopback IPs) per CA/Browser Forum baseline requirements, and
+  # require any commonName to also appear in dnsNames. Only request the
+  # externally-resolvable SANs the operator configured — no defaults, no IPs.
+  {{- if .Values.certManager.serverDnsNames }}
+  commonName: {{ first .Values.certManager.serverDnsNames | quote }}
+  {{- end }}
+  dnsNames:
+    {{- range .Values.certManager.serverDnsNames }}
+    - {{ . | quote }}
+    {{- end }}
+  {{- else }}
+  # The chart fullname is always the first entry of defaultServerDnsNames
+  # below, so it's always valid as a commonName.
+  #
+  # Upgrade note: this replaces the previously-hardcoded "openshell-server"
+  # commonName. The ACME commonName-must-be-a-SAN constraint this exists for
+  # only applies to the external-issuer branch above, but this branch was
+  # changed too for consistency. Existing self-signed/internal-CA deployments
+  # will get a new server cert with a different Subject CN on next reissue
+  # after upgrading — SAN-based hostname verification (what TLS clients
+  # actually check) is unaffected.
+  commonName: {{ include "openshell.fullname" . }}
   dnsNames:
     {{- range (include "openshell.defaultServerDnsNames" . | fromYamlArray) }}
     - {{ . | quote }}
@@ -65,6 +89,7 @@ spec:
   ipAddresses:
     {{- toYaml .Values.certManager.serverIpAddresses | nindent 4 }}
   {{- end }}
+  {{- end }}
   privateKey:
     algorithm: ECDSA
     size: 256
@@ -73,9 +98,15 @@ spec:
     - digital signature
     - key encipherment
   issuerRef:
+    {{- if .Values.certManager.serverIssuerRef.name }}
+    name: {{ .Values.certManager.serverIssuerRef.name }}
+    kind: {{ .Values.certManager.serverIssuerRef.kind | default "ClusterIssuer" }}
+    group: {{ .Values.certManager.serverIssuerRef.group | default "cert-manager.io" }}
+    {{- else }}
     name: {{ include "openshell.fullname" . }}-ca-issuer
     kind: Issuer
     group: cert-manager.io
+    {{- end }}
 ---
 apiVersion: cert-manager.io/v1
 kind: Certificate
@@ -97,7 +128,13 @@ spec:
     - digital signature
     - key encipherment
   issuerRef:
+    {{- if .Values.certManager.clientIssuerRef.name }}
+    name: {{ .Values.certManager.clientIssuerRef.name }}
+    kind: {{ .Values.certManager.clientIssuerRef.kind | default "ClusterIssuer" }}
+    group: {{ .Values.certManager.clientIssuerRef.group | default "cert-manager.io" }}
+    {{- else }}
     name: {{ include "openshell.fullname" . }}-ca-issuer
     kind: Issuer
     group: cert-manager.io
+    {{- end }}
 {{- end }}

deploy/helm/openshell/templates/route.yaml

@@ -0,0 +1,28 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+{{- if .Values.openshiftRoute.enabled }}
+apiVersion: route.openshift.io/v1
+kind: Route
+metadata:
+  name: {{ include "openshell.fullname" . }}
+  namespace: {{ .Release.Namespace }}
+  labels:
+    {{- include "openshell.labels" . | nindent 4 }}
+  {{- with .Values.openshiftRoute.annotations }}
+  annotations:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
+spec:
+  {{- if .Values.openshiftRoute.host }}
+  host: {{ .Values.openshiftRoute.host }}
+  {{- end }}
+  to:
+    kind: Service
+    name: {{ include "openshell.fullname" . }}
+  port:
+    targetPort: grpc
+  tls:
+    termination: passthrough
+  wildcardPolicy: None
+{{- end }}

deploy/helm/openshell/tests/cert_manager_pki_test.yaml

@@ -0,0 +1,140 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+suite: cert-manager PKI issuerRef overrides
+templates:
+  - templates/cert-manager-pki.yaml
+release:
+  name: openshell
+  namespace: my-namespace
+
+tests:
+  - it: defaults both server and client Certificates to the chart's own CA issuer
+    template: templates/cert-manager-pki.yaml
+    set:
+      certManager.enabled: true
+    asserts:
+      - equal:
+          path: spec.issuerRef.name
+          value: openshell-ca-issuer
+        documentIndex: 3
+      - equal:
+          path: spec.issuerRef.kind
+          value: Issuer
+        documentIndex: 3
+      - equal:
+          path: spec.issuerRef.name
+          value: openshell-ca-issuer
+        documentIndex: 4
+      - equal:
+          path: spec.issuerRef.kind
+          value: Issuer
+        documentIndex: 4
+
+  - it: default server Certificate includes internal SANs, IPs, and a fixed commonName
+    template: templates/cert-manager-pki.yaml
+    set:
+      certManager.enabled: true
+    asserts:
+      - equal:
+          path: spec.commonName
+          value: openshell
+        documentIndex: 3
+      - contains:
+          path: spec.dnsNames
+          content: openshell.my-namespace.svc.cluster.local
+        documentIndex: 3
+      - contains:
+          path: spec.dnsNames
+          content: localhost
+        documentIndex: 3
+      - equal:
+          path: spec.ipAddresses[0]
+          value: 127.0.0.1
+        documentIndex: 3
+
+  - it: overrides the server Certificate issuerRef when serverIssuerRef is set
+    template: templates/cert-manager-pki.yaml
+    set:
+      certManager.enabled: true
+      certManager.serverIssuerRef.name: letsencrypt-prod
+      certManager.serverIssuerRef.kind: ClusterIssuer
+      certManager.serverDnsNames:
+        - openshell.example.com
+    asserts:
+      - equal:
+          path: spec.issuerRef.name
+          value: letsencrypt-prod
+        documentIndex: 3
+      - equal:
+          path: spec.issuerRef.kind
+          value: ClusterIssuer
+        documentIndex: 3
+      - equal:
+          path: spec.issuerRef.group
+          value: cert-manager.io
+        documentIndex: 3
+
+  - it: scopes server Certificate SANs to only the configured external hostnames when serverIssuerRef is set
+    template: templates/cert-manager-pki.yaml
+    set:
+      certManager.enabled: true
+      certManager.serverIssuerRef.name: letsencrypt-prod
+      certManager.serverIssuerRef.kind: ClusterIssuer
+      certManager.serverDnsNames:
+        - openshell.example.com
+    asserts:
+      - equal:
+          path: spec.commonName
+          value: openshell.example.com
+        documentIndex: 3
+      - equal:
+          path: spec.dnsNames
+          value:
+            - openshell.example.com
+        documentIndex: 3
+      - notExists:
+          path: spec.ipAddresses
+        documentIndex: 3
+      - notContains:
+          path: spec.dnsNames
+          content: localhost
+        documentIndex: 3
+      - notContains:
+          path: spec.dnsNames
+          content: openshell.my-namespace.svc.cluster.local
+        documentIndex: 3
+
+  - it: overrides the client Certificate issuerRef when clientIssuerRef is set
+    template: templates/cert-manager-pki.yaml
+    set:
+      certManager.enabled: true
+      certManager.clientIssuerRef.name: internal-pki-issuer
+      certManager.clientIssuerRef.kind: ClusterIssuer
+    asserts:
+      - equal:
+          path: spec.issuerRef.name
+          value: internal-pki-issuer
+        documentIndex: 4
+      - equal:
+          path: spec.issuerRef.kind
+          value: ClusterIssuer
+        documentIndex: 4
+
+  - it: client Certificate issuerRef is unaffected by serverIssuerRef
+    template: templates/cert-manager-pki.yaml
+    set:
+      certManager.enabled: true
+      certManager.serverIssuerRef.name: letsencrypt-prod
+      certManager.serverIssuerRef.kind: ClusterIssuer
+      certManager.serverDnsNames:
+        - openshell.example.com
+    asserts:
+      - equal:
+          path: spec.issuerRef.name
+          value: openshell-ca-issuer
+        documentIndex: 4
+      - equal:
+          path: spec.issuerRef.kind
+          value: Issuer
+        documentIndex: 4