Skip to content

feat(helm): cert-manager external issuer + OpenShift passthrough Route#2468

Open
jhjaggars wants to merge 10 commits into
NVIDIA:mainfrom
jhjaggars:2466-cert-manager-openshift-tls/jhjaggars
Open

feat(helm): cert-manager external issuer + OpenShift passthrough Route#2468
jhjaggars wants to merge 10 commits into
NVIDIA:mainfrom
jhjaggars:2466-cert-manager-openshift-tls/jhjaggars

Conversation

@jhjaggars

Copy link
Copy Markdown
Contributor

Lets cert-manager issue the gateway's server TLS certificate from a real Issuer/ClusterIssuer (for example an ACME issuer) instead of only the chart's built-in self-signed CA, and adds an OpenShift Route template with TLS passthrough so the gateway can be exposed externally with a publicly-trusted certificate while it keeps terminating its own TLS/mTLS. Also fixes a related sandbox-supervisor TLS trust bug this uncovered.
Closes #2466

  • certManager.serverIssuerRef / certManager.clientIssuerRef (new values): override the issuerRef on the server/client Certificate resources, falling back to the chart's own self-signed CA issuer when unset (fully backward compatible).
  • Server Certificate's SANs/commonName are scoped to only the operator-configured external hostnames when serverIssuerRef is set, since ACME issuers reject internal-only names (*.svc.cluster.local, localhost, loopback IPs) and require commonName to also be a SAN.
  • New templates/route.yaml: an OpenShift Route with tls.termination: passthrough, gated by openshiftRoute.enabled.
  • crates/openshell-core: the sandbox supervisor's gRPC client only trusted the CA configured via OPENSHELL_TLS_CA. Once the server cert comes from a different (public) issuer than the client cert, every supervisor connection failed the TLS handshake with UnknownCA. Fixed by also trusting the public root stores (native + webpki) in addition to the configured CA — this is additive, not a replacement, so existing self-signed deployments are unaffected.
  • Docs: production section in docs/kubernetes/openshift.mdx, new section in docs/kubernetes/managing-certificates.mdx documenting the new fields and why the client (mTLS) certificate should stay on the chart's internal CA even when the server certificate moves to a public issuer.
  • New CI render-coverage values file (ci/values-openshift-route-cert-manager.yaml) and Helm unittest coverage (tests/route_test.yaml, tests/cert_manager_pki_test.yaml).
  • Synced .agents/skills/debug-openshell-cluster/SKILL.md with troubleshooting steps for the new failure modes.
    Validated end-to-end against a live OpenShift (ROSA) cluster: passthrough Route serving a real Let's Encrypt certificate issued via a Route53 DNS-01 ClusterIssuer, verified with a standard TLS client (no custom CA trust required), OIDC-authenticated CLI access via Keycloak, and a sandbox pod's supervisor successfully mTLS-connecting back to the gateway with the fixed trust store.
  • mise run pre-commit passes
  • Unit tests added/updated (Helm unittest coverage for the Route template and issuerRef overrides; existing statefulset_client_ca_test.yaml coverage re-verified with no regressions)
  • E2E tests added/updated (if applicable) — validated manually against a live OpenShift cluster instead; see PR description above
  • Follows Conventional Commits
  • Commits are signed off (DCO)

jhjaggars added 10 commits July 23, 2026 13:27
… passthrough

Adds certManager.serverIssuerRef/clientIssuerRef so the gateway and mTLS
client certificates can be issued by a real Issuer/ClusterIssuer (e.g. an
ACME issuer) instead of only the chart's built-in self-signed CA, plus a
new openshiftRoute template for exposing the gateway via a passthrough
Route so the gateway keeps terminating its own TLS/mTLS.
The server Certificate now excludes internal-only SANs (cluster-local,
localhost, loopback) when an external issuer is configured, since ACME
issuers reject those per CA/Browser Forum baseline requirements, and
requires the commonName to be one of the requested SANs. The gateway's
client-CA-for-verification also no longer depends on the server secret's
issuer, sourcing from certManager.caSecretName when
clientCaFromServerTlsSecret is false so it stays correct regardless of
what issues the server cert.
Validated against a live OpenShift cluster: passthrough Route serving a
real Let's Encrypt certificate issued via Route53 DNS-01, verified with a
standard TLS client (no custom CA trust required).

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
The supervisor's gRPC client only trusted the CA configured via
OPENSHELL_TLS_CA, since tonic's ClientTlsConfig starts with an empty
root store unless with_native_roots()/with_webpki_roots() is also
enabled. Deployments where the gateway's server certificate is issued
by a public CA (e.g. cert-manager against an ACME issuer, as opposed to
the chart's default self-signed CA that also signs this client's own
identity) caused every supervisor connection to fail the TLS handshake
with "UnknownCA", since the sandbox mTLS CA and the server's cert
issuer were no longer the same.
tonic's root store is a union of all configured sources, so trusting
the public roots in addition to the configured CA does not weaken
verification for existing self-signed deployments — it only adds
coverage for the case where the two now differ. webpki-roots (compiled
in) is enabled alongside native-roots since the supervisor binary may
run in minimal sandbox images without a populated system CA bundle.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
…t path

The commonName fix for the external-issuer branch (ACME requires the
commonName to also be a SAN) was applied to the internal self-signed/CA
branch too, even though that constraint doesn't apply there. Existing
certManager.enabled=true deployments — OpenShift or not — will get a
server cert with a different Subject CN (the release fullname instead
of the hardcoded "openshell-server") on next reissue after upgrading to
this change. SAN-based hostname verification, which is what TLS clients
actually check, is unaffected either way.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
helm-docs output was out of sync with the certManager.serverIssuerRef,
certManager.clientIssuerRef, and openshiftRoute.* values added earlier
on this branch.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
… Route

Adds a production path to the OpenShift install guide covering
certManager.serverIssuerRef, clientCaFromServerTlsSecret=false, and
openshiftRoute for exposing the gateway externally with a real
certificate while the gateway keeps terminating its own TLS/mTLS.
Documents the new certManager.serverIssuerRef/clientIssuerRef fields in
Managing Certificates, including why the client (mTLS) certificate
stays on the chart's internal CA even when the server certificate moves
to a public issuer.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
Exercises certManager.serverIssuerRef, clientCaFromServerTlsSecret=false,
server.grpcEndpoint, and openshiftRoute.enabled together for helm lint
coverage. Auto-discovered by the helm:lint task's ci/values-*.yaml glob,
no separate registration needed.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
…nstead

The earlier _gateway-workload.tpl change to auto-source the client CA
from certManager.caSecretName when clientCaFromServerTlsSecret=false
broke two existing, intentional behaviors caught by
tests/statefulset_client_ca_test.yaml:
- Setting server.tls.clientCaSecretName to a custom secret (the
  chart's existing "bring your own client CA" escape hatch) was
  silently overridden.
- Setting server.tls.clientCaSecretName="" to disable client cert
  verification entirely (HTTPS-only) stopped working, since the
  auto-redirect now mounted a client-CA volume unconditionally
  whenever certManager.enabled was true.
The chart already supported what the external-issuer feature needs:
clientCaFromServerTlsSecret=false already falls through to mounting
server.tls.clientCaSecretName verbatim. No template change was needed
at all — just setting that value to name the actual CA secret (e.g.
certManager.caSecretName's value) alongside clientCaFromServerTlsSecret.
_gateway-workload.tpl is now byte-identical to before this branch.
Updated values.yaml, docs, and the CI values file to reflect this.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
…errides

route_test.yaml covers templates/route.yaml: no-op when disabled,
passthrough termination, host/annotations, and the default-omitted
host case.
cert_manager_pki_test.yaml covers templates/cert-manager-pki.yaml:
default issuerRef falls back to the chart's own CA issuer for both
server and client Certificates, serverIssuerRef/clientIssuerRef
override independently, and the server Certificate's SANs/commonName
are scoped to only the configured external hostnames (no internal
cluster-local SANs, no default IP SANs) when serverIssuerRef is set.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
…ute changes

Adds troubleshooting steps for certManager.serverIssuerRef (checking
Certificate/CertificateRequest/Challenge status when ACME issuance
fails) and the supervisor UnknownCA failure mode that results from
misconfigured server.tls.clientCaSecretName once the server and client
certs diverge in issuer, plus two corresponding Common Failure Patterns
table rows.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
…ction

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
@copy-pr-bot

copy-pr-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(helm): cert-manager external issuer + OpenShift passthrough Route for gateway TLS

1 participant