bug(gator): Codex auth refresh failures cause missing-sentinel retries

[elezar]

Agent Diagnostic

• Skills used: create-github-issue; gator debugging was performed from the local launcher/sandbox logs rather than GitHub checks.
• OpenShell version tested: current local CLI reports openshell 0.0.88. The observed gator launch used the local openshell gateway, which earlier reported 0.0.75 during preflight before main was updated.
• Current branch state checked: local main is clean and tracking origin/main.
• Known fixes reviewed: current main includes the gator payload readability fix in the Docker build path, but the Codex harness still writes fallback/placeholder auth material when provider credentials do not include complete Codex auth fields.
• Possible duplicate issues checked with gh api: no open issues matched gator Codex refresh, CODEX_AUTH_ACCESS_TOKEN refresh, or OPENSHELL_AGENT_RESULT token_expired.
• Findings:
  • A supervised gator run against PR refactor(supervisor): pass agent proposal state explicitly #2421 had early Codex cycles fail with websocket 401 Unauthorized followed by Failed to refresh token: 401 Unauthorized / token_expired.
  • The harness exited without emitting OPENSHELL_AGENT_RESULT, so the wrapper treated the cycle as a transient upstream transport failure and retried.
  • Later cycles recovered and completed successfully (merge_ready, then pr_merged), so the immediate PR watcher did not remain stuck.
  • scripts/agents/runtime/harnesses/codex/exec.sh creates /sandbox/home/.codex/auth.json, but when CODEX_AUTH_ID_TOKEN or CODEX_AUTH_REFRESH_TOKEN are absent it currently falls back to synthetic/placeholder values.
  • scripts/agents/gator/agent.yaml and scripts/agents/gator/providers/codex-gator.yaml provide access token/account ID fields but do not provide CODEX_AUTH_ID_TOKEN; refresh material is gateway-managed rather than placed in the sandbox.
• Reason for filing: the run eventually succeeded, but the current behavior makes long-lived supervised gator watchers vulnerable to intermittent or expired Codex auth state. A few bad refresh attempts can consume retries, delay review cycles, or leave the failure looking like a generic missing-sentinel harness crash.

Description

Supervised gator runs that use the Codex harness can fail early cycles when the Codex access token expires or cannot be refreshed from the auth material written into the sandbox. In the observed PR #2421 run, Codex failed to connect to the websocket with 401 Unauthorized, then failed refresh with token_expired, and exited before writing the required OPENSHELL_AGENT_RESULT sentinel.

The supervisor wrapper did detect this as a transient upstream transport failure and retry. That was enough for this specific run to recover, post merge_ready, and later complete after the PR merged. However, the behavior is still fragile: the sandbox receives enough auth material to start Codex, but not necessarily enough valid material for Codex's own refresh path, while the gateway owns provider refresh outside the already-running process environment.

Expected behavior:

• Gator should either receive complete, usable Codex auth material for the harness lifecycle, or refresh Codex auth through a gateway-mediated mechanism before each Codex cycle.
• Missing unsupported auth fields should fail fast with an actionable preflight error instead of writing synthetic values into auth.json.
• If a transient upstream auth/transport failure does happen, the wrapper should continue to emit a clear structured/sentinel result so the failure mode is not indistinguishable from a generic harness crash.

Reproduction Steps

1. Configure a local openshell gateway with the codex-gator provider and run gator through the Codex harness.

2. Launch a supervised PR watcher, for example:

   ./scripts/agents/run.sh --agent gator --gateway openshell --name gator-pr-2421-supervised --watch --background "Review and monitor PR #2421 through the gator-gate workflow. Scope this invocation only to PR #2421."

3. Inspect the sandbox/launcher log at scripts/agents/gator/logs/<name>.log.

4. When Codex auth is stale, the cycle fails with websocket 401 Unauthorized, then refresh token_expired, then exits without OPENSHELL_AGENT_RESULT.

5. The wrapper retries the cycle as a transient watch failure. In the PR refactor(supervisor): pass agent proposal state explicitly #2421 run, later cycles recovered and completed.

Environment

• OS: Linux local development workstation.
• OpenShell CLI: openshell 0.0.88 on current local main.
• Gateway: local openshell gateway at https://127.0.0.1:17670 using mTLS.
• Agent: scripts/agents/gator.
• Harness: Codex, model gpt-5.5, reasoning high.
• Latest release checked: not checked against public releases in this session; investigation was against updated origin/main.
• Possible duplicate issues checked: yes, no open matching issues found.

Logs

Sanitized excerpts from scripts/agents/gator/logs/gator-pr-2421-supervised.log:

ERROR codex_api::endpoint::responses_websocket: failed to connect to websocket: HTTP error: 401 Unauthorized, url: wss://chatgpt.com/backend-api/codex/responses
ERROR codex_login::auth::manager: Failed to refresh token: 401 Unauthorized: {
  "detail": {
    "message": "Your authentication token has expired. Please sign in again to continue.",
    "code": "token_expired"
  }
}
ERROR: Your access token could not be refreshed. Please log out and sign in again.
openshell-agent: transient watch failure 1 (missing OPENSHELL_AGENT_RESULT after harness exit 1; upstream transport failure detected); retrying in 30s

The same supervised run later recovered:

OPENSHELL_AGENT_RESULT {"status":"waiting","next_poll_seconds":900,"reason":"merge_ready"}
OPENSHELL_AGENT_RESULT {"status":"complete","reason":"pr_merged"}

Suggested Direction

• Extend the codex-gator provider schema and gator agent manifest to include the real CODEX_AUTH_ID_TOKEN if Codex requires it for stable operation.
• Stop fabricating fallback ID tokens in scripts/agents/runtime/harnesses/codex/exec.sh; fail preflight with a clear message when required Codex auth fields are unavailable.
• Keep the real refresh token gateway-owned. Do not copy it into the sandbox unless there is a deliberate security decision to do so.
• Add a gateway-brokered refresh hook for supervised agents, such as a small openshell agent provider-env refresh helper, so long-lived watcher loops can request fresh provider env before each Codex invocation without exposing refresh tokens inside the sandbox.
• Add a regression test or smoke path that verifies stale/expired Codex auth produces an actionable structured failure rather than a missing-sentinel harness exit.

Checklist

• I have searched for existing related issues.
• I included an agent diagnostic section.
• I included the relevant sanitized logs.
• I described the expected behavior and suggested implementation direction.

[elezar]

Additional observation from PR #2153 after rebasing:

• Sandbox gator-pr-2153-supervised is still present on the local openshell gateway and reports Ready.
• The local watcher log was still updating at 2026-07-24 17:37:40 CEST.
• Current PR head observed by gator: fc29302dc35a32a536b932de3350aa037a573570.
• One cycle handled a reviewer sub-agent Codex auth failure correctly and emitted a structured transient result.
• Subsequent cycles failed earlier in the main Codex harness with repeated websocket 401 Unauthorized and refresh token_expired, then exited without OPENSHELL_AGENT_RESULT.
• The sandbox supervisor continued retrying as transient watch failure 5, 6, and 7, eventually backing off to 900s.

Sanitized excerpt:

Cycle summary: PR #2153 is open and no longer dirty; current head is `fc29302dc35a32a536b932de3350aa037a573570`. Required checks are still pending, but the required `principal-engineer-reviewer` sub-agent failed with Codex auth/token refresh errors before producing usable output. Per gator-gate rules, I did not post a gator comment, change labels, or consume the current-head disposition slot.

OPENSHELL_AGENT_RESULT {"status":"transient_failure","next_poll_seconds":120,"reason":"reviewer_subagent_failed"}
openshell-agent: transient watch failure 4 (reviewer_subagent_failed); retrying in 120s

2026-07-24T15:21:31.348215Z ERROR codex_api::endpoint::responses_websocket: failed to connect to websocket: HTTP error: 401 Unauthorized, url: wss://chatgpt.com/backend-api/codex/responses
2026-07-24T15:21:31.860445Z ERROR codex_login::auth::manager: Failed to refresh token: 401 Unauthorized: {
  "error": {
    "message": "Could not validate your token. Please try signing in again.",
    "type": "invalid_request_error",
    "param": null,
    "code": "token_expired"
  }
}
ERROR: Your access token could not be refreshed. Please log out and sign in again.
openshell-agent: transient watch failure 5 (missing OPENSHELL_AGENT_RESULT after harness exit 1; upstream transport failure detected); still retrying in 240s

2026-07-24T15:25:34.960683Z ERROR codex_api::endpoint::responses_websocket: failed to connect to websocket: HTTP error: 401 Unauthorized, url: wss://chatgpt.com/backend-api/codex/responses
openshell-agent: transient watch failure 6 (missing OPENSHELL_AGENT_RESULT after harness exit 1; upstream transport failure detected); retrying in 480s

2026-07-24T15:33:39.218210Z ERROR codex_api::endpoint::responses_websocket: failed to connect to websocket: HTTP error: 401 Unauthorized, url: wss://chatgpt.com/backend-api/codex/responses
2026-07-24T15:33:39.633908Z ERROR codex_login::auth::manager: Failed to refresh token: 401 Unauthorized: {
  "error": {
    "message": "Could not validate your token. Please try signing in again.",
    "type": "invalid_request_error",
    "param": null,
    "code": "token_expired"
  }
}
openshell-agent: transient watch failure 7 (missing OPENSHELL_AGENT_RESULT after harness exit 1; upstream transport failure detected); retrying in 900s

Operator-local evidence, not expected to be accessible from GitHub:

• scripts/agents/gator/logs/gator-pr-2153-supervised.log:132586
• scripts/agents/gator/logs/gator-pr-2153-supervised.log:132602
• scripts/agents/gator/logs/gator-pr-2153-supervised.log:132604
• scripts/agents/gator/logs/gator-pr-2153-supervised.log:132662
• scripts/agents/gator/logs/gator-pr-2153-supervised.log:132730
• scripts/agents/gator/logs/gator-pr-2153-supervised.log:132802
• scripts/agents/gator/logs/gator-pr-2153-supervised.log:132816

This suggests the fix needs to cover both the top-level Codex harness invocation and the principal-engineer-reviewer sub-agent invocation. The sandbox supervisor itself remains alive; the failure is stale/unrefreshable Codex auth material inside repeated watcher cycles.