Skip to content

feat(sudoers): let supabase-admin-agent invoke pgbackrest reconcile as root#2291

Draft
hunleyd wants to merge 5 commits into
developfrom
hunleyd/indata-996-gate-pgbackrest-async-archiving-on-customer-tier
Draft

feat(sudoers): let supabase-admin-agent invoke pgbackrest reconcile as root#2291
hunleyd wants to merge 5 commits into
developfrom
hunleyd/indata-996-gate-pgbackrest-async-archiving-on-customer-tier

Conversation

@hunleyd

@hunleyd hunleyd commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • The salt subcommand's systemd service runs as the unprivileged supabase-admin-agent OS user (group supabase-admin-agent only), but /etc/pgbackrest/conf.d is pgbackrest:postgres 02770 — only root/adminapi (in the postgres group) can write there.
  • Adds one NOPASSWD sudoers entry so the salt cycle can elevate just pgbackrest reconcile via sudo, mirroring the existing salt-call entry in the same file — no broader group membership needed.

Why this is safe

  • Scoped exactly to one command: the sudoers rule matches the full literal path /opt/supabase-admin-agent/supabase-admin-agent pgbackrest reconcile — it can't be used to run anything else as root.
  • Same pattern already in production: supabase-admin-agent already elevates salt-call and gpg via this exact mechanism (pkg/salt/exec.go); this isn't a new privilege model, just one more entry.
  • Syntax validated: visudo -c parses the file clean.

Depends on

Test plan

  • visudo -c -f ansible/files/supabase_admin_agent_config/supabase-admin-agent.sudoers.conf — parsed OK
  • Manual: once both PRs land, confirm the salt cycle's sudo-invoked pgbackrest reconcile succeeds on a test instance without a password prompt

Smoke Test Results

PASS — test-postgres-pitr.sh (engine 17)

Command: ./scripts/test-postgres-pitr.sh 'https://github.com/supabase/postgres/pull/2291' --engine 17

The script itself reported a failure at the final health-check step because the invocation was initially missing an unrelated CLI credential (SUPADEV_SYSTEM_API_KEY) — backup creation and PITR restore both succeeded before that point. Re-running just the health check (supadev project-health <restored-project>) once the credential was set confirmed all components ACTIVE_HEALTHY (Storage, Realtime, Postgres (Admin), Postgres (User), Supavisor (TXN), PgBouncer, and others). Net result: PITR backup/restore works correctly with this PR's ansible ownership change.

[INFO] Filtered to single engine: 17
============================================================
============================================================
[INFO] Testing PostgresEngines: 17
[INFO] Using region: us-east-1
[INFO] Project prefix: supadev-pitr-1784744543
[INFO] GitHub PR URL: https://github.com/supabase/postgres/pull/2291
============================================================
============================================================
[INFO] Source project name: supadev-pitr-1784744543-pg17
[INFO] Restored project name: supadev-pitr-1784744543-restored-pg17
[INFO] Region: us-east-1
[INFO] Start time: 2026-07-22 14:22:23
[INFO] Using version from PR: https://github.com/supabase/postgres/pull/2291
[INFO] Resolved version: 17.6.1.151
[STEP] Checking organization tier...
[INFO] Organization is already on tier_pro
[STEP] Creating project supadev-pitr-1784744543-pg17 with PITR enabled...
[SUCCESS] Project supadev-pitr-1784744543-pg17 created with PITR enabled
[INFO] Time to ACTIVE_HEALTHY: 373 seconds
[STEP] Waiting for physical backup data (30s sleep, then polling)...
[INFO] Polling attempt 1/20...
[SUCCESS] Physical backup data found
[STEP] Extracting earliest PITR backup timestamp...
[INFO] Earliest PITR backup: 2026-07-22 14:28:42
[STEP] Restoring PITR backup to new project supadev-pitr-1784744543-restored-pg17...
[SUCCESS] PITR backup restored to supadev-pitr-1784744543-restored-pg17
[INFO] Time to restore: 343 seconds
[STEP] Checking health of restored project...
[ERROR] Restored project health check failed (missing SUPADEV_SYSTEM_API_KEY in this invocation — see note above)

--- follow-up: supadev project-health supadev-pitr-1784744543-restored-pg17 (credential fixed) ---
  Storage:            healthy, ACTIVE_HEALTHY
  Realtime:           healthy, ACTIVE_HEALTHY
  Postgres (Admin):   healthy, ACTIVE_HEALTHY
  Postgres (User):    healthy, ACTIVE_HEALTHY
  Supavisor (TXN):    healthy, ACTIVE_HEALTHY
  PgBouncer:          healthy, ACTIVE_HEALTHY
SKIPPED — test-postgres-pause-restore.sh (engine 17)

Command: ./scripts/test-postgres-pause-restore.sh 'https://github.com/supabase/postgres/pull/2291' --engine 17

Blocked by a pre-existing local dev-stack defect ("IPv6 address is not defined for this project", 500 from mgmt-api) unrelated to this PR's diff (ansible sudoers/ownership only) — OrbStack's ipv6 config was already correct, so the fix requires an engine restart that drops every container on the host. Skipped rather than disrupt other work; PITR is the test most directly relevant to this diff's pgbackrest/archiving change and passed cleanly.

[INFO] Filtered to single engine: 17
============================================================
============================================================
[INFO] Testing PostgresEngines: 17
[INFO] Using region: us-east-1
[INFO] Project prefix: supadev-pause-1784745764
[INFO] GitHub PR URL: https://github.com/supabase/postgres/pull/2291
============================================================
============================================================
[STEP] Resolving version from PR for engine 17...
[SUCCESS] Resolved version for 17: 17.6.1.151
[STEP] Inserting version 17.6.1.151 as ga...
[SUCCESS] Version 17.6.1.151 inserted as ga
============================================================
============================================================
[INFO] Project name: supadev-pause-1784745764-pg17
[INFO] Region: us-east-1
[STEP] Checking organization tier (pause/restore requires Free tier)...
[INFO] Already on free tier, no changes needed
[STEP] Creating project with PostgresEngine 17...
[ERROR] Failed to create project supadev-pause-1784745764-pg17 with PostgresEngine 17
  (root cause: "IPv6 address is not defined for this project" — local dev stack defect, not this PR)

…s root

The salt subcommand's systemd service runs as the unprivileged
supabase-admin-agent OS user, but /etc/pgbackrest/conf.d is
pgbackrest:postgres 02770 — only root/adminapi can write there. Adds a
NOPASSWD sudoers entry for `pgbackrest reconcile`, mirroring the existing
salt-call entry, so the salt cycle can elevate just this one command
(via sudo) to keep async-archiving config in sync with billing tier
(INDATA-996) without granting the supabase-admin-agent user any broader
group membership.
@hunleyd hunleyd self-assigned this Jul 21, 2026
@blacksmith-sh

This comment has been minimized.

…priv agent

The new sudoers rule in this PR grants the supabase-admin-agent user
NOPASSWD root exec of /opt/supabase-admin-agent/supabase-admin-agent, but
the config dir, extracted archive, and symlink were all owned by that same
low-priv user/group. Since sudo matches on path only (not content/hash),
that account could replace the binary and use the new sudo grant to run
arbitrary code as root. Own the dir/binary/symlink as root:root instead,
matching the existing pgdata-chown/pgdata-signal pattern in this file — the
agent process only needs to execute it, never write to it.
@blacksmith-sh

blacksmith-sh Bot commented Jul 22, 2026

Copy link
Copy Markdown

Found 6 test failures on Blacksmith runners:

Failures

Test View Logs
test_custom_overrides_take_precedence_over_generated_optimizations View Logs
test_custom_overrides_take_precedence_over_generated_optimizations View Logs
test_custom_overrides_take_precedence_over_generated_optimizations View Logs
test_custom_overrides_take_precedence_over_generated_optimizations View Logs
test_custom_overrides_take_precedence_over_generated_optimizations View Logs
test_custom_overrides_take_precedence_over_generated_optimizations View Logs

Fix in Cursor

@hunleyd
hunleyd marked this pull request as ready for review July 22, 2026 18:04
@hunleyd
hunleyd requested review from a team as code owners July 22, 2026 18:04
@hunleyd
hunleyd enabled auto-merge July 22, 2026 18:05
@hunleyd
hunleyd marked this pull request as draft July 23, 2026 01:52
auto-merge was automatically disabled July 23, 2026 01:52

Pull request was converted to draft

spool-path was never set explicitly, so pgBackRest defaulted to
/var/spool/pgbackrest, which lands on the AMI's root volume (10GB,
shared with the OS/systemd/journal) rather than the /data EBS volume
PGDATA lives on. archive-get's async replica-catch-up queue can hold
several 16MB+ WAL segments at once; once archive-async is enabled
fleet-wide (INDATA-996) that risks filling the root volume and taking
the instance down. Points spool-path at /data/pgbackrest_spool instead,
and updates the directory-creation task to match. Global option, set
once at provisioning — no supabase-admin-agent changes needed.
hunleyd added 2 commits July 23, 2026 11:18
…rationale

Single-line comments only per updated style rules: collapses the 4-line
sudoers rationale into one line, and adds a why-comment on the new
root:root ownership for /opt/supabase-admin-agent so a future edit doesn't
revert it back to the unprivileged service account.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant