Skip to content

fix(ansible): move pgBackRest spool-path off the 10GB root volume#2312

Draft
hunleyd wants to merge 2 commits into
developfrom
hunleyd/indata-1038-pgbackrest-archive-async-spool-path-defaults-onto-the-10gb-root-volume
Draft

fix(ansible): move pgBackRest spool-path off the 10GB root volume#2312
hunleyd wants to merge 2 commits into
developfrom
hunleyd/indata-1038-pgbackrest-archive-async-spool-path-defaults-onto-the-10gb-root-volume

Conversation

@hunleyd

@hunleyd hunleyd commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

  • pgBackRest's spool-path (used by async archive-push/archive-get to queue transient data) was never set explicitly, so it defaulted to /var/spool/pgbackrest — which sits on the AMI's 10GB root volume, not /data (the dedicated EBS data volume PGDATA lives on).
  • Sets spool-path = /data/pgbackrest_spool in the ansible-managed [global] pgbackrest.conf, and updates the directory-creation task to create that path instead.
  • /data is the volume that survives pause/restore, pg_upgrade, resize-compute, and restore-to-new-project — any event that can throw away the existing EC2 instance and its root volume. The root volume does not survive those events, so anything written there is lost/reset along with it.
  • This is a [global]-scope pgBackRest option, set once at provisioning — no supabase-admin-agent changes needed. Split out of feat(sudoers): let supabase-admin-agent invoke pgbackrest reconcile as root #2291 (sudoers grant for pgbackrest reconcile) since the two are unrelated concerns: one is a privilege-escalation grant, this one is a storage-location fix, and neither depends on the other.

Why this is safe

  • Once INDATA-996 enables archive-async fleet-wide, archive-get's replica-catch-up queue can hold several 16MB+ WAL segments at once — on the 10GB root volume (which also holds the OS/systemd/journal) that risks filling it and taking the instance down.
  • Spool-path data is disposable: per pgBackRest docs, spool-path contents aren't durable state — archive-push rechecks each WAL segment against the repo and archive-get rebuilds its queue on loss. No migration needed, and pgBackRest isn't fleet-deployed yet (still "Wire up the deployment" milestone).
  • Matches the existing ownership pattern (pgbackrest:postgres) already applied to every other entry in the same directory-creation loop.

Test plan

  • ansible-playbook --syntax-check on the full playbook — clean
  • Manual: confirm /data/pgbackrest_spool exists with pgbackrest:postgres ownership after provisioning, and archive-get/archive-push operate against it

Resolves INDATA-1038

spool-path was never set explicitly, so pgBackRest defaulted to
/var/spool/pgbackrest, which lands on the AMI's root volume (10GB,
shared with the OS/systemd/journal) rather than the /data EBS volume
PGDATA lives on. archive-get's async replica-catch-up queue can hold
several 16MB+ WAL segments at once; once archive-async is enabled
fleet-wide (INDATA-996) that risks filling the root volume and taking
the instance down. Points spool-path at /data/pgbackrest_spool instead,
and updates the directory-creation task to match. Global option, set
once at provisioning — no supabase-admin-agent changes needed.
@hunleyd hunleyd self-assigned this Jul 23, 2026
…est.yml

The pgbackrest spool directory path is now split across two files
(pgbackrest.conf's spool-path and this task's dir-creation loop); add a
why-comment so a future edit to one doesn't silently drift from the other.
@blacksmith-sh

blacksmith-sh Bot commented Jul 23, 2026

Copy link
Copy Markdown

Found 1 test failure on Blacksmith runners:

Failure

Test View Logs
test_ami_nix/test_postgrest_starting_apikey_query_parameter_is_removed View Logs

Fix in Cursor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant