Skip to content

fix(server): visibility-map health canary for metric_series_rollups (JEF-594) - #156

Merged
thejefflarson merged 1 commit into
mainfrom
thejefflarson/jef-594-index-only-scan-visibility-map-health-canary-for
Jul 29, 2026
Merged

fix(server): visibility-map health canary for metric_series_rollups (JEF-594)#156
thejefflarson merged 1 commit into
mainfrom
thejefflarson/jef-594-index-only-scan-visibility-map-health-canary-for

Conversation

@thejefflarson

Copy link
Copy Markdown
Owner

Summary

The JEF-591 covering index only produces index-only scans on
metric_series_rollups while its visibility map stays current. If
autovacuum falls behind on this high-churn table (post-JEF-591 deploy
we already observed a transient Heap Fetches: 2268 while the VM
caught up), scans silently degrade to heap fetches and the JEF-548-class
latency quietly returns -- with no signal until dashboards get slow.
Autovacuum is tuned (migration 0013, 2% scale factor) but unverified
under sustained Pi ingest.

This adds a self-monitoring canary (fits ADR 0014) on the existing
self-telemetry interval, tagged table=metric_series_rollups:

  • watcher.db.dead_tuple_ratio -- n_dead_tup / (n_live_tup + n_dead_tup)
    from pg_stat_user_tables (always available, no extension needed).
  • watcher.db.last_autovacuum_age_seconds -- age of the last real
    autovacuum run; skipped (not zeroed) before the first one ever
    completes.
  • watcher.db.vm_all_visible_fraction -- all-visible pages / relpages
    via pg_visibility_map_summary (reads only the visibility-map fork,
    not pg_visibility(), which would also touch every heap page).

pg_visibility availability: it's a contrib extension and not
"trusted", so watcher's non-superuser app role can't (and doesn't try
to) CREATE EXTENSION it itself -- installing it is a cluster-ops
action, not read-only introspection. When it's absent, the query fails
with SQLSTATE 42883 (undefined_function); the code treats that as
"unavailable" (logged once, not every tick) rather than failing the
whole self-telemetry snapshot. Dead-tuple ratio and autovacuum age
still cover the canary on their own either way.

DECISION NEEDED (cross-repo): alert rules are declarative (ADR
0012) and the rule content lives in the ../cluster chart's
server.alerts values, not this repo -- this PR only makes the
metrics alertable, it doesn't add a rule. Suggested starting
thresholds, given the 2% autovacuum scale factor tuned in migration
0013: dead_tuple_ratio > 0.10 (5x the tuned scale factor) or
vm_all_visible_fraction < 0.5, agg=avg, window_secs=300. Flagging
for the architect / cluster-repo follow-up.

How I tested it

  • Pure unit tests for the ratio math (dead_tuple_ratio,
    vm_all_visible_fraction, both routed through a shared safe_ratio
    helper) and the SQLSTATE-42883 classifier.
  • New integration test (against a real local Postgres) that seeds a
    metric_series_rollups row via ingest, runs ANALYZE (not VACUUM
    -- manual VACUUM doesn't set last_autovacuum, so that gauge's
    correct absence is also asserted) to make pg_class.relpages
    deterministic, then asserts dead_tuple_ratio and
    vm_all_visible_fraction (when the test role can CREATE EXTENSION IF NOT EXISTS pg_visibility) land in /api/metrics?service=watcher
    with plausible values.
  • cargo fmt --check, cargo check, cargo clippy --all-targets -- -D warnings, cargo test --locked all green locally (67 smoke tests +
    unit tests, including the two new ones).
  • Ran /soundcheck:pr-review and a /simplify pass on the diff: no
    Critical/High findings; simplified the two near-identical ratio
    functions into one shared safe_ratio helper and dropped a
    hand-rolled DatabaseError test mock in favor of matching the
    existing otlp::is_failover_error convention (SQLSTATE classifiers
    here are only unit-tested indirectly, via a real-Postgres-error
    integration test).

Closes JEF-594

…(JEF-594)

Adds a JEF-594 self-monitoring canary for the JEF-591 covering index's
core assumption: index-only scans on metric_series_rollups only stay
cheap while the visibility map is current, and autovacuum falling
behind on this high-churn table would silently regress it back to the
JEF-548-class latency with no signal until dashboards get slow.

On the existing self-telemetry interval (ADR 0014), collect_metrics now
emits, tagged table=metric_series_rollups:

- watcher.db.dead_tuple_ratio -- n_dead_tup / (n_live_tup + n_dead_tup)
  from pg_stat_user_tables, always available (no extension needed).
- watcher.db.last_autovacuum_age_seconds -- age of the last real
  autovacuum run; skipped (not zeroed) before the first one ever
  completes.
- watcher.db.vm_all_visible_fraction -- all-visible pages / relpages via
  pg_visibility_map_summary (reads only the VM fork, not
  pg_visibility(), which would also touch every heap page).

pg_visibility is a contrib extension and not "trusted", so the app's
non-superuser role can't (and doesn't try to) CREATE EXTENSION it
itself -- that's a cluster-ops action, not read-only introspection.
When it's absent, the query fails with SQLSTATE 42883
(undefined_function); that's treated as "unavailable" (logged once,
not per tick) rather than failing the whole self-telemetry snapshot,
so dead-tuple ratio and autovacuum age still cover the canary on their
own.

Tests (against a real Postgres): pure unit tests for the ratio math,
plus an integration test that seeds a rollup row, forces deterministic
pg_stat_user_tables/pg_class state (ANALYZE, not VACUUM -- manual
VACUUM doesn't set last_autovacuum, so that gauge's absence is asserted
too), and exercises the pg_visibility path end to end when the test
role can install it.

DECISION NEEDED (cross-repo): a declarative alert-rule threshold for
these metrics (ADR 0012) lives in the ../cluster chart's
server.alerts values, not this repo -- this PR only makes the metrics
alertable. Suggested starting thresholds, given the 2% autovacuum
scale factor tuned in migration 0013: dead_tuple_ratio > 0.10 (5x the
tuned scale factor) or vm_all_visible_fraction < 0.5, agg=avg,
window_secs=300.

Closes JEF-594

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
@thejefflarson
thejefflarson merged commit c60cbf2 into main Jul 29, 2026
5 checks passed
@thejefflarson
thejefflarson deleted the thejefflarson/jef-594-index-only-scan-visibility-map-health-canary-for branch July 29, 2026 02:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant