fix(server): visibility-map health canary for metric_series_rollups (JEF-594) - #156
Merged
thejefflarson merged 1 commit intoJul 29, 2026
Conversation
…(JEF-594) Adds a JEF-594 self-monitoring canary for the JEF-591 covering index's core assumption: index-only scans on metric_series_rollups only stay cheap while the visibility map is current, and autovacuum falling behind on this high-churn table would silently regress it back to the JEF-548-class latency with no signal until dashboards get slow. On the existing self-telemetry interval (ADR 0014), collect_metrics now emits, tagged table=metric_series_rollups: - watcher.db.dead_tuple_ratio -- n_dead_tup / (n_live_tup + n_dead_tup) from pg_stat_user_tables, always available (no extension needed). - watcher.db.last_autovacuum_age_seconds -- age of the last real autovacuum run; skipped (not zeroed) before the first one ever completes. - watcher.db.vm_all_visible_fraction -- all-visible pages / relpages via pg_visibility_map_summary (reads only the VM fork, not pg_visibility(), which would also touch every heap page). pg_visibility is a contrib extension and not "trusted", so the app's non-superuser role can't (and doesn't try to) CREATE EXTENSION it itself -- that's a cluster-ops action, not read-only introspection. When it's absent, the query fails with SQLSTATE 42883 (undefined_function); that's treated as "unavailable" (logged once, not per tick) rather than failing the whole self-telemetry snapshot, so dead-tuple ratio and autovacuum age still cover the canary on their own. Tests (against a real Postgres): pure unit tests for the ratio math, plus an integration test that seeds a rollup row, forces deterministic pg_stat_user_tables/pg_class state (ANALYZE, not VACUUM -- manual VACUUM doesn't set last_autovacuum, so that gauge's absence is asserted too), and exercises the pg_visibility path end to end when the test role can install it. DECISION NEEDED (cross-repo): a declarative alert-rule threshold for these metrics (ADR 0012) lives in the ../cluster chart's server.alerts values, not this repo -- this PR only makes the metrics alertable. Suggested starting thresholds, given the 2% autovacuum scale factor tuned in migration 0013: dead_tuple_ratio > 0.10 (5x the tuned scale factor) or vm_all_visible_fraction < 0.5, agg=avg, window_secs=300. Closes JEF-594 Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
thejefflarson
deleted the
thejefflarson/jef-594-index-only-scan-visibility-map-health-canary-for
branch
July 29, 2026 02:31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The JEF-591 covering index only produces index-only scans on
metric_series_rollupswhile its visibility map stays current. Ifautovacuum falls behind on this high-churn table (post-JEF-591 deploy
we already observed a transient
Heap Fetches: 2268while the VMcaught up), scans silently degrade to heap fetches and the JEF-548-class
latency quietly returns -- with no signal until dashboards get slow.
Autovacuum is tuned (migration 0013, 2% scale factor) but unverified
under sustained Pi ingest.
This adds a self-monitoring canary (fits ADR 0014) on the existing
self-telemetry interval, tagged
table=metric_series_rollups:watcher.db.dead_tuple_ratio--n_dead_tup / (n_live_tup + n_dead_tup)from
pg_stat_user_tables(always available, no extension needed).watcher.db.last_autovacuum_age_seconds-- age of the last realautovacuum run; skipped (not zeroed) before the first one ever
completes.
watcher.db.vm_all_visible_fraction-- all-visible pages /relpagesvia
pg_visibility_map_summary(reads only the visibility-map fork,not
pg_visibility(), which would also touch every heap page).pg_visibility availability: it's a contrib extension and not
"trusted", so watcher's non-superuser app role can't (and doesn't try
to)
CREATE EXTENSIONit itself -- installing it is a cluster-opsaction, not read-only introspection. When it's absent, the query fails
with SQLSTATE
42883(undefined_function); the code treats that as"unavailable" (logged once, not every tick) rather than failing the
whole self-telemetry snapshot. Dead-tuple ratio and autovacuum age
still cover the canary on their own either way.
DECISION NEEDED (cross-repo): alert rules are declarative (ADR
0012) and the rule content lives in the
../clusterchart'sserver.alertsvalues, not this repo -- this PR only makes themetrics alertable, it doesn't add a rule. Suggested starting
thresholds, given the 2% autovacuum scale factor tuned in migration
0013:
dead_tuple_ratio > 0.10(5x the tuned scale factor) orvm_all_visible_fraction < 0.5,agg=avg,window_secs=300. Flaggingfor the architect / cluster-repo follow-up.
How I tested it
dead_tuple_ratio,vm_all_visible_fraction, both routed through a sharedsafe_ratiohelper) and the SQLSTATE-42883 classifier.
metric_series_rollupsrow via ingest, runsANALYZE(notVACUUM-- manual
VACUUMdoesn't setlast_autovacuum, so that gauge'scorrect absence is also asserted) to make
pg_class.relpagesdeterministic, then asserts
dead_tuple_ratioandvm_all_visible_fraction(when the test role canCREATE EXTENSION IF NOT EXISTS pg_visibility) land in/api/metrics?service=watcherwith plausible values.
cargo fmt --check,cargo check,cargo clippy --all-targets -- -D warnings,cargo test --lockedall green locally (67 smoke tests +unit tests, including the two new ones).
/soundcheck:pr-reviewand a/simplifypass on the diff: noCritical/High findings; simplified the two near-identical ratio
functions into one shared
safe_ratiohelper and dropped ahand-rolled
DatabaseErrortest mock in favor of matching theexisting
otlp::is_failover_errorconvention (SQLSTATE classifiershere are only unit-tested indirectly, via a real-Postgres-error
integration test).
Closes JEF-594