perf(migration): drop O(n^2) inheritance query from endpoint-to-locations migration, make re-runs converge

[Maffooch]

Shortcut: sc-14059

Problem

A migration run of 962 endpoints took 32 minutes, with throughput decaying from 79.9 to 0.5 endpoints/sec:

Migrated  50/962 —  79.9 endpoints/sec
Migrated 100/962 —   1.2 endpoints/sec
...
Done. Migrated 962/962 endpoints in 32m 21s (0.50 endpoints/sec).

The decay is the tell. Every Location created by the migration fires inherit_tags_on_instance, which calls Location.all_related_products(). That query ORs two multi-join paths, and Postgres cannot index an OR spanning separate join branches, so it materialises the whole join and filters afterwards:

Hash Right Join  (actual time=108.253..108.272 rows=0 loops=1)
  Filter: ((lpr.location_id = 963) OR (lfr.location_id = 963))
  Rows Removed by Filter: 231564

Because the migration is itself filling locationfindingreference, that query grows with every endpoint migrated. Measured: Location.objects.create() cost 132 ms, of which 105 ms was this query, returning zero products every time.

Fix

The signal work is structurally incapable of doing anything here: the Location is created inside URL.pre_save_logic(), before any reference row exists, so all_related_products() returns empty and auto_inherit_product_tags early-returns. Inheritance is applied in bulk by _run_tag_inheritance() once the references exist, which is why that pass was added in #15311. The main loop now runs inside the purpose-built suppress_tag_inheritance(), with the bulk pass still running outside it.

Writes are also batched per chunk instead of per endpoint:

• URL.bulk_get_or_create resolves a chunk in ~3 queries, replacing a savepoint pair + 2 SELECTs + 2 INSERTs per endpoint.
• DojoMeta, LocationFindingReference, LocationProductReference and tag rows each go out as one bulk write per chunk, deduplicated on their unique constraint.

Per-endpoint resilience is preserved. A failed batch write retries row by row and attributes the failure to the endpoint that produced it; a failed bulk location resolve falls back to per-endpoint get_or_create.

Idempotency

The command now converges on the state the source Endpoints/Endpoint_Statuses describe:

• LocationFindingReference upserts on (location, finding), so status, auditor and audit_time re-sync when the source Endpoint_Status has changed. created is preserved.
• LocationProductReference status is recomputed from the finding references after each chunk, using the rule Location.status_from_product already applies. This repairs drifted statuses, makes the outcome independent of endpoint visit order, and fixes the first-write-wins bug the previous code documented, where a Mitigated status stuck even when later Active findings existed for the same product.
• DojoMeta rows and tag copies stay insert-only, so re-runs neither duplicate them nor overwrite post-migration edits.

Results

Same dataset, same machine:

endpoints	before	after
962	32s	2.6s
3,962	25+ min (2,965 done in 12m49s, still decelerating)	25s

Per-500-endpoint windows before: 27s, 63s, 99s, 130s, 186s, 271s. After, the cumulative rate rises over the run (95 to 158 endpoints/sec).

Verification

• Output parity: byte-identical to the previous implementation, compared via md5 digests over every location, reference, meta and tag row at both 962 and 3,962 endpoints.
• Idempotency: a second run is a byte-identical no-op at both scales.
• Status sync: verified that a changed source Endpoint_Status propagates, that a genuine Active-to-Mitigated downgrade propagates, and that an externally corrupted product-reference status is repaired.
• Not a regression of perf(migration): batch endpoint tags and restore inheritance #15311: with _run_tag_inheritance() stubbed out, both the previous and the new implementation produce inherited_tags = [], and a spy shows the pre-change signal fired once for a Location and found 0 products. perf(migration): batch endpoint tags and restore inheritance #15311 replaced tags.add() with bulk_add_tag_mapping, which writes through-rows via bulk_create and emits no m2m_changed, so the explicit bulk pass was already the only mechanism. That PR's own regression test passes unchanged.
• Validation gap closed: bulk_get_or_create only calls clean(), whereas the save() path it replaces ran full_clean(). Without restoring it, an endpoint with an empty host would be silently written as an invalid Location.

Tests

7 new tests covering re-run row counts and created preservation, finding-reference status sync, product-reference reconciliation and downgrade, order independence for a location shared across chunks, invalid-endpoint reporting, the bulk-resolve fallback, and that the inheritance signal is suppressed while inherited tags still land. The 3 existing tests pass unchanged.

Ran 10 tests in 5.215s
OK

🤖 Generated with Claude Code

https://claude.ai/code/session_01WqDAY5ktvcpa1JioMwsiPa