Skip to content

FIX Batch v1.0 conversation migration backfills#2266

Merged
romanlutz merged 5 commits into
microsoft:mainfrom
romanlutz:romanlutz-batch-conversation-migration
Jul 23, 2026
Merged

FIX Batch v1.0 conversation migration backfills#2266
romanlutz merged 5 commits into
microsoft:mainfrom
romanlutz:romanlutz-batch-conversation-migration

Conversation

@romanlutz

Copy link
Copy Markdown
Contributor

Description

The v1.0 upgrade path performs one database round trip per conversation while backfilling conversation and target-link data. At the production data shape, this leaves tens of thousands of SQL Server operations in both b2f4c6a8d1e3 and e5f7a9c1b3d2; the original migration has already rolled back after communication-link failures, and disposable SQL Server profiling showed the later target-link backfill becoming the dominant bottleneck after fixing the first one.

This change:

  • inserts Conversations in bounded 400-row, 800-parameter statements;
  • groups identical retained target identifiers so each graph is materialized once;
  • stages conversation target links in the same bounded batches, then performs one SQL Server join update or one SQLite correlated update;
  • preserves first-non-null target selection, conflict warnings, attack-only placeholders, existing-row idempotency, nullable pyrit_version, best-effort identifier handling, and the outer Alembic transaction;
  • adds focused failure, rollback, fallback, cleanup, batching-limit, and idempotency coverage plus an opt-in production-scale SQLite upgrade test.

This intentionally modifies two committed migration revisions. Production is still before both revisions, so a new later revision cannot reduce the round trips that the upgrade must execute first. The Alembic immutability guard therefore requires an explicit release-maintainer exception for this hotfix.

Disposable SQL Server 2022 validation reduced the complete 9c8b7a6d5e4f to v1 head upgrade to 33.141 seconds. e5f7a9c1b3d2 fell from 398.68 seconds to 9.679 seconds. No production database was accessed.

Tests and Documentation

  • 8 focused migration batching and fallback tests passed.
  • Complete tests/unit/memory/test_migration.py: 47 passed.
  • Production-scale SQLite upgrade: 135,069 prompt rows, 81,432 conversations, 204 conversation inserts, 204 target-link staging inserts, 1,227 total statements, 6.785-second migration runtime.
  • Disposable SQL Server 2022 full upgrade and forced rollback validation passed, including staging-table cleanup.
  • Changed-file pre-commit checks and git diff --check passed.
  • Documentation: N/A; migration behavior is unchanged apart from bounded execution.

Copilot AI added 3 commits July 22, 2026 07:35
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 3987fdd3-8422-4312-9d60-647cba9c6802
Reduce SQL Server round trips when backfilling Conversations and target identifier links.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: fdbe13ad-99bf-443d-a98e-95ff5d7d023e
Revert v1.0 release metadata so this PR contains only the migration fix.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: fdbe13ad-99bf-443d-a98e-95ff5d7d023e
Copilot AI added 2 commits July 23, 2026 11:03
Report bounded progress while inserting conversations, materializing target graphs, staging target links, and applying the final update.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: fdbe13ad-99bf-443d-a98e-95ff5d7d023e
Batch scorer, scenario, converter, and attack identifier phases while preserving migration fallback behavior and bounded progress output.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: fdbe13ad-99bf-443d-a98e-95ff5d7d023e
@romanlutz
romanlutz merged commit 481ce17 into microsoft:main Jul 23, 2026
50 of 53 checks passed
@romanlutz
romanlutz deleted the romanlutz-batch-conversation-migration branch July 23, 2026 22:42
romanlutz added a commit that referenced this pull request Jul 23, 2026
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3987fdd3-8422-4312-9d60-647cba9c6802
Copilot-Session: fdbe13ad-99bf-443d-a98e-95ff5d7d023e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants