api/services/nginx/app/modules/xlvask/AUTOMATION_RUNBOOK.md

XL Vask AI automation runbook

This runbook is an operator procedure. None of its gates are applied by deployment, HTTP GETs, constructors, or workers. Every production-changing step requires a human approval tied to the exact deployed backend and frontend SHAs.

0. Deploy-order and rollback invariant

The legacy attachment/creation config values are kill switches, but an old backend treats them as direct enable switches. Old code cannot interpret the new policy stages, calibration identity, rolling caps, action latches, or canary soak. Therefore old-backend traffic is forbidden whenever either legacy switch is true, including during a new-policy canary.

Use this exact forward sequence:

1. While the old backend is still serving, set both legacy automatic-order switches to false through the approved config procedure and verify the persisted values from every serving instance.
2. Stop/disable old XL Vask automation workers and verify there is no active automatic run. Ordinary XL Vask synchronization may continue.
3. Deploy the new backend with policy effectively off; verify ordinary synchronization still completes and scheduled automation no-ops while schema readiness is false.
4. Run read-only migration preflight, then the separately approved explicit additive migration. If it is partial or fails, keep the new backend deployed, policy off, both legacy switches false, and workers no-op; repair or complete the migration before continuing. Never route old code as a partial-migration workaround.
5. Verify migration readiness and the new backend SHA, then deploy/verify the compatible frontend. Only after that generate advisory evidence and use preview-bound policy transitions.

Use this exact rollback sequence before any old-code traffic:

1. Keep all traffic on the new backend, call the dedicated halt endpoint, and verify policy halted plus both persisted legacy switches false.
2. Stop new-backend automation workers, wait for or safely reconcile the active run, and verify no financial mutation is in flight.
3. Roll back the frontend if required, then deploy the old backend with both legacy switches still false. Verify ordinary sync only.
4. Do not re-enable either legacy switch on old code. Recovery of automatic actions requires redeploying the new policy-aware backend and repeating readiness, advisory calibration, canary, and soak.

1. Read-only preflight

1. Record the backend/frontend SHAs, environment, operator, invoice period, and scanner-hall scope.
2. Call the scoped capabilities and admin-readiness GETs with dateFrom and dateTo.
3. Confirm migration.ready, missing_tables, missing_columns, missing_indexes, preflight_conflicts, worker_healthy, WashId uniqueness, planner identity, resolved model, active run, scoped eligible counts, rolling budgets, and reviewed soak counts.
4. Stop if dates are invalid, hall scope is empty, an execute run is active, identity changed, a latch is halted, or any readiness field fails closed.

2. Explicit schema migration

Use the controlled database migration procedure to invoke only migration_20260804_xlvask_ai_auto_policy_v2::apply(). First retain its read-only preflight() output. Review the additive SQL and backup/restore point, approve the exact SHA, run it once, retain the returned status, and rerun readiness. Do not invoke applyExplicitMigration() from a request, worker, cron task, or application startup. If preflight reports multiple legacy execute runs in queued, running, or retry_wait, stop. Reconcile those runs through a separately approved operational procedure; the migration never auto-resolves or modifies the conflicting run records.

3. WashId uniqueness

Inspect normalized duplicate WashIds. Resolve conflicts through an independently approved data procedure. Only then use the guarded uniqueness activation with the exact typed phrase. Recheck the generated normalized column and unique index before any automatic action.

4. Advisory evidence and calibration

Keep policy at advisory. Run explicit dry_run requests to import and persist plans, or replay for cache-only read-only evaluation. Review suggestions in hall scope. Label exact OpenAI attach/create suggestions; model identity, prompt hash, schema hash, policy version, resolved model, and chronological label snapshot are part of the artifact identity. Generate inactive backtests, independently review qualification thresholds and contradictions, then activate the exact artifact hash with its typed phrase.

5. Staged policy transitions

Every transition uses a bounded human reason, server-generated policy preview, exact confirmation phrase, and apply-time revalidation. The reason is bound into the preview hash and retained in the immutable policy event:

off -> advisory -> ai_attach_canary -> ai_attach_verified -> ai_create_canary -> verified_capped

Stages may not be skipped. An active execute run, stale preview, changed policy version, changed model/planner identity, missing exact calibration, incomplete reviewed soak, invalid period scope, or exhausted readiness gate blocks promotion.

6. Reviewed soak and caps

Volume alone never completes soak. Every auto-accepted action must be adjudicated. Only explicit correct outcomes from the current action canary activation epoch count: 200 correct reviewed links before attach verification/create eligibility and 50 correct reviewed creates before verified_capped. incorrect, duplicate, cross_hall, or unaudited persistently halts the relevant action latch, invalidates the active action calibration in the same transaction, and requires investigation. Re-entering that canary creates a fresh soak epoch after a new qualifying calibration is activated.

Caps are atomic rolling 24-hour limits: 100 links globally and 10 per hall; 20 creates globally and 3 per hall. Cap exhaustion is a normal policy stop: the suggestion remains reviewable and the execute run pauses without recording a permanent action failure. Cap reservation, policy/model/calibration revalidation, current-candidate requery, financial locks, mutation, and audit commit in one transaction.

List responses intentionally use only persisted revision/hash eligibility and do not reconstruct same-day candidates per row. This avoids an unbounded N+1 query path. Candidate existence, uniqueness, customer/department/registration/lane/date/items/totals, and financial locks are authoritatively rebuilt during preview/apply and again inside the mutation transaction. Treat a preview/apply stale-candidate rejection as a normal fail-closed refresh signal; monitor list latency and preview rejection rates during advisory/canary.

7. Halt, recovery, and rollback

Use the dedicated halt endpoint immediately on any unexplained result, duplicate, cross-hall action, missing audit, model mismatch, financial invariant, worker lease failure, or upstream revision anomaly. Halt disables legacy compatibility switches and preserves the reason. Generic config may disable a switch but cannot enable it.

Rollback means: follow the exact sequence in section 0; halt; stop new execute runs; retain audit/action/review evidence; reconcile affected orders and invoice collections; restore data only through a separately approved, previewed procedure; fix and redeploy; repeat advisory calibration and staged previews. Recovery from halted starts at off or advisory and requires new exact-SHA human approval. Never infer activation, soak completion, or production safety from green CI alone.