Jeppe B
801c8e1f6d
Adds `scripts/xlvask-automation-migrate.php` mirroring the existing schema-script pattern (`check` / `apply --yes`), a new AUTOMATION_RUNBOOK §2a documenting both operator entry points, and `XLVaskAutomationMigrateScriptTest` pinning the gate, the WD check, and the bootstrap references. Production autopilot-runs (POST /modules/xlvask/services/usage/autopilot-runs) was returning 500 with: ``` XL Vask automation schema is not ready. Apply migration 20260804_xlvask_ai_auto_policy_v2 explicitly. ``` because no operator had invoked the gated `applyExplicitMigration()` since PR #348 shipped the migration class. The frontend half of the fix is the companion change in pleno-vue#280 (missing `minimax_integration_enabled` key). Operator action required once merged: ``` php index.php run xlvask-automation-migrate # or php scripts/xlvask-automation-migrate.php apply --yes ``` Both produce identical JSON status; retain the artifact and rerun `check` to confirm postflight is green. Diff: +140/-0 (3 files). Tests: 87/87 XL Vask unit + 5/5 new migration script test pass.
87 lines
8.3 KiB
Markdown
87 lines
8.3 KiB
Markdown
# XL Vask AI automation runbook
|
|
|
|
This runbook is an operator procedure. None of its gates are applied by deployment, HTTP GETs, constructors, or workers. Every production-changing step requires a human approval tied to the exact deployed backend and frontend SHAs.
|
|
|
|
## 0. Deploy-order and rollback invariant
|
|
|
|
The legacy attachment/creation config values are kill switches, but an old backend treats them as direct enable switches. Old code cannot interpret the new policy stages, calibration identity, rolling caps, action latches, or canary soak. Therefore old-backend traffic is forbidden whenever either legacy switch is true, including during a new-policy canary.
|
|
|
|
Use this exact forward sequence:
|
|
|
|
1. While the old backend is still serving, set both legacy automatic-order switches to false through the approved config procedure and verify the persisted values from every serving instance.
|
|
2. Stop/disable old XL Vask automation workers and verify there is no active automatic run. Ordinary XL Vask synchronization may continue.
|
|
3. Deploy the new backend with policy effectively `off`; verify ordinary synchronization still completes and scheduled automation no-ops while schema readiness is false.
|
|
4. Run read-only migration preflight, then the separately approved explicit additive migration. If it is partial or fails, keep the new backend deployed, policy `off`, both legacy switches false, and workers no-op; repair or complete the migration before continuing. Never route old code as a partial-migration workaround.
|
|
5. Verify migration readiness and the new backend SHA, then deploy/verify the compatible frontend. Only after that generate advisory evidence and use preview-bound policy transitions.
|
|
|
|
Use this exact rollback sequence before any old-code traffic:
|
|
|
|
1. Keep all traffic on the new backend, call the dedicated halt endpoint, and verify policy `halted` plus both persisted legacy switches false.
|
|
2. Stop new-backend automation workers, wait for or safely reconcile the active run, and verify no financial mutation is in flight.
|
|
3. Roll back the frontend if required, then deploy the old backend with both legacy switches still false. Verify ordinary sync only.
|
|
4. Do not re-enable either legacy switch on old code. Recovery of automatic actions requires redeploying the new policy-aware backend and repeating readiness, advisory calibration, canary, and soak.
|
|
|
|
## 1. Read-only preflight
|
|
|
|
1. Record the backend/frontend SHAs, environment, operator, invoice period, and scanner-hall scope.
|
|
2. Call the scoped capabilities and admin-readiness GETs with `dateFrom` and `dateTo`.
|
|
3. Confirm `migration.ready`, `missing_tables`, `missing_columns`, `missing_indexes`, `preflight_conflicts`, `worker_healthy`, WashId uniqueness, planner identity, resolved model, active run, scoped eligible counts, rolling budgets, and reviewed soak counts.
|
|
4. Stop if dates are invalid, hall scope is empty, an execute run is active, identity changed, a latch is halted, or any readiness field fails closed.
|
|
|
|
## 2. Explicit schema migration
|
|
|
|
Use the controlled database migration procedure to invoke only
|
|
`migration_20260804_xlvask_ai_auto_policy_v2::apply()`. First retain its read-only
|
|
`preflight()` output. Review the additive SQL and backup/restore point, approve the exact SHA, run it once, retain the returned status, and rerun readiness. Do not invoke `applyExplicitMigration()` from a request, worker, cron task, or application startup.
|
|
If preflight reports multiple legacy execute runs in `queued`, `running`, or `retry_wait`, stop. Reconcile those runs through a separately approved operational procedure; the migration never auto-resolves or modifies the conflicting run records.
|
|
|
|
### 2a. Operator entry points
|
|
|
|
There are two equivalent ways to apply the migration from a privileged
|
|
container with the configured DB credentials. Both call the same gated
|
|
`migration_20260804_xlvask_ai_auto_policy_v2::apply()` entry point and
|
|
produce identical status output. Pick whichever fits the workflow.
|
|
|
|
```
|
|
# Option A — standalone script (mirrors scripts/account-deletion-schema.php)
|
|
php scripts/xlvask-automation-migrate.php check # read-only preflight
|
|
php scripts/xlvask-automation-migrate.php apply --yes # apply, gated by --yes
|
|
|
|
# Option B — CLI dispatcher inside index.php (defines WD + composes bootstrap)
|
|
php index.php run xlvask-automation-migrate # preflight, applies if !ready
|
|
```
|
|
|
|
Both exit 0 when `ready=true` and 1 otherwise. Always retain the JSON
|
|
status artifact for the audit log and rerun `check` to confirm the
|
|
postflight is green.
|
|
|
|
## 3. WashId uniqueness
|
|
|
|
Inspect normalized duplicate WashIds. Resolve conflicts through an independently approved data procedure. Only then use the guarded uniqueness activation with the exact typed phrase. Recheck the generated normalized column and unique index before any automatic action.
|
|
|
|
## 4. Advisory evidence and calibration
|
|
|
|
Keep policy at `advisory`. Run explicit `dry_run` requests to import and persist plans, or `replay` for cache-only read-only evaluation. Review suggestions in hall scope. Label exact OpenAI attach/create suggestions; model identity, prompt hash, schema hash, policy version, resolved model, and chronological label snapshot are part of the artifact identity. Generate inactive backtests, independently review qualification thresholds and contradictions, then activate the exact artifact hash with its typed phrase.
|
|
|
|
## 5. Staged policy transitions
|
|
|
|
Every transition uses a bounded human reason, server-generated policy preview, exact confirmation phrase, and apply-time revalidation. The reason is bound into the preview hash and retained in the immutable policy event:
|
|
|
|
`off` -> `advisory` -> `ai_attach_canary` -> `ai_attach_verified` -> `ai_create_canary` -> `verified_capped`
|
|
|
|
Stages may not be skipped. An active execute run, stale preview, changed policy version, changed model/planner identity, missing exact calibration, incomplete reviewed soak, invalid period scope, or exhausted readiness gate blocks promotion.
|
|
|
|
## 6. Reviewed soak and caps
|
|
|
|
Volume alone never completes soak. Every auto-accepted action must be adjudicated. Only explicit `correct` outcomes from the current action canary activation epoch count: 200 correct reviewed links before attach verification/create eligibility and 50 correct reviewed creates before `verified_capped`. `incorrect`, `duplicate`, `cross_hall`, or `unaudited` persistently halts the relevant action latch, invalidates the active action calibration in the same transaction, and requires investigation. Re-entering that canary creates a fresh soak epoch after a new qualifying calibration is activated.
|
|
|
|
Caps are atomic rolling 24-hour limits: 100 links globally and 10 per hall; 20 creates globally and 3 per hall. Cap exhaustion is a normal policy stop: the suggestion remains reviewable and the execute run pauses without recording a permanent action failure. Cap reservation, policy/model/calibration revalidation, current-candidate requery, financial locks, mutation, and audit commit in one transaction.
|
|
|
|
List responses intentionally use only persisted revision/hash eligibility and do not reconstruct same-day candidates per row. This avoids an unbounded N+1 query path. Candidate existence, uniqueness, customer/department/registration/lane/date/items/totals, and financial locks are authoritatively rebuilt during preview/apply and again inside the mutation transaction. Treat a preview/apply stale-candidate rejection as a normal fail-closed refresh signal; monitor list latency and preview rejection rates during advisory/canary.
|
|
|
|
## 7. Halt, recovery, and rollback
|
|
|
|
Use the dedicated halt endpoint immediately on any unexplained result, duplicate, cross-hall action, missing audit, model mismatch, financial invariant, worker lease failure, or upstream revision anomaly. Halt disables legacy compatibility switches and preserves the reason. Generic config may disable a switch but cannot enable it.
|
|
|
|
Rollback means: follow the exact sequence in section 0; halt; stop new execute runs; retain audit/action/review evidence; reconcile affected orders and invoice collections; restore data only through a separately approved, previewed procedure; fix and redeploy; repeat advisory calibration and staged previews. Recovery from `halted` starts at `off` or `advisory` and requires new exact-SHA human approval. Never infer activation, soak completion, or production safety from green CI alone.
|