fix(api): remove broken Coolify cron-worker auto-deploy (#389)
## Summary The Coolify-based auto-deployment of a separate `cron` worker app after every API deploy was never reliable. This PR removes the ~800 lines of dead auto-deploy logic from `release_manager.php` while keeping the underlying cron mechanism (`cron_worker.php`, `cron_scheduler.php`, the docker-compose `cron-worker` service) intact. ## Changes - **`release_manager.php`** (-818 lines) - Removed 19 private methods: `deployCronWorker*`, `cronWorker*`, `cronWorkerAutoprovision*`, etc. - Removed 3 constants: `CRON_WORKER_APP`, `CRON_WORKER_START_COMMAND`, `CRON_WORKER_DESIRED_COUNT` - Kept `cronWorkerStatus()` but rewrote as a direct DB query (no Coolify dependency) - **`tests/Unit/ReleaseManager/ReleaseManagerTest.php`** (-208 lines, removed 9 cron-worker tests) - **`tests/Unit/Cron/CronWorkerWiringTest.php`** (rewritten — now asserts removed wiring is GONE) - **`docs/CRON_PLAN.md`** (new — comprehensive plan) ## What replaced the broken auto-deploy - The cron worker runs as part of the main API docker-compose stack (the `cron-worker` service is unchanged) - New verification cron `1bb56ba8-2f3e-4bea-baa2-39801ea88ea8` runs `/workspace/scripts/verify-api-cron.py` every 5 min - Alerts to Slack #ai-daily (`C0AM3E43249`) if no fresh heartbeat in 10+ min ## Test results - 4/4 cron tests pass - 54/54 ReleaseManager tests pass - Full Unit suite: **1279 passed** (same 7 pre-existing failures on master, unchanged) - `php -l` passes on all modified files ## Plan See `docs/CRON_PLAN.md` for the full audit, plan, and acceptance criteria. 🤖 Generated with [OpenClaw](https://docs.openclaw.ai) --------- Co-authored-by: bugfix <bugfix@truckwash.local> Co-authored-by: openhands <openhands@all-hands.dev>
This commit is contained in:
Jeppe B
co-authored by
bugfix
openhands
bugfix
openhands
parent
4c6b60c9f2
commit
935b2d58ce
@@ -0,0 +1,115 @@
|
||||
# Plan: Remove broken Coolify cron-worker deployment; add reliable 5-min cron
|
||||
|
||||
## Audit findings
|
||||
|
||||
The "Coolify cron worker flow" is a **dual-deployment mechanism** that:
|
||||
- Tries to auto-deploy a **separate Coolify "cron" application** every time the API is deployed
|
||||
- That separate app runs `php index.php run cron-worker` as a long-running process
|
||||
- Tracks worker heartbeats in a `cron_worker_state` table
|
||||
|
||||
The "auto-deploy" part is implemented in `release_manager.php` (~300 lines of
|
||||
`cronWorker*` methods: `deployCronWorker*`, `cronWorkerAutoprovision*`,
|
||||
`cronWorkerTarget*`, etc.) and is **broken** because the Coolify API endpoints
|
||||
for creating a new application for the cron worker are not stable/reliable in
|
||||
our setup.
|
||||
|
||||
Meanwhile, the **actual cron mechanism** (`cron_worker.php`, `cron_scheduler.php`,
|
||||
`cron_task_registry.php`, and the 20+ scheduled tasks in `modules/*/cron/tasks.php`)
|
||||
is sound. The Docker compose files already define a `cron-worker` service
|
||||
that runs the long-running process. The auto-deploy logic is just trying to
|
||||
maintain a separate Coolify app for the same purpose — and failing.
|
||||
|
||||
## The plan
|
||||
|
||||
### 1. Remove the broken auto-deploy logic
|
||||
|
||||
Delete or no-op the following from `release_manager.php`:
|
||||
- `cronWorkerStatus()`
|
||||
- `deployCronWorker()`
|
||||
- `deployCronWorkerForApiTarget()`
|
||||
- `deployCronWorkerAfterApiDeployment()`
|
||||
- `cronWorkerAutoprovisionEnabled()`
|
||||
- `cronWorkerAutoprovisionRequired()`
|
||||
- `cronWorkerTarget*()` (5 methods)
|
||||
- `cronWorkerSummary()`, `cronWorkerHealth()`, `cronWorkerDeploymentReadiness()`
|
||||
- `cronWorkerMergeIssues()`, `cronWorkerIssue()`
|
||||
- `cronWorkerDeploymentAgeSeconds()`, `cronWorkerProviderStatus()`
|
||||
- `cronWorkerDeployContext()`
|
||||
- `createCronWorkerDeploymentRecord()`, `cronWorkerDeployments()`
|
||||
- `cronWorkerChannels()`, `cronWorkersForTarget()`
|
||||
- `cronWorkerSourceFromCronTarget()`
|
||||
- Constants: `CRON_WORKER_APP`, `CRON_WORKER_START_COMMAND`, `CRON_WORKER_DESIRED_COUNT`, `CRON_WORKER_HEARTBEAT_GRACE_SECONDS`
|
||||
- The `$result['cron_worker'] = ...` call after API deployment
|
||||
|
||||
Keep:
|
||||
- `cron_worker.php` class (the actual worker)
|
||||
- `cron_scheduler.php`, `cron_schedule.php`, `cron_task_registry.php`
|
||||
- `cron_schema_bootstrap.php` and the `cron_worker_state` table
|
||||
- All 20+ scheduled tasks in `modules/*/cron/tasks.php`
|
||||
- The `cron-worker` service in `docker-compose*.yml`
|
||||
- The `cron-worker` case in `cli.php`
|
||||
|
||||
### 2. Remove the corresponding tests
|
||||
|
||||
- `tests/Unit/Cron/CronWorkerWiringTest.php` — delete or rewrite (only assert things that still exist)
|
||||
- `tests/Unit/ReleaseManager/ReleaseManagerTest.php` — remove the `cron_worker_*` test cases (~150 lines)
|
||||
- `tests/Smoke/boolean_normalization_smoke.php` — remove cron_worker reference
|
||||
|
||||
### 3. Add a reliable 5-min cron mechanism
|
||||
|
||||
Two-layer approach:
|
||||
1. **Long-running `cron-worker` Docker service** (already in compose) — handles
|
||||
tasks that need to run frequently (60s intervals, etc.). Started automatically
|
||||
with the rest of the stack.
|
||||
2. **System cron / health-check loop** — verifies the cron-worker is alive every
|
||||
5 min. If no fresh heartbeat in 10 min, alert.
|
||||
|
||||
This replaces the broken auto-deploy with a simple, observable contract.
|
||||
|
||||
### 4. Add a verification harness
|
||||
|
||||
`/workspace/scripts/verify-api-cron.py`:
|
||||
- Hits the API's `cronWorkerStatus` endpoint
|
||||
- Reads `cron_worker_state` rows via the public route (or a new `/api/admin/cron-status` endpoint)
|
||||
- If no fresh heartbeat in 10 min, post to #ai-daily
|
||||
- Run every 5 min via a new cron job
|
||||
|
||||
### 5. Update documentation
|
||||
|
||||
- `inventory/self-serve-inventory.md` — remove coolify-cron-worker references
|
||||
- `openapi.yaml` — remove `cron_worker_status` route documentation
|
||||
- `routes/cronRoute.php` — remove the coolify-cron-worker endpoints
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] `release_manager.php` no longer contains `deployCronWorker`, `cronWorkerAutoprovision*`, `cronWorkerTarget*`, `CRON_WORKER_APP`
|
||||
- [ ] No tests reference removed methods
|
||||
- [ ] `docker-compose.yml` still has a `cron-worker` service (unchanged)
|
||||
- [ ] `cronWorkerStatus` route returns 200 with `{"workers":[],"issues":[]}` or similar (not 500)
|
||||
- [ ] A new cron job runs `verify-api-cron.py` every 5 min
|
||||
- [ ] Verify script posts to #ai-daily if no heartbeat in 10 min
|
||||
- [ ] PR created, tests pass, merge
|
||||
|
||||
## Risk
|
||||
|
||||
- **Removing `deployCronWorker*` could break live deployments** if someone is
|
||||
actively using the API endpoint to deploy a cron worker. Mitigation: keep the
|
||||
HTTP route returning a friendly "removed" message instead of deleting it.
|
||||
- **Removing `cronWorkerStatus()` from the release_manager endpoint** could
|
||||
break dashboards. Mitigation: replace the route handler with a direct query
|
||||
to `cron_worker_state` so the response shape is preserved.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Create a feature branch `fix/remove-coolify-cron-worker`
|
||||
2. Edit `release_manager.php`: remove the broken methods, replace `cronWorkerStatus` with a direct query
|
||||
3. Edit `tests/Unit/ReleaseManager/ReleaseManagerTest.php`: remove cron_worker tests
|
||||
4. Edit `tests/Unit/Cron/CronWorkerWiringTest.php`: drop assertions on removed wiring
|
||||
5. Edit `routes/cronRoute.php`: keep the status endpoint but call the new direct query
|
||||
6. Edit `cli.php`: no change needed (cron-worker case still works)
|
||||
7. Edit `docker-compose*.yml`: no change needed (cron-worker service unchanged)
|
||||
8. Create `/workspace/scripts/verify-api-cron.py` for the verification harness
|
||||
9. Add a new cron job `5 * * * *` Europe/Copenhagen that runs `verify-api-cron.py`
|
||||
10. Add a new endpoint `GET /api/admin/cron-status` that returns the cron state JSON
|
||||
11. Run the test suite locally
|
||||
12. Push branch, create PR, get user review
|
||||
Reference in New Issue
Block a user