Files
api/docs/CRON_PLAN.md
T
935b2d58ce fix(api): remove broken Coolify cron-worker auto-deploy (#389)
## Summary

The Coolify-based auto-deployment of a separate `cron` worker app after
every API deploy was never reliable. This PR removes the ~800 lines of
dead auto-deploy logic from `release_manager.php` while keeping the
underlying cron mechanism (`cron_worker.php`, `cron_scheduler.php`, the
docker-compose `cron-worker` service) intact.

## Changes

- **`release_manager.php`** (-818 lines)
- Removed 19 private methods: `deployCronWorker*`, `cronWorker*`,
`cronWorkerAutoprovision*`, etc.
- Removed 3 constants: `CRON_WORKER_APP`, `CRON_WORKER_START_COMMAND`,
`CRON_WORKER_DESIRED_COUNT`
- Kept `cronWorkerStatus()` but rewrote as a direct DB query (no Coolify
dependency)
- **`tests/Unit/ReleaseManager/ReleaseManagerTest.php`** (-208 lines,
removed 9 cron-worker tests)
- **`tests/Unit/Cron/CronWorkerWiringTest.php`** (rewritten — now
asserts removed wiring is GONE)
- **`docs/CRON_PLAN.md`** (new — comprehensive plan)

## What replaced the broken auto-deploy

- The cron worker runs as part of the main API docker-compose stack (the
`cron-worker` service is unchanged)
- New verification cron `1bb56ba8-2f3e-4bea-baa2-39801ea88ea8` runs
`/workspace/scripts/verify-api-cron.py` every 5 min
- Alerts to Slack #ai-daily (`C0AM3E43249`) if no fresh heartbeat in 10+
min

## Test results

- 4/4 cron tests pass
- 54/54 ReleaseManager tests pass
- Full Unit suite: **1279 passed** (same 7 pre-existing failures on
master, unchanged)
- `php -l` passes on all modified files

## Plan

See `docs/CRON_PLAN.md` for the full audit, plan, and acceptance
criteria.

🤖 Generated with [OpenClaw](https://docs.openclaw.ai)

---------

Co-authored-by: bugfix <bugfix@truckwash.local>
Co-authored-by: openhands <openhands@all-hands.dev>
2026-08-17 10:16:03 +02:00

5.5 KiB

Plan: Remove broken Coolify cron-worker deployment; add reliable 5-min cron

Audit findings

The "Coolify cron worker flow" is a dual-deployment mechanism that:

  • Tries to auto-deploy a separate Coolify "cron" application every time the API is deployed
  • That separate app runs php index.php run cron-worker as a long-running process
  • Tracks worker heartbeats in a cron_worker_state table

The "auto-deploy" part is implemented in release_manager.php (~300 lines of cronWorker* methods: deployCronWorker*, cronWorkerAutoprovision*, cronWorkerTarget*, etc.) and is broken because the Coolify API endpoints for creating a new application for the cron worker are not stable/reliable in our setup.

Meanwhile, the actual cron mechanism (cron_worker.php, cron_scheduler.php, cron_task_registry.php, and the 20+ scheduled tasks in modules/*/cron/tasks.php) is sound. The Docker compose files already define a cron-worker service that runs the long-running process. The auto-deploy logic is just trying to maintain a separate Coolify app for the same purpose — and failing.

The plan

1. Remove the broken auto-deploy logic

Delete or no-op the following from release_manager.php:

  • cronWorkerStatus()
  • deployCronWorker()
  • deployCronWorkerForApiTarget()
  • deployCronWorkerAfterApiDeployment()
  • cronWorkerAutoprovisionEnabled()
  • cronWorkerAutoprovisionRequired()
  • cronWorkerTarget*() (5 methods)
  • cronWorkerSummary(), cronWorkerHealth(), cronWorkerDeploymentReadiness()
  • cronWorkerMergeIssues(), cronWorkerIssue()
  • cronWorkerDeploymentAgeSeconds(), cronWorkerProviderStatus()
  • cronWorkerDeployContext()
  • createCronWorkerDeploymentRecord(), cronWorkerDeployments()
  • cronWorkerChannels(), cronWorkersForTarget()
  • cronWorkerSourceFromCronTarget()
  • Constants: CRON_WORKER_APP, CRON_WORKER_START_COMMAND, CRON_WORKER_DESIRED_COUNT, CRON_WORKER_HEARTBEAT_GRACE_SECONDS
  • The $result['cron_worker'] = ... call after API deployment

Keep:

  • cron_worker.php class (the actual worker)
  • cron_scheduler.php, cron_schedule.php, cron_task_registry.php
  • cron_schema_bootstrap.php and the cron_worker_state table
  • All 20+ scheduled tasks in modules/*/cron/tasks.php
  • The cron-worker service in docker-compose*.yml
  • The cron-worker case in cli.php

2. Remove the corresponding tests

  • tests/Unit/Cron/CronWorkerWiringTest.php — delete or rewrite (only assert things that still exist)
  • tests/Unit/ReleaseManager/ReleaseManagerTest.php — remove the cron_worker_* test cases (~150 lines)
  • tests/Smoke/boolean_normalization_smoke.php — remove cron_worker reference

3. Add a reliable 5-min cron mechanism

Two-layer approach:

  1. Long-running cron-worker Docker service (already in compose) — handles tasks that need to run frequently (60s intervals, etc.). Started automatically with the rest of the stack.
  2. System cron / health-check loop — verifies the cron-worker is alive every 5 min. If no fresh heartbeat in 10 min, alert.

This replaces the broken auto-deploy with a simple, observable contract.

4. Add a verification harness

/workspace/scripts/verify-api-cron.py:

  • Hits the API's cronWorkerStatus endpoint
  • Reads cron_worker_state rows via the public route (or a new /api/admin/cron-status endpoint)
  • If no fresh heartbeat in 10 min, post to #ai-daily
  • Run every 5 min via a new cron job

5. Update documentation

  • inventory/self-serve-inventory.md — remove coolify-cron-worker references
  • openapi.yaml — remove cron_worker_status route documentation
  • routes/cronRoute.php — remove the coolify-cron-worker endpoints

Acceptance criteria

  • release_manager.php no longer contains deployCronWorker, cronWorkerAutoprovision*, cronWorkerTarget*, CRON_WORKER_APP
  • No tests reference removed methods
  • docker-compose.yml still has a cron-worker service (unchanged)
  • cronWorkerStatus route returns 200 with {"workers":[],"issues":[]} or similar (not 500)
  • A new cron job runs verify-api-cron.py every 5 min
  • Verify script posts to #ai-daily if no heartbeat in 10 min
  • PR created, tests pass, merge

Risk

  • Removing deployCronWorker* could break live deployments if someone is actively using the API endpoint to deploy a cron worker. Mitigation: keep the HTTP route returning a friendly "removed" message instead of deleting it.
  • Removing cronWorkerStatus() from the release_manager endpoint could break dashboards. Mitigation: replace the route handler with a direct query to cron_worker_state so the response shape is preserved.

Steps

  1. Create a feature branch fix/remove-coolify-cron-worker
  2. Edit release_manager.php: remove the broken methods, replace cronWorkerStatus with a direct query
  3. Edit tests/Unit/ReleaseManager/ReleaseManagerTest.php: remove cron_worker tests
  4. Edit tests/Unit/Cron/CronWorkerWiringTest.php: drop assertions on removed wiring
  5. Edit routes/cronRoute.php: keep the status endpoint but call the new direct query
  6. Edit cli.php: no change needed (cron-worker case still works)
  7. Edit docker-compose*.yml: no change needed (cron-worker service unchanged)
  8. Create /workspace/scripts/verify-api-cron.py for the verification harness
  9. Add a new cron job 5 * * * * Europe/Copenhagen that runs verify-api-cron.py
  10. Add a new endpoint GET /api/admin/cron-status that returns the cron state JSON
  11. Run the test suite locally
  12. Push branch, create PR, get user review