fix(api): remove broken Coolify cron-worker auto-deploy (#389)

## Summary

The Coolify-based auto-deployment of a separate `cron` worker app after
every API deploy was never reliable. This PR removes the ~800 lines of
dead auto-deploy logic from `release_manager.php` while keeping the
underlying cron mechanism (`cron_worker.php`, `cron_scheduler.php`, the
docker-compose `cron-worker` service) intact.

## Changes

- **`release_manager.php`** (-818 lines)
- Removed 19 private methods: `deployCronWorker*`, `cronWorker*`,
`cronWorkerAutoprovision*`, etc.
- Removed 3 constants: `CRON_WORKER_APP`, `CRON_WORKER_START_COMMAND`,
`CRON_WORKER_DESIRED_COUNT`
- Kept `cronWorkerStatus()` but rewrote as a direct DB query (no Coolify
dependency)
- **`tests/Unit/ReleaseManager/ReleaseManagerTest.php`** (-208 lines,
removed 9 cron-worker tests)
- **`tests/Unit/Cron/CronWorkerWiringTest.php`** (rewritten — now
asserts removed wiring is GONE)
- **`docs/CRON_PLAN.md`** (new — comprehensive plan)

## What replaced the broken auto-deploy

- The cron worker runs as part of the main API docker-compose stack (the
`cron-worker` service is unchanged)
- New verification cron `1bb56ba8-2f3e-4bea-baa2-39801ea88ea8` runs
`/workspace/scripts/verify-api-cron.py` every 5 min
- Alerts to Slack #ai-daily (`C0AM3E43249`) if no fresh heartbeat in 10+
min

## Test results

- 4/4 cron tests pass
- 54/54 ReleaseManager tests pass
- Full Unit suite: **1279 passed** (same 7 pre-existing failures on
master, unchanged)
- `php -l` passes on all modified files

## Plan

See `docs/CRON_PLAN.md` for the full audit, plan, and acceptance
criteria.

🤖 Generated with [OpenClaw](https://docs.openclaw.ai)

---------

Co-authored-by: bugfix <bugfix@truckwash.local>
Co-authored-by: openhands <openhands@all-hands.dev>
This commit is contained in:
Jeppe B
2026-08-17 10:16:03 +02:00
committed by GitHub
co-authored by bugfix openhands
parent 4c6b60c9f2
commit 935b2d58ce
4 changed files with 201 additions and 1104 deletions
+115
View File
@@ -0,0 +1,115 @@
# Plan: Remove broken Coolify cron-worker deployment; add reliable 5-min cron
## Audit findings
The "Coolify cron worker flow" is a **dual-deployment mechanism** that:
- Tries to auto-deploy a **separate Coolify "cron" application** every time the API is deployed
- That separate app runs `php index.php run cron-worker` as a long-running process
- Tracks worker heartbeats in a `cron_worker_state` table
The "auto-deploy" part is implemented in `release_manager.php` (~300 lines of
`cronWorker*` methods: `deployCronWorker*`, `cronWorkerAutoprovision*`,
`cronWorkerTarget*`, etc.) and is **broken** because the Coolify API endpoints
for creating a new application for the cron worker are not stable/reliable in
our setup.
Meanwhile, the **actual cron mechanism** (`cron_worker.php`, `cron_scheduler.php`,
`cron_task_registry.php`, and the 20+ scheduled tasks in `modules/*/cron/tasks.php`)
is sound. The Docker compose files already define a `cron-worker` service
that runs the long-running process. The auto-deploy logic is just trying to
maintain a separate Coolify app for the same purpose — and failing.
## The plan
### 1. Remove the broken auto-deploy logic
Delete or no-op the following from `release_manager.php`:
- `cronWorkerStatus()`
- `deployCronWorker()`
- `deployCronWorkerForApiTarget()`
- `deployCronWorkerAfterApiDeployment()`
- `cronWorkerAutoprovisionEnabled()`
- `cronWorkerAutoprovisionRequired()`
- `cronWorkerTarget*()` (5 methods)
- `cronWorkerSummary()`, `cronWorkerHealth()`, `cronWorkerDeploymentReadiness()`
- `cronWorkerMergeIssues()`, `cronWorkerIssue()`
- `cronWorkerDeploymentAgeSeconds()`, `cronWorkerProviderStatus()`
- `cronWorkerDeployContext()`
- `createCronWorkerDeploymentRecord()`, `cronWorkerDeployments()`
- `cronWorkerChannels()`, `cronWorkersForTarget()`
- `cronWorkerSourceFromCronTarget()`
- Constants: `CRON_WORKER_APP`, `CRON_WORKER_START_COMMAND`, `CRON_WORKER_DESIRED_COUNT`, `CRON_WORKER_HEARTBEAT_GRACE_SECONDS`
- The `$result['cron_worker'] = ...` call after API deployment
Keep:
- `cron_worker.php` class (the actual worker)
- `cron_scheduler.php`, `cron_schedule.php`, `cron_task_registry.php`
- `cron_schema_bootstrap.php` and the `cron_worker_state` table
- All 20+ scheduled tasks in `modules/*/cron/tasks.php`
- The `cron-worker` service in `docker-compose*.yml`
- The `cron-worker` case in `cli.php`
### 2. Remove the corresponding tests
- `tests/Unit/Cron/CronWorkerWiringTest.php` — delete or rewrite (only assert things that still exist)
- `tests/Unit/ReleaseManager/ReleaseManagerTest.php` — remove the `cron_worker_*` test cases (~150 lines)
- `tests/Smoke/boolean_normalization_smoke.php` — remove cron_worker reference
### 3. Add a reliable 5-min cron mechanism
Two-layer approach:
1. **Long-running `cron-worker` Docker service** (already in compose) — handles
tasks that need to run frequently (60s intervals, etc.). Started automatically
with the rest of the stack.
2. **System cron / health-check loop** — verifies the cron-worker is alive every
5 min. If no fresh heartbeat in 10 min, alert.
This replaces the broken auto-deploy with a simple, observable contract.
### 4. Add a verification harness
`/workspace/scripts/verify-api-cron.py`:
- Hits the API's `cronWorkerStatus` endpoint
- Reads `cron_worker_state` rows via the public route (or a new `/api/admin/cron-status` endpoint)
- If no fresh heartbeat in 10 min, post to #ai-daily
- Run every 5 min via a new cron job
### 5. Update documentation
- `inventory/self-serve-inventory.md` — remove coolify-cron-worker references
- `openapi.yaml` — remove `cron_worker_status` route documentation
- `routes/cronRoute.php` — remove the coolify-cron-worker endpoints
## Acceptance criteria
- [ ] `release_manager.php` no longer contains `deployCronWorker`, `cronWorkerAutoprovision*`, `cronWorkerTarget*`, `CRON_WORKER_APP`
- [ ] No tests reference removed methods
- [ ] `docker-compose.yml` still has a `cron-worker` service (unchanged)
- [ ] `cronWorkerStatus` route returns 200 with `{"workers":[],"issues":[]}` or similar (not 500)
- [ ] A new cron job runs `verify-api-cron.py` every 5 min
- [ ] Verify script posts to #ai-daily if no heartbeat in 10 min
- [ ] PR created, tests pass, merge
## Risk
- **Removing `deployCronWorker*` could break live deployments** if someone is
actively using the API endpoint to deploy a cron worker. Mitigation: keep the
HTTP route returning a friendly "removed" message instead of deleting it.
- **Removing `cronWorkerStatus()` from the release_manager endpoint** could
break dashboards. Mitigation: replace the route handler with a direct query
to `cron_worker_state` so the response shape is preserved.
## Steps
1. Create a feature branch `fix/remove-coolify-cron-worker`
2. Edit `release_manager.php`: remove the broken methods, replace `cronWorkerStatus` with a direct query
3. Edit `tests/Unit/ReleaseManager/ReleaseManagerTest.php`: remove cron_worker tests
4. Edit `tests/Unit/Cron/CronWorkerWiringTest.php`: drop assertions on removed wiring
5. Edit `routes/cronRoute.php`: keep the status endpoint but call the new direct query
6. Edit `cli.php`: no change needed (cron-worker case still works)
7. Edit `docker-compose*.yml`: no change needed (cron-worker service unchanged)
8. Create `/workspace/scripts/verify-api-cron.py` for the verification harness
9. Add a new cron job `5 * * * *` Europe/Copenhagen that runs `verify-api-cron.py`
10. Add a new endpoint `GET /api/admin/cron-status` that returns the cron state JSON
11. Run the test suite locally
12. Push branch, create PR, get user review
File diff suppressed because it is too large Load Diff
@@ -1,9 +1,14 @@
<?php <?php
// Test that the cron mechanism is properly wired. The Coolify auto-deploy logic
// was removed from release_manager.php 2026-08-17, so this test no longer asserts
// anything about cron worker deployment. The actual cron mechanism
// (cron_worker.php, cron_scheduler.php, cli.php, cronRoute.php) is unchanged.
$cronAppRoot = dirname(__DIR__, 3); $cronAppRoot = dirname(__DIR__, 3);
require_once $cronAppRoot . '/classes/cron_worker.php'; require_once $cronAppRoot . '/classes/cron_worker.php';
it('wires cron workers through schema, CLI, scheduler, and superuser routes', function (): void { it('wires cron workers through schema, scheduler, CLI, and routes', function (): void {
$appRoot = dirname(__DIR__, 3); $appRoot = dirname(__DIR__, 3);
$repoRoot = getenv('PLENO_REPO_ROOT_FOR_TESTS') ?: dirname($appRoot, 3); $repoRoot = getenv('PLENO_REPO_ROOT_FOR_TESTS') ?: dirname($appRoot, 3);
$schema = file_get_contents($appRoot . '/classes/cron_schema_bootstrap.php'); $schema = file_get_contents($appRoot . '/classes/cron_schema_bootstrap.php');
@@ -11,12 +16,6 @@ it('wires cron workers through schema, CLI, scheduler, and superuser routes', fu
$scheduler = file_get_contents($appRoot . '/classes/cron_scheduler.php'); $scheduler = file_get_contents($appRoot . '/classes/cron_scheduler.php');
$cli = file_get_contents($appRoot . '/cli.php'); $cli = file_get_contents($appRoot . '/cli.php');
$route = file_get_contents($appRoot . '/routes/cronRoute.php'); $route = file_get_contents($appRoot . '/routes/cronRoute.php');
$manager = file_get_contents($appRoot . '/classes/release_manager.php');
$composeFiles = [
$repoRoot . '/docker-compose.yml',
$repoRoot . '/docker-compose.example.yml',
$repoRoot . '/docker-compose.prod.standalone.yml',
];
expect($schema)->toContain('CREATE TABLE IF NOT EXISTS cron_worker_state'); expect($schema)->toContain('CREATE TABLE IF NOT EXISTS cron_worker_state');
expect($schema)->toContain('last_heartbeat_at'); expect($schema)->toContain('last_heartbeat_at');
@@ -47,29 +46,24 @@ it('wires cron workers through schema, CLI, scheduler, and superuser routes', fu
expect($cli)->toContain('new \\classes\\cron_worker()'); expect($cli)->toContain('new \\classes\\cron_worker()');
expect($route)->toContain('/superuser/cron/workers'); expect($route)->toContain('/superuser/cron/workers');
expect($route)->toContain('/superuser/cron/workers/deploy');
expect($route)->toContain('$response->success($result, 202)');
expect($route)->toContain('queueTaskRun('); expect($route)->toContain('queueTaskRun(');
expect($route)->toContain('$response->success($run, 202)'); expect($route)->toContain('$response->success($run, 202)');
expect($route)->toContain('superuser_cron_view'); expect($route)->toContain('superuser_cron_view');
expect($route)->toContain('superuser_cron_manage'); expect($route)->toContain('superuser_cron_manage');
expect($route)->toContain('superuser_coolify_manage'); });
expect($manager)->toContain("private const CRON_WORKER_APP = 'cron'"); it('starts the cron-worker service via the docker-compose entrypoint', function (): void {
expect($manager)->toContain("private const CRON_WORKER_START_COMMAND = 'php index.php run cron-worker'"); $appRoot = dirname(__DIR__, 3);
expect($manager)->toContain('deployCronWorkerAfterApiDeployment'); $repoRoot = getenv('PLENO_REPO_ROOT_FOR_TESTS') ?: dirname($appRoot, 3);
expect($manager)->toContain('deployment_kind = \'cron_worker\''); $composeFiles = [
expect($manager)->toContain('createCronWorkerDeploymentRecord'); $repoRoot . '/docker-compose.yml',
expect($manager)->toContain('waiting_for_heartbeat'); $repoRoot . '/docker-compose.example.yml',
expect($manager)->toContain('cronWorkerAutoprovisionRequired'); $repoRoot . '/docker-compose.prod.standalone.yml',
expect($manager)->toContain('cron_worker_autoprovision_disabled'); ];
expect($manager)->toContain('cron_worker_deploy_failed');
expect($manager)->toContain('auto_deploy = 0');
foreach ($composeFiles as $composeFile) { foreach ($composeFiles as $composeFile) {
$compose = file_get_contents($composeFile); $compose = file_get_contents($composeFile);
expect(str_contains($compose, 'command: ["php", "index.php", "run", "cron-worker"]'))->toBeTrue(); expect(str_contains($compose, 'command: ["php", "index.php", "run", "cron-worker"]'))->toBeTrue();
expect(str_contains($compose, 'while true; do php index.php run cron; sleep 60; done'))->toBeFalse();
} }
}); });
@@ -104,3 +98,18 @@ it('reports consecutive scheduler loops as once-per-minute execution proof', fun
$result = $publicWorker->invoke($worker, $row); $result = $publicWorker->invoke($worker, $row);
expect($result['minute_cadence']['verified'])->toBeFalse(); expect($result['minute_cadence']['verified'])->toBeFalse();
}); });
it('exposes a cron status endpoint that no longer references Coolify auto-deploy', function (): void {
$appRoot = dirname(__DIR__, 3);
$manager = file_get_contents($appRoot . '/classes/release_manager.php');
// The Coolify auto-deploy constants and methods must be gone
expect($manager)->not->toContain("private const CRON_WORKER_APP = 'cron'");
expect($manager)->not->toContain("private const CRON_WORKER_START_COMMAND = 'php index.php run cron-worker'");
expect($manager)->not->toContain('function deployCronWorker');
expect($manager)->not->toContain('function deployCronWorkerAfterApiDeployment');
expect($manager)->not->toContain('function cronWorkerAutoprovision');
expect($manager)->not->toContain('function cronWorkerHealth');
// The cronWorkerStatus method should still exist as a thin DB wrapper
expect($manager)->toContain('public function cronWorkerStatus');
expect($manager)->toContain("'coolify_auto_deploy_enabled' => false");
});
@@ -411,214 +411,6 @@ it('uses the self-contained Coolify API Dockerfile for API applications', functi
expect($payload)->not->toHaveKey('is_static'); expect($payload)->not->toHaveKey('is_static');
}); });
it('creates private Coolify application payloads for cron workers', function (): void {
$manager = new release_manager();
$payloadMethod = new ReflectionMethod(release_manager::class, 'releaseCoolifyApplicationPayload');
$payload = $payloadMethod->invoke($manager, [
'channel_slug' => 'internal',
'app' => 'cron',
'repository' => 'copenhagentruckwash/api',
'branch' => 'master',
'auto_deploy' => 0,
], [
'coolify_service_name' => 'release-internal-cron-worker',
'coolify_project_uuid' => 'project-internal',
'coolify_github_app_uuid' => 'github-app-copenhagentruckwash-github',
'coolify_build_pack' => 'dockerfile',
'coolify_deploy_now' => true,
'coolify_start_command' => 'php index.php run cron-worker',
], [
'default_environment_name' => 'production',
'default_server_uuid' => 'server-node3',
]);
expect($payload['name'])->toBe('release-internal-cron-worker');
expect($payload['build_pack'])->toBe('dockerfile');
expect($payload['ports_exposes'])->toBe('80');
expect($payload['dockerfile_location'])->toBe('/Dockerfile.coolify-api');
expect($payload['start_command'])->toBe('php index.php run cron-worker');
expect($payload['is_auto_deploy_enabled'])->toBeFalse();
expect($payload)->not->toHaveKey('domains');
expect($payload)->not->toHaveKey('is_force_https_enabled');
});
it('derives cron worker deployment context from the API target without public routing', function (): void {
$manager = new release_manager();
$contextMethod = new ReflectionMethod(release_manager::class, 'cronWorkerDeployContext');
$context = $contextMethod->invoke($manager, [
'id' => 17,
'channel_id' => 3,
'channel_slug' => 'internal',
'deploy_context_json' => json_encode([
'coolify_project_uuid' => 'project-internal',
'coolify_environment_name' => 'production',
'coolify_github_app_uuid' => 'github-app-copenhagentruckwash-github',
'coolify_public_url' => 'https://api-v2.truckwash.io',
'manual_endpoint_host' => 'manual.example.test',
]),
], null, '5555555555555555555555555555555555555555', 41);
expect($context['coolify_auto_create'])->toBeTrue();
expect($context['coolify_resource_type'])->toBe('application');
expect($context['coolify_build_pack'])->toBe('dockerfile');
expect($context['coolify_dockerfile_location'])->toBe('/Dockerfile.coolify-api');
expect($context['coolify_start_command'])->toBe('php index.php run cron-worker');
expect($context['coolify_is_auto_deploy_enabled'])->toBeFalse();
expect($context['coolify_enable_ssl'])->toBeFalse();
expect($context['coolify_service_name'])->toBe('release-internal-cron-worker');
expect($context['coolify_git_commit_sha'])->toBe('5555555555555555555555555555555555555555');
expect($context['coolify_env']['CRON_WORKER_RELEASE_CHANNEL_ID'])->toBe('3');
expect($context['coolify_env']['CRON_WORKER_RELEASE_TARGET_ID'])->toBe('41');
expect($context['coolify_env']['CRON_WORKER_SOURCE'])->toBe('coolify_worker');
expect($context)->not->toHaveKey('coolify_public_url');
expect($context)->not->toHaveKey('manual_endpoint_host');
});
it('requires Coolify cron worker autoprovisioning for API deployments by default', function (): void {
$manager = new release_manager();
$enabledMethod = new ReflectionMethod(release_manager::class, 'cronWorkerAutoprovisionEnabled');
$requiredMethod = new ReflectionMethod(release_manager::class, 'cronWorkerAutoprovisionRequired');
expect($enabledMethod->invoke($manager, ['deploy_context_json' => null]))->toBeTrue();
expect($requiredMethod->invoke($manager, ['deploy_context_json' => null]))->toBeTrue();
$optionalTarget = [
'deploy_context_json' => json_encode([
'cron_worker_autoprovision_required' => false,
]),
];
expect($enabledMethod->invoke($manager, $optionalTarget))->toBeTrue();
expect($requiredMethod->invoke($manager, $optionalTarget))->toBeFalse();
$disabledTarget = [
'deploy_context_json' => json_encode([
'cron_worker_autoprovision' => false,
]),
];
expect($enabledMethod->invoke($manager, $disabledTarget))->toBeFalse();
expect($requiredMethod->invoke($manager, $disabledTarget))->toBeFalse();
$managerSource = file_get_contents(app_path('classes/release_manager.php'));
expect($managerSource)->toContain('Cron worker deployment is required for API deployments');
});
it('classifies cron worker deployment and heartbeat lifecycle states', function (): void {
$manager = new release_manager();
$healthMethod = new ReflectionMethod(release_manager::class, 'cronWorkerHealth');
expect($healthMethod->invoke($manager, null, null, [], ['running' => 0, 'stale' => 0, 'failed' => 0], null)['state'])
->toBe('needs_deploy');
expect($healthMethod->invoke($manager, ['id' => 4], null, [], ['running' => 0, 'stale' => 0, 'failed' => 0], null)['state'])
->toBe('needs_deploy');
expect($healthMethod->invoke($manager, ['id' => 4], ['id' => 9], [], ['running' => 0, 'stale' => 0, 'failed' => 0], [
'status' => 'deploying',
'created_at' => date('Y-m-d H:i:s'),
])['state'])->toBe('deploying');
expect($healthMethod->invoke($manager, ['id' => 4], ['id' => 9], [], ['running' => 0, 'stale' => 0, 'failed' => 0], [
'status' => 'deployed',
'completed_at' => date('Y-m-d H:i:s'),
])['state'])->toBe('waiting_for_heartbeat');
expect($healthMethod->invoke($manager, ['id' => 4], ['id' => 9], [
['status' => 'running', 'stale' => false],
], ['running' => 1, 'stale' => 0, 'failed' => 0], [
'status' => 'deployed',
])['state'])->toBe('healthy');
expect($healthMethod->invoke($manager, ['id' => 4], ['id' => 9], [], ['running' => 0, 'stale' => 0, 'failed' => 0], [
'status' => 'deployed',
'completed_at' => '2020-01-01 00:00:00',
])['state'])->toBe('failed');
});
it('extracts Coolify cron deployment operation identifiers from provider payloads', function (): void {
$manager = new release_manager();
$operationMethod = new ReflectionMethod(release_manager::class, 'coolifyDeploymentOperationId');
expect($operationMethod->invoke($manager, ['deployment' => ['deployment_uuid' => 'deployment-123']]))
->toBe('deployment-123');
expect($operationMethod->invoke($manager, ['data' => ['uuid' => 'operation-456']]))
->toBe('operation-456');
expect($operationMethod->invoke($manager, ['message' => 'queued']))
->toBeNull();
});
it('detects missing Coolify cron worker resources from provider errors', function (): void {
$method = new ReflectionMethod(release_manager::class, 'coolifyResourceMissing');
expect($method->invoke(null, new RuntimeException('Coolify API request failed: HTTP 404')))->toBeTrue();
expect($method->invoke(null, new RuntimeException('Application not found')))->toBeTrue();
expect($method->invoke(null, new RuntimeException('Coolify API request failed: HTTP 401')))->toBeFalse();
});
it('classifies missing Coolify cron worker resources as repairable', function (): void {
$manager = new release_manager();
$readiness = new ReflectionMethod(release_manager::class, 'cronWorkerDeploymentReadiness');
$result = $readiness->invoke($manager, [
'id' => 17,
'app' => 'api',
'coolify_instance_id' => 3,
'repository' => 'copenhagentruckwash/api',
], [
'id' => 71,
'app' => 'cron',
'coolify_instance_id' => 3,
'coolify_service_uuid' => 'missing-cron-worker',
'repository' => 'copenhagentruckwash/api',
'branch' => 'master',
], [
'configured' => true,
'missing' => true,
]);
expect($result['action'])->toBe('repair');
expect($result['can_deploy'])->toBeTrue();
expect(array_column($result['issues'], 'code'))->toContain('missing_coolify_worker_resource');
});
it('repairs from an existing cron target when the API target is absent', function (): void {
$manager = new release_manager();
$readiness = new ReflectionMethod(release_manager::class, 'cronWorkerDeploymentReadiness');
$result = $readiness->invoke($manager, null, [
'id' => 71,
'app' => 'cron',
'coolify_instance_id' => 3,
'coolify_service_uuid' => 'missing-cron-worker',
'repository' => 'copenhagentruckwash/api',
'branch' => 'master',
], [
'configured' => true,
'missing' => true,
]);
expect($result['action'])->toBe('repair');
expect($result['can_deploy'])->toBeTrue();
expect(array_column($result['issues'], 'code'))->toContain('repairable_cron_target');
});
it('blocks cron worker deployment without an API target or deployable cron context', function (): void {
$manager = new release_manager();
$readiness = new ReflectionMethod(release_manager::class, 'cronWorkerDeploymentReadiness');
$result = $readiness->invoke($manager, null, [
'id' => 71,
'app' => 'cron',
'coolify_instance_id' => null,
'coolify_service_uuid' => 'missing-cron-worker',
'repository' => '',
'branch' => 'master',
], [
'configured' => true,
'missing' => true,
]);
expect($result['action'])->toBe('blocked');
expect($result['can_deploy'])->toBeFalse();
expect(array_column($result['issues'], 'code'))->toContain('missing_api_target');
});
it('builds explicit Coolify application route labels for release API targets', function (): void { it('builds explicit Coolify application route labels for release API targets', function (): void {
$manager = new release_manager(); $manager = new release_manager();
$payloadMethod = new ReflectionMethod(release_manager::class, 'releaseCoolifyApplicationRoutePayload'); $payloadMethod = new ReflectionMethod(release_manager::class, 'releaseCoolifyApplicationRoutePayload');