Ledger failure policy

What happens when an agent_runs run fails.

Why this exists

src/pipeline/failureRouter.ts has classified failures, applied backoff and escalated since Phase 1e — but only for the daemon's issues.status state machine. Everything dispatched through the ledger (mainark dispatch, coordinate --watch, cursor-cloud) had no policy at all.

The practical consequence, observed on a real dispatch: a run failed with [usage_limit_exceeded] from the Cursor API, and the issue stayed claimed in_progress indefinitely. Nothing retried it, nothing released the claim, and nothing told anyone. Because dispatch auto-pick only selects work_status=todo, that issue was permanently invisible to every future dispatcher.

The table

Caps mirror the daemon's: rate_limit 5, timeout 3, network 3 retries. The attempt number counts the run that just failed, so attempt cap still retries (on the last backoff tier) and attempt cap + 1 escalates. An earlier version compared attempt < cap, allowing one retry fewer than the daemon and leaving the 15m tier unreachable.

Failures that are recorded but not acted on

The kind is always recorded, but three cases never release the claim:

A permanent failure's Slack note says "not retried — needs a human (issue left In Progress)", because the policy leaves work_status alone. It used to say "moved to Todo", which was wrong.

Design notes

Retry state lives on the run, not the issue. issues.next_retry_at and retry_count belong to the daemon's state machine; reusing them would conflate the two task graphs (see dual task graphs). A ledger retry is a new agent_run, so the failed run carries failure_kind and next_retry_at; the continuous-dispatch sweep honours them via isBackoffPending() (listRunsDueForRetry() exists for callers that want the due list directly).

Transient failures release the claim. work_status goes back to todo, or nothing would ever pick the issue up again. The run row keeps the history.

Permanent failures leave work_status alone. Moving them to paused was tried and reverted. Auto-dispatch only picks todo, so an issue sitting at in_progress with a failed run is already safe from being re-picked — paused bought no protection. What it did cost is recovery: prepareImplementDispatch rejects paused, so parking the issue there breaks the documented mainark dispatch <ID> --retry path. The bug being fixed here is silence, and outcome=needs_human plus Slack fixes that without touching the claim.

Routing is idempotent. A run that already has a failure_kind is skipped, so a coordinator tick seeing the same terminal run twice will not double-notify or restart a backoff that is already counting down.

A retried run clears needs_human. recordPollFailure sets it when it gives up polling; if the policy is going to retry, the review queue should not show the issue as stuck.

Where it runs

Two places, because failures happen in two places:

• At dispatch. When the provider rejects the create call — the [usage_limit_exceeded] case above — dispatch() writes the run failed and returns normally. The run is never in flight. The continuous-dispatch sweep routes it immediately; the tick also routes every implement run that is failed with no failure_kind, which covers dispatches from any caller.
• In flight. runCoordinatorTick routes every run that reached failed while being polled.

The first bullet was missing from the original version: its tests called routeAgentRunFailure directly and never went through dispatch(), so the exact failure that motivated the policy was never routed. The result is returned as routedFailures on the tick result.

Who acts on a scheduled retry

coordinate --watch --auto-dispatch — see continuous dispatch. It honours next_retry_at, so an issue the policy released to todo is not re-dispatched until its backoff has elapsed.

Without --auto-dispatch, a scheduled retry simply means the issue is dispatchable again and mainark dispatch <ID> picks it up (no --retry needed, because the policy already returned it to todo).