ADR-003: Incidents — health contract and the detect → resolve loop

Date: 2026-10-10
Status: Proposed
Repos: Mainark (incidents, watcher, UI), Fulreach (GET /internal/health)
Domain model: ADR-004 (Product, Goal, Task, Run, PR, Service, Deployment, Incident)

Decision

Coolify runs the infrastructure. Mainark understands what is happening and coordinates what happens next.

Mainark gains incidents: something in production needs attention. An incident is detected from two sources only — a service's health contract and Coolify deploy events — and Mainark drives it toward resolution with the machinery it already has: Task → Run → PR → review → merged, then verifies the fix in production.

Mainark does not become a monitoring backend. No CPU/memory graphs, no log explorer, no metric storage. Coolify shows containers, logs and server metrics; every service row in Mainark links to it.

Boundary

The last row is not optional. The coordinator runs on fulreach-workers (/opt/mainark/mainark), the same host as the Fulreach workers. When that host dies, Mainark dies with it and cannot report it. An external monitor (UptimeRobot, Better Stack, healthchecks.io — any) checks Mainark's GET /health and Fulreach's public URL.

Health contract (mainark.health/v1)

Any service can implement it; Mainark knows nothing product-specific. The product computes every status itself — Mainark never interprets a queue depth. That is what lets "0 sends in 12 minutes" be normal at 3 a.m. or when every account has hit its LinkedIn limit: only Fulreach knows its send windows.

GET /internal/health
Authorization: Bearer <MAINARK_HEALTH_TOKEN>

The endpoint is never routed publicly; the token is required even on the private network. It answers 200 whenever the process can answer — a degraded system is reported in the body, not with a 503.

{
  "contract": "mainark.health/v1",
  "service": "fulreach",
  "generatedAt": "2026-10-10T14:03:00Z",
  "version": { "commit": "abc1234", "deployedAt": "2026-10-10T12:01:00Z" },
  "status": "degraded",
  "checks": [
    {
      "name": "workers.heartbeat",
      "component": "fulreach-workers",
      "status": "ok",
      "summary": "4 of 4 workers reported in the last 60 s",
      "observed": { "active": 4, "expected": 4, "lastHeartbeatAgeSeconds": 12 }
    },
    {
      "name": "queue.invite",
      "component": "fulreach-workers",
      "status": "degraded",
      "summary": "182 waiting, oldest 94 s, 7 failed in 24 h",
      "observed": { "waiting": 182, "active": 4, "failed24h": 7, "oldestWaitingSeconds": 94 }
    },
    {
      "name": "sending.invites",
      "component": "fulreach-workers",
      "status": "ok",
      "summary": "31 sent in the last hour",
      "observed": { "sentLastHour": 31, "lastSuccessAgeSeconds": 40 }
    }
  ]
}

Fulreach's first checks: worker heartbeats, one check per queue (waiting, oldest waiting, failed in 24 h), last successful send, and Unipile error rate. Fulreach reads its own queues; Mainark never connects to Valkey.

Incident model

An incident is not a Task. The Task is the code work; the incident is the production problem on a Service. One incident creates or links to at most one Task (never a Goal, ADR-004); many incidents never get one. There is no separate incident work queue: the Task goes through the normal chain, and may sit at the Product root or under a Goal (e.g. "Reliability").

The two lifecycles meet at verification:

• the Task is merged when its fix PR merges — the dependency boundary, so other Tasks waiting on the fix may proceed;
• the Task is done when its outcome is verified, and for an incident Task that verification is the incident's verification (see Verification);
• the incident is resolved on that same verification.

Until ADR-004 phase 4 ships, a Task is still marked done on merge (ADR-004: "done = merged in effect"). The incident does not follow it: it resolves only on verification.

Stored

CREATE TABLE incidents (
  id            TEXT PRIMARY KEY,          -- FUL-INC-1, MAI-INC-7 … one sequence per Product
  service       TEXT NOT NULL,             -- registry id, e.g. fulreach-workers
  fingerprint   TEXT NOT NULL,             -- '<service>:<check name>' or '<service>:deploy'
  source        TEXT NOT NULL,             -- health | deploy | manual
  title         TEXT NOT NULL,
  severity      TEXT NOT NULL,             -- degraded | down
  status        TEXT NOT NULL DEFAULT 'open',  -- open | resolved | dismissed
  issue_id      TEXT REFERENCES issues(id) ON DELETE SET NULL,  -- the linked Task; must have kind = 'task'
  diagnosis_json TEXT,                     -- RenderFailureDiagnosis shape: category, confidence, reason, fixSteps, evidence
  commit_sha    TEXT,                      -- running / failed commit when detected
  detected_at   DATETIME NOT NULL,
  resolved_at   DATETIME,
  created_at    DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
  updated_at    DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE UNIQUE INDEX idx_incidents_open_fingerprint
  ON incidents (fingerprint) WHERE status = 'open';

CREATE TABLE incident_events (
  id          INTEGER PRIMARY KEY AUTOINCREMENT,
  incident_id TEXT NOT NULL REFERENCES incidents(id) ON DELETE CASCADE,
  at          DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
  kind        TEXT NOT NULL,
  detail_json TEXT NOT NULL DEFAULT '{}'
);

Incident ids are Product-prefixed (<PRODUCT>-INC-<n>, per ADR-004), so they stay unique if the Product databases are ever unified and never match a Goal/Task id (<PRODUCT>-<n>). allocateIdentifier currently strips non-alphanumerics from its prefix; it must learn the INC family (sequence key FUL-INC) before incidents ship. The column keeps the physical name issue_id like every other reference into the issues table (ADR-004, storage names); triggers enforce that it points at a Task.

incident_events.kind: detected, severity_changed, diagnosed, task_linked, dispatch_skipped (with the reason), deploy_started, deploy_succeeded, deploy_failed, fix_running (production commit contains the fix), verified, resolved, reopened, dismissed, note. Only state changes are recorded — never one row per poll.

Derived, not stored

Agent progress already lives in agent_runs and issue_events. The incident timeline in the UI merges incident_events with the linked Task's issue_events and agent_runs; nothing is copied. Same rule as mainark ready in dual task graphs and ADR-004's stored vs derived statuses.

Detection rules

Deploy events are pulled from the Coolify API, not pushed by webhook. The Mainark API binds to a private interface only (HTTP API), and a hosted Coolify cannot reach it. Pulling matches the polling stance in ADR-002 and leaves room for a webhook adapter later on the same apply path.

The diagnoser is src/render/failureDiagnosis.ts, made provider-neutral. Its categories and output shape stay.

What Mainark does with an incident

Fails closed like every cost guard: anything not listed here records dispatch_skipped and needs a human.

Further rules:

• No auto-merge for incident PRs, regardless of AUTO_MERGE. Revisit once diagnoses have a track record.
• One dispatch per fingerprint per 24 h. A reopened incident whose previous fix already merged is flagged needs-attention, never re-dispatched — that is the loop where a bad fix deploys and causes the next incident.
• The Task belongs to the Product that owns the service (FUL-* for Fulreach services, dual task graphs), created through getTaskProvider().createIssue, the same path as POST /api/issues, with kind = 'task'. Its document is self-contained: service, check or deploy, summary, observed, diagnosis, failing commit, Coolify link — and, as acceptance criteria, the verification below.
• Other Tasks that need the fix may depend on the incident Task; they are unblocked when it is merged, not when the incident resolves (ADR-004 dependency semantics).
• Slack (existing SLACK_WEBHOOK_URL): one message on open, on needs-attention, and on resolve. Not per poll.

Verification

Incident verification is one instance of ADR-004's Task verification — evidence that the Task's acceptance criteria hold in the target environment — not a separate mechanism. For an incident Task the acceptance criteria are "the fix is running and the failing check has recovered", and the evidence is:

1. the health contract's version.commit contains the merge commit (git merge-base --is-ancestor), recorded as fix_running;
2. the incident's check has then been ok for M consecutive polls.

Without a health URL, step 1 uses the commit of the latest successful Coolify deploy (deployment-state evidence in ADR-004's terms).

When both hold, Mainark records the evidence once and it drives both lifecycles: the incident gets verified then resolved, and the linked Task gets a verified event (evidence kind incident_recovery) that promotes it from merged to done once ADR-004 phase 4 separates the two. If the check fails again after the fix is running, the incident stays open (or reopens), the Task stays merged and is shown as verification failed — the merge is never undone.

An incident with no linked Task (configuration fixed by hand, external cause) resolves on step 2 alone.

Service registry

.mainark/services.json in the Mainark checkout (MAINARK_SERVICES_FILE to override):

[
  { "id": "fulreach-api", "taskGraph": "FUL", "coolifyAppUuid": "…", "healthUrl": "http://…/internal/health" },
  { "id": "fulreach-workers", "taskGraph": "FUL", "coolifyAppUuid": "…" }
]

Checks are attached to a service by checks[].component, so one health URL can report on several Coolify apps. taskGraph names the Product that owns the service; ADR-004 phase 5 moves this registry into a services table keyed by product_id.

Non-goals

• CPU, memory, disk or any time-series metric in Mainark.
• Log search. Incidents link to Coolify logs.
• Monitoring Mainark or the host from Mainark.
• Health-incident diagnosis beyond the product's own summary (v1).
• Moving Fulreach's frontend off Vercel.

Open items

• Confirm the Coolify API endpoints for listing deployments and fetching a deploy log, and whether this setup is Coolify Cloud or self-hosted.
• Decide how Mainark reaches /internal/health (tailnet address vs. Coolify's internal network) — it must not be on a public route.
• Agree Fulreach's thresholds for each check, including send windows.
• Choose the external uptime monitor and point it at Mainark /health.