Unattended Autonomic Agents are meant to wake on a schedule, read their backlog, call tools, and decide what to do next, often without a human in the loop. That is powerful when it works. It is painful when a run looks alive in the database but nothing is actually executing anymore.
We call those stuck runs zombie agents (or zombie runs): rows in agent_runs that remain in an inflight status long after the process that was driving them has stopped sending heartbeats. It could be worse than that because one zombie can not only block other agents from running but also make other agents zombies by killing them but not letting them die due to locking.
This post explains the failure mode, what it costs operators, and how Tealfabric handles it — without turning your tenant into a debugging exercise.
The problem in one sentence
A zombie run is an Autonomic Agent execution that never reached a terminal status even though no worker is still working on it.
Normal failures are fine. When an LLM provider returns a rate limit, when a tool throws, or when an operator cancels a delegated task, the platform records a terminal outcome (failed, cancelled, completed, and so on). The run releases its slot. The next wake can proceed.
Zombies are different: the run stays in statuses like running, tool_loop, or dispatching, so the platform still counts it against max concurrent runs. From the outside it can look like “the agent is busy forever.”
How zombies are born
Zombie runs are almost always an orphaned lifecycle problem, not a model “going rogue” in the sci-fi sense. Common causes:
Deployments and process restarts
Autonomic Agent turns run inside the API worker pool. A deploy, pod eviction, OOM kill, or rolling restart can terminate the Node process after the run row was created and before finish() runs. The database still says tool_loop. The worker that was mid-turn is gone.
Provider or network interruption without a clean catch
Most provider errors are caught and persisted. But if the worker dies abruptly — or an unexpected fault escapes the run wrapper — the run never transitions to failed. The journal stops updating while the status field does not.
Long stalls without heartbeats
A run might block on an external dependency longer than operators expect. Heartbeats are the platform’s signal that something is still alive. If heartbeats stop while status remains inflight, the run is a zombie candidate even if the original cause was benign.
Misread “still running” in the UI
Operators see a green “running” indicator and assume work is progressing. Without a freshness signal, inflight status alone is misleading. That gap between status and liveness is what zombie handling closes.
What rogue agents cost you
We use “rogue” here in the operational sense: runs that occupy capacity without doing useful work. The implications stack quickly.
Concurrency starvation
Each Autonomic Agent has a configured max concurrent runs (often 1 for unattended agents). One zombie holds the slot. Every subsequent wake — cron trigger, mailbox message, process invoke_agent step, or Trace delegation — gets rejected with “agent already has a run in progress.” The agent appears bricked until someone intervenes.
False operational picture
Monitor views, run inspectors, and automation that keys off inflight counts all lie by omission. Dashboards show activity; backlog does not drain; downstream process steps wait on an agent that will never answer.
Wasted token and wake budget
Even when no LLM calls are happening, zombie runs distort history and auditing: runs that never completed skew success rates, confuse postmortems, and make it harder to tell a rate-limit retry from a dead worker.
Human time on manual recovery
Before automated reaping, clearing a zombie meant database surgery or support tickets. That does not scale when you run many agents across tenants.
Zombie handling is therefore a reliability and capacity feature, not a cosmetic status tweak.
How Tealfabric models agent liveness
Autonomic runs move through a small state machine. Statuses such as queued, running, tool_loop, dispatching, and cancel_requested are treated as inflight: they consume concurrency. Terminal statuses — completed, failed, need_human, cancelled, and related outcomes — release the slot.
Heartbeats
During an active turn the run service updates heartbeat_at at meaningful boundaries: turn start, tool progress, dispatch, and similar checkpoints. Heartbeats are cheap insurance: they decouple “what phase are we in?” from “is anyone still driving this?”
If heartbeats go quiet while status stays inflight, the run is stale even if the label still says running.
Stale thresholds (configurable, documented)
Operators can tune how aggressive the platform is:
| Threshold | Role | Typical default |
|---|---|---|
| Zombie stale window | How long without a fresh heartbeat (or old started_at) before a run is eligible to reap | 10 minutes |
| Reaper scan interval | How often the background reaper scans | 1 minute |
| UI stale warning | When the Agents run inspector shows an amber “likely stuck” hint | 2 minutes |
These defaults are documented for operators and can be adjusted in dedicated or self-hosted deployments. Align them with your longest legitimate tool call and your deploy cadence.
The reaper also consults the configured chat.agent_turn timeout from policy: a run that exceeds the turn budget without finishing is treated as stale even if heartbeats were recent. That ties zombie detection to the same upper bound the LLM runtime already respects.
The zombie reaper: technical deep dive
When Autonomic Agents are enabled for a tenant, an Autonomic run reaper service runs in the platform API layer.
What it does
On a schedule (and at startup), the reaper:
- Selects
agent_runsrows in inflight statuses whoseheartbeat_atorstarted_atis older than the stale threshold (or older than the agent-turn timeout). - Atomically updates each candidate to
failedwith a clear, operator-facing error message indicating the worker was likely interrupted. - Appends a
failedevent to the run’s event log with metadata (source: reaper, previous status) for audit.
Reaping is idempotent at the row level: the update includes inflight agents, so a run that finishes normally between selection and update is not clobbered.
Where it runs (not only in the background)
Stale-run cleanup is deliberately multi-trigger so zombies do not block work until the next cron tick:
- API startup — clears survivors from the previous deploy generation.
- Periodic interval — catches mid-day worker deaths.
- Before concurrency checks — when a new
invokeis attempted, the reaper scopes to that tenant and agent first so a fresh wake is not rejected because of a dead predecessor. - When listing runs — operators see accurate state without waiting for the interval.
This “reap before you count” pattern is important: concurrency is a safety property, so the gate that enforces it should not trust stale rows.
Operator force-fail
Automation handles the common case; humans retain an escape hatch. Tenant administrators can force-fail a stuck inflight run from the Agents UI or via POST /api/v1/autonomic-agents/:id/runs/:runId/force-fail. The same guarded update path runs: only inflight rows transition; an event records operator_force_fail with the reason.
The UI labels runs whose heartbeat is older than the stale warning threshold and offers force-fail with copy that explains the likely cause (worker restart, provider error) and mentions automatic purge.
Zombie runs vs rate-limit retries
These problems often show up in the same incident timeline but need different medicine.
| Symptom | Likely cause | Platform response |
|---|---|---|
| Run stuck inflight, no heartbeats | Dead worker / unclean shutdown | Zombie reaper → failed, slot released |
Run fails with provider 429 / rate limit | Transient LLM quota | Deferred retry with exponential backoff; new wake later |
Run failed with explicit error | Normal controlled failure | No reaper; operator reads error_message |
Rate-limit handling ends the current run (releasing concurrency) and schedules a later retry via scheduled_tasks. Zombie handling reclaims rows that never ended at all. Together they cover “the provider said no” and “nobody is home anymore.”
Design principles we kept
Fail closed on capacity, fail open on recovery. It is better to mark a run failed with an honest message than to block an agent indefinitely on a ghost row.
Observability over magic. Reaped runs keep an event trail. Operators can tell automatic purge apart from force-fail.
No silent cross-tenant effects. Reaping respects tenant and agent scope; one tenant’s deploy does not sweep another’s runs.
UI and backend share semantics. The frontend stale indicator uses the same default threshold family as the reaper so warnings appear before the automatic purge, not after operators have already panicked.
What we are not disclosing here
This post describes behavior and architecture. If you operate a self-hosted or dedicated environment, tune the stale and scan-interval thresholds with your SRE team — do not cargo-cult production defaults from a blog post.
Practical guidance for operators
- Treat inflight + stale heartbeat as incident signal, not as “slow AI.”
- After deploys, watch agent run lists for a few minutes; startup reaping should drain pre-crash zombies automatically.
- Prefer force-fail when the UI shows the amber stale warning and you know the worker restarted.
- Align the zombie stale window with your longest safe tool loop; too low creates false positives, too high prolongs starvation.
- Separate rate-limit storms (many fast failures + scheduled retries) from single stuck inflight rows (zombie pattern).
Closing thought
Autonomic Agents are only trustworthy if their lifecycle is trustworthy. A model can be brilliant and still leave a run row behind when the process hosting it disappears. Zombie reaping is the platform admitting that distributed systems fail mid-flight — and choosing to recover capacity instead of pretending the agent is still thinking.
Unattended automation should fail loudly, recover cleanly, and never hold the door shut on the next wake.
Related reading: Autonomic Agents in the Library (configuration and wake model), Monitor execution logs for process-level retries, and the Agents console run inspector for per-run events.