# How status works

The agent only reports facts. It never decides whether something is healthy. The server reads each report, gives it a status of `online`, `warning` or `error` with a one-line reason, stores it, and decides whether to raise an alert. This page explains those rules so you can predict when you'll be paged.

## Reports, services and systems

Every plugin report arrives at `PUT /api/clientstate/update` with three headers: the system key, a type such as `docker`, and the plugin's `uniqueId`. The server keeps one row per system and `uniqueId`. That row is a service in the dashboard. Each new report replaces the row and adds a sample to its history.

Some plugins send several reports per cycle. `docker` sends one per compose project, `web` one per endpoint, `lxc` one per host that the server splits into one service per container.

## From report to status

Each type has its own rules. The main ones:

| Type | `error` when | `warning` when |
|---|---|---|
| `os` | | CPU above 90%, or memory above 90% used |
| `pm2` | the process status is `errored` | the process is in any other non-`online` state |
| `docker`, `lxc` | a container is `unhealthy`, or stopped with a non-zero exit code | a container is stopped with exit code 0 |
| `web` | the site is unreachable, the check failed, a forbidden string matched, or an expected string is missing | the origin check failed while the public URL worked, or the response took longer than 5000 ms |
| `wordpress` | any posture finding, such as a new administrator or a crontab for the web user | a collector failed, so the posture is unknown |
| `logwatch` | a finding the plugin marked as an error | a finding the plugin marked as a warning |
| `patroni` | the node's API is unreachable, its state isn't `running` or `streaming`, the cluster has no leader, or lag has stayed over `lagCritBytes` for 5 minutes | Patroni is paused, lag has stayed over `lagWarnBytes` for 5 minutes, another member isn't running, or a restart is pending. See [Clusters](./clusters.md#patroni-node-health) |

The reason names the specific thing, for example `api exited (code 137), 4 restarts` rather than `1/13 containers unhealthy`. It shows on the service in the dashboard and in the alert text.

The dashboard maps these to its own levels. `online` is OK, `warning` is WARN and `error` is CRIT. Two more levels come from the dashboard, not from the server.

### Offline

A service that stops reporting goes OFFLINE in the dashboard after 5 minutes. `wordpress` and Cloudflare services get 15 minutes, `adsense` and `ga4` get 30, because they poll slowly.

Offline is a display state only. The server raises no alert when an agent goes quiet. A host that loses power stops reporting, and nothing pages. If you need to know when a host disappears, watch it from a second machine with a `web` or `netscan` check. A [cluster](./clusters.md) rule with placement `all` also warns once a member has been silent for 15 minutes.

### Standby

A host in a cluster that is correctly not running a clustered service shows STANDBY, not OFFLINE. The cluster card carries the real verdict.

## When an alert is raised

A bad status turns the dashboard amber or red immediately. Raising an alert follows different rules depending on the kind of problem.

### Single alerts wait for a second report

Most types, including `os`, `pm2`, `docker`, `patroni` and the others not listed below, raise one alert per service. The server holds that alert until the problem shows up in two consecutive reports. A container that goes unhealthy for 40 seconds and heals itself never pages. A real outage pages one interval later, which for a 60-second plugin costs a minute.

### Findings alert at once, one per finding

`web`, `wordpress` and `logwatch` report security findings. One WordPress site can have a rogue administrator and a trashed login page at the same time, and each matters on its own. These types raise one alert per finding, with no second-report wait. Each finding has a stable key, such as the site plus the matched strings, so a new forbidden string on the same site alerts again even while the first alert is open.

### Network devices

`netscan`, `snmp` and `tplink` reports that name the same IP address merge into one device service. The device takes the worst status across the plugins that watch it, and it alerts on the first bad report.

### Clusters

A service declared in a cluster is judged across all members instead of per host, and waits for the cluster's grace period. See [Clusters](./clusters.md).

### LXC containers

LXC containers show their status in the dashboard. They do not raise alerts.

## One open alert at a time

An alert stays open until the condition clears. While it is open, the same problem on the same service raises nothing new. The server checks this on every report and enforces it with a unique index, so two reports racing each other can't open two alerts.

Reading an alert does not close it. Opening it in the inbox to go and look at the system marks it read and nothing else. Only recovery re-arms alerting, so investigating a fault never causes it to page again.

## Auto-resolve on recovery

When a service reports `online` again, the server closes every open alert for that service. The alert stays in your history with the time it resolved. It just stops claiming to be current.

Findings close a little differently. On each report, the server closes any finding from that service that the report no longer contains. A forbidden string that disappears from the page closes its finding on the next check.

## Re-alert cooldown

If an alert resolves and the same problem returns within 30 minutes, the server records the new event but does not send a push notification or a Slack message. The recurrence shows in the inbox and the event history. This stops a value hovering around a threshold from paging every few minutes. After 30 minutes, a recurrence pages normally.

## Where to see what happened

Each alert is an event that belongs to its system. The system page lists unresolved events as active issues and keeps a paged history of the rest. Every event has its own page at `/systems/<id>/events/<event id>`, which names the finding, shows any before and after detail, and lists recurrences. Slack messages link straight to it.

## History retention

Every report is stored as a history sample. A daily job at 02:00 UTC thins it out.

- The last 24 hours keep full resolution, up to 2000 samples per service.
- Older days keep 24 samples per service per day, roughly one an hour.
- Anything older than 30 days is deleted.

Charts for 7 and 30 days therefore show hourly points. `adsense` and `ga4` are not sampled this way, because each report already carries its own 30-day series.

Separately, an hourly job deletes services that have stopped appearing while their agent keeps reporting. If the system has checked in within 15 minutes and a service on it has been silent for 2 hours, the service is removed. Removed containers, retired web checks and stopped pm2 processes leave the fleet on their own. If the whole system goes quiet, nothing is removed, so you can still see what the machine was running when it went down.