Clusters

Some services run on one host out of a set, not on every host. A job that must run exactly once, an app that follows a database primary, a quorum of three voters. Judged one host at a time, these look broken. Every host that correctly isn't running the service reads as an outage. The failure that does real damage is also invisible per host. When a singleton runs on two hosts at once, each host looks fine on its own.

A cluster fixes both. You declare which systems form a set and how each shared service should be placed across it. daemonitor then judges those services across the members and raises one alert for the set.

Set up a cluster

Settings, Clusters.

1. Click + New and give the cluster a name.
2. Pick a kind.
   • generic. Placement is the only evidence.
   • patroni. The cluster accepts a leader reported by the patroni plugin. Needed for the leader placement.
3. Set the grace period, grace_seconds. Default 120, range 0 to 3600.
4. Add member systems. You can only add systems you own.
5. Click suggest from what the members report, then add the rules you want. Change a rule's placement with its dropdown afterwards, or disable or remove it.

Until a cluster has rules, every service on its members still alerts per host.

Suggestions

Once a cluster has two or more members, the suggest button reads what the members are running right now and proposes rules. It looks for services that more than one member has reported, and splits compose projects into one candidate per container. Its guesses follow the current shape:

Running on	Suggested placement
1 member	singleton
every member	all
some members	quorum, with min_running set to the current count
no member	singleton, with a note to check the service isn't down

Suggestions are never applied on their own. A service on one of three hosts might be a healthy singleton, or an ordinary service that just died on two hosts. Those look the same from one snapshot. Check how the service decides where to run before you accept a rule.

Rules

A rule names a service and a placement.

Matching a service

Field	Meaning
match_type	The plugin type, such as docker.
match_unique_id	The service's uniqueId. It must be the same on every member, which is why you should set uniqueId explicitly in the agent config.
match_container	Optional. One container inside the row, matched by compose service name or container name.

Use match_container for Docker most of the time. The docker plugin sends one row per compose project, and one project often mixes placements. A project might hold a worker that must run on exactly one host next to a web container that runs on all of them. A rule without match_container speaks for the whole project. A rule with it speaks for that one container.

Placements

Placement	Healthy when	Critical when
singleton	exactly one member runs it	it runs on more than one member, or on none
leader	the current leader runs it, and no one else does	it runs anywhere other than the leader, or not on the leader
all	every member runs it	any member that is reporting doesn't run it
quorum	at least min_running members run it	fewer than min_running do

A few cases produce a warning instead.

• A member whose agent has been silent for 15 minutes counts as unknown, not as "not running". If the service isn't running anywhere and some members are unknown, the verdict is a warning, because the service may be on a host that can't report.
• A member running the service in a degraded state, such as an unhealthy container, still counts as running it. A singleton that is up but unhealthy is a warning.
• quorum warns when the count is at or above min_running but below the full member count.
• all warns when every reporting member runs it but some members are unknown.

A member that is reporting but hasn't mentioned the service for 5 minutes counts as not running it. That is the normal state of a follower.

leader falls back to singleton while no leader is known. Exactly one copy is still the strongest true statement available.

What changes on each host

A rule without match_container hands the whole row to the cluster. That row no longer raises its own per-host alert, and a host that isn't running it shows STANDBY instead of OFFLINE.

A rule with match_container removes only that container from the host's own status. A stopped worker on a follower stops turning the host amber. The other containers in the same project still count, so declaring the worker clustered does not hide a crashed web container next to it.

Alerting

Any report from any member re-judges the cluster, not only reports of the clustered service. When a leader dies it reports nothing at all, so the followers' ordinary reports are the evidence. Rules are swept at most every 30 seconds this way. A report of the clustered service itself always re-judges its rule.

Nothing pages until a bad verdict has held for grace_seconds. That stops a normal failover, where the service is briefly on zero or two hosts, from paging. The dashboard card shows the true state at once.

The cluster raises one alert per rule. The alert text names the cluster, the rule, what is wrong and each member's state, for example:

SINGLETON worker: running on 2 members at once (node-a, node-b), should be exactly one.
Members: node-a=running, node-b=running, node-c=absent.

The alert is always filed against the cluster's first member, the one added earliest, wherever the fault is. That keeps one open alert per rule through failovers. The text says which host is involved. When the rule turns healthy again, the alert resolves.

Each verdict is also recorded, one sample per rule per minute plus every change of level. The cluster card charts the placed count over time.

Leader from Patroni

The leader placement needs to know which member holds the leader. The patroni plugin supplies it. Run it on every member of a patroni-kind cluster:

"patroni": {
  "name": "patroni",
  "uniqueId": "patroni",
  "refreshInterval": 30000,
  "url": "http://localhost:8008",
  "lagWarnBytes": 33554432,
  "lagCritBytes": 268435456
}

The plugin reads the node's local Patroni REST API. /patroni gives this node's own role, and /cluster gives every member's role, state and lag. These are unauthenticated GETs by default.

A node that reports itself as leader identifies the leader exactly. If no member reports that, the server matches a follower's view of the leader's name against the names the members report and then against system names. Patroni calls the leader primary in version 4 and master or leader in older versions. All of these, and standby_leader, count.

If no member confirms a leader for 180 seconds, the recorded leader stops counting and leader falls back to singleton. A report that finds no leader at all clears it. An unknown leader is better than a confident wrong one.

Patroni node health

The patroni plugin also reports on its own node, as a normal per-host service. This is separate from the cluster verdict.

Status	When
error	the node's Patroni API is unreachable, its state isn't running or streaming, the cluster has no leader, or replication lag has stayed at or above lagCritBytes for 5 minutes
warning	Patroni is paused, lag has stayed at or above lagWarnBytes for 5 minutes, another member isn't running, or a restart is pending

Lag only counts once it has held for 5 minutes, so a burst of writes doesn't page.

Limits

all can't see an app that decides internally whether to work. Some apps run on every member and check for themselves whether they are on the primary, doing nothing when they aren't. To daemonitor, every member shows a healthy container, whether one copy is working or all of them are. The container state says nothing about the app's own decision. Catching a split brain there needs the app to report its own verdict.

Before you write a rule, find out how the service actually decides where to run. An app that self-gates belongs under all, not leader. Declaring it leader raises a correct-looking but wrong critical alert on every follower.

The cluster card is computed in the browser. The external API's fleet endpoint marks members that aren't holding a clustered service as standby, but it does not include the cluster card itself.