Incidents

How a confirmed state change opens and recovers an incident.

Confirmation, not a single observation

A monitor's consensus state can change on any evaluation, but an incident does not open or close on a single change. Pinggy confirms a state change over a run of consistent evidence first, so one noisy or borderline observation does not open an incident that recovers on the next check.

Thresholds

Two thresholds govern confirmation, each counted as a streak of consecutive matching observations:

Open threshold — how many consecutive DOWN observations confirm an incident. Defaults to 3.
Recovery threshold — how many consecutive UP observations confirm a recovery. Defaults to 2.

An operator may configure these per deployment within a bounded range; the defaults above apply unless a deployment states otherwise.

How a streak moves

A DOWN observation advances the down-streak (up to the open threshold) and resets the up-streak to zero.
An UP observation advances the up-streak (up to the recovery threshold) and resets the down-streak to zero.
A DEGRADED or UNKNOWN observation holds: neither streak changes, and nothing opens or recovers on it. Evidence that cannot conclusively say UP or DOWN does not count toward either direction.
A PAUSED monitor resets both streaks to zero, and an open incident on it closes administratively rather than staying open against a monitor nobody is asking to answer.

What this means for node loss

Losing an assigned node does not, by itself, open an incident. A lost node produces stale or missing observations, which consensus counts toward UNKNOWN rather than DOWN — see States — and an UNKNOWN observation holds the streak rather than advancing it. An incident opens only from a genuine run of DOWN evidence.

Maintenance interaction

A DOWN observation that occurs while an announced maintenance window is in force does not open a new incident: the evidence is still consumed — the streak still advances — but the action taken is suppression rather than opening, so maintenance is recorded as maintenance rather than as an alert. See Maintenance.

Assurance changes

Changing a monitor's assurance level — its desired replicas or quorum — resets both streaks, because the evidence gathered under the old policy is not necessarily comparable to what the new one asks for. It does not, by itself, close an incident that is already open.

This article is part of this repository and is served by the same binary, so what you read here is the documentation for the version you are running.