Draining and retiring nodes
Draining or retiring several nodes in one operation without losing quorum.
Draining or retiring several nodes
A tenant Owner can take several of the tenant's own monitoring nodes out of service in one durable fleet operation, from the /nodes page: tick the nodes in the Select column, choose a mode and a drain deadline, and Preview. Nothing changes until the confirmation on the next screen.
There are two modes:
- Drain Only — the default. Work is moved off the nodes; they end draining and drained, and stay enrolled.
- Drain and Retire — chosen explicitly. Each node is drained and then retired: its credential is revoked, permanently. A retired node keeps its history but cannot be restored; it would have to be enrolled again.
At most 25 nodes can be in one operation, and a tenant has at most one decommission running at a time.
What the preview checks
The preview compares what each of the tenant's monitors has now with what it would have once the selected nodes - and any node already draining - have left and placement has refilled what it can.
- A removal that would leave a monitor below quorum, or make one already below quorum worse, or remove a monitor's last ICMP-capable replica, is refused. There is no override: add capacity or select fewer nodes.
- A removal that reduces coverage - a monitor below its desired replicas, fewer regions or providers, more monitors one site, region or provider failure from losing quorum, a region's last eligible node - must be acknowledged: the confirmation asks you to type each consequence's code. The acknowledgements are recorded on the operation.
- Anything the removal does not change, and selected nodes that are already unreachable, are shown for information.
The check is made again when you confirm, so the operation is admitted against the fleet as it is then, not as it was when you previewed.
How an operation progresses
A node's work moves only to an eligible replacement, at the usual bounded rate, and is kept where it is when there is none: a drain never lowers the number of replicas serving a monitor. A node counts as drained once it holds nothing and every monitor moved off it has a replacement that has reported a result since the move.
In Drain and Retire mode a node is retired once it is drained, or once it is unreachable - no heartbeat for the offline threshold (5 minutes by default) - because an unreachable node serves nothing. Just before retiring a node, Pinggy checks that this one node holds no serving replica; if it unexpectedly does, that node is marked as refused and waits, and the other nodes carry on.
Each operation has a drain deadline: 2 hours unless you choose otherwise, from 15 minutes to 24 hours. A node not finished by then is reported stalled, with the reason: no replacement capacity, or a replacement not yet observed. You can extend the deadline once, cancel, or abandon.
- Cancel returns every node not yet retired to the state it had before the operation (enabled, disabled or draining). Retired nodes stay retired.
- Abandon stops the operation where it stands and changes no node.
Operations survive restarts of the control plane: they resume from where they were.
History
Every member of the tenant can read the operation history on /nodes/operations: who started each operation and when, its mode, each node's state before and its outcome, the preview's findings, the acknowledgements, and every step. Fleet operation history is never expired. Retiring a node deletes none of its check history, observations or evidence.
Single-node actions during an operation
While a node belongs to an operation, editing it, disabling it, cancelling its drain and enabling it are refused, naming the operation; draining it again is accepted and changes nothing. Revoke is never blocked: revoking a node during an operation is recorded on the operation, which carries on with the others. The Revoke button on /nodes shows what revoking that node now would do to quorum, or says the impact could not be computed - it never blocks.
Cancel-drain now only returns a draining node to enabled; a disabled node is re-enabled with Enable.
This article is part of this repository and is served by the same binary, so what you read here is the documentation for the version you are running.