When the link drops¶

Every operator eventually asks a version of the same question: "the Hub can't reach a Scout (or the Scout can't reach the Hub) — what actually happens?" This page is the one place that answers it, for every direction the link can fail, and lists the alerts that fire when it does.

The short version: nothing here needs a durable message queue to be safe. Scans keep happening locally, in-flight commands run to completion, renewals are crash-safe on the Scout's disk, and the two obligations that genuinely can be lost — a command dispatched to an offline Scout, and a report generated while the Hub is unreachable — are both now visible even though neither retries itself.

Hub down¶

The Scout does not need the Hub for the work it does moment to moment:

Scanning continues on the Scout's own ticker. The scan cadence lives on the Scout (persisted in state.json) independently of the connection — it keeps scanning its configured targets and inventorying local certificates whether or not anything is listening on the other end.
Reports and heartbeats queue only in memory, briefly. With the NATS connection down, a report the Scout tries to publish simply fails to send; the client library holds a small in-memory buffer, not a durable local queue. A short Hub outage is invisible once the Scout reconnects and resumes reporting on schedule. A long outage, or the Scout process itself restarting mid-outage, can lose whatever was sitting in that buffer — it is not replayed from disk. Nothing is lost from what the Scout has already done (scans, renewals); what can be lost is the Hub's timely knowledge of it, which repairs itself on the Scout's next successful report.
Which reconnect behavior applies depends on timing. If the Scout was already connected when the Hub went away, its NATS client reconnects forever on its own — see the two-tier reconnect in Transport → Resilience. If instead the Scout's own process starts or restarts while the Hub is still down (a reboot, a cert-rotation reconnect landing in the gap), that first dial fails fast and the process exits; systemd restarts it roughly every 30 seconds until the Hub is reachable again. Either way, no manual intervention is needed once the Hub comes back — the Scout finds its own way home.
A command sent to an unreachable Scout is not queued for later. Command delivery is at-most-once, fire-and-forget core NATS — there is no redelivery. If a Renew Now, a revoke, or any other action was dispatched while the Scout could not be reached, it simply never arrives; the fix is to re-issue the same action once the Scout is confirmed back (see the alerts that now fire below for how you find out this happened at all).
A per-Scout scan-interval change self-converges — "reconcile on reconnect." If you change a Scout's scan interval on the Scouts page while it happens to be offline, the Scout keeps running its last-known cadence for the whole outage. It does not need a dispatched command to catch up: every report a Scout sends states the cadence it is actually running, and the Hub compares that against the setting on file. The first report after reconnecting carries the stale value, the Hub notices the mismatch, and it dispatches the correction then — no re-click required. The same mechanism converges a Scout that missed the change entirely.

Scout down¶

From the Hub's side, a Scout going quiet looks the same whether the Scout process died, its host is off, or the network between them is broken — the Hub only ever sees an absence of reports and heartbeats. Three things follow from that absence, on three different clocks:

Within about the same short window (default 2 hours), the Scout reads as stale on the Fleet health area and on the Scouts page — "Last seen" drifts from moments-ago into minutes, then hours. See Scout fleet health.
A one-time push alert fires the first time the Scout crosses that window. This is new, decoupled behavior: the Hub raises a Warning on the Activity feed naming the Scout, once, the moment it has gone unreported for longer than CYPHERS_SCOUT_DARK_ALERT_HOURS (default 2 hours). It will not repeat every sweep while the Scout stays down, and it clears itself the instant the Scout reports again — a later outage raises a fresh alert. It does not disable the Scout, does not touch its fleet management state, and does not revoke anything. It exists purely so an operator finds out promptly, without configuring anything. Full detail: Scout-went-dark alerts.
After 30 days with no active Scout observing them, an endpoint's own rows read "Stale" on Manage — a separate, longer clock about the endpoint's served proof rather than the Scout's connectivity. See Monitoring → Staleness: the 30-day window.

The destructive response — auto-disabling the Scout and revoking its certificates — is a fourth, entirely separate thing, gated behind the opt-in AGENT_STALENESS_HOURS setting. It is off unless you turn it on, it answers to its own threshold, and turning it on does not change when the warning above fires. Details and the reasoning for keeping the two apart: Scout-went-dark alerts.

Mid-command¶

The case operators worry about most: the link drops while the Scout is in the middle of doing something — issuing, delivering, or rotating a certificate.

A running command is not interrupted by a dropped connection. Execution happens on its own worker goroutine, decoupled from the NATS client; a disconnect mid-command does not cancel or re-run it. Only the acknowledgement can be delayed — it is sent once the connection is back (or, if the process itself restarted, replayed from what was recorded on disk — see below).
A renewal is crash-safe across a Scout restart, not just a network blip. Every deploy-relevant step of a renewal (which domain, which renewal, which command, what state it reached) is written to an on-disk journal as it happens — the deployment-lifecycle record for that domain. If the Scout process is killed mid-renewal (host reboot, OOM, a bad upgrade) and restarted, it does not guess what state the filesystem is in: before it dials NATS again, it reconciles every journal entry it finds. An entry whose outcome can be determined is completed or replayed (an ACME HTTP-01 issuance that already succeeded is not re-requested from the CA — the archived result is replayed instead); an entry whose outcome is genuinely undecided quarantines that one domain — cert-mutating commands for it are refused, with the reason logged, until the stale record is cleared — rather than risk a double issuance or a half-applied file swap. Every other domain on the Scout is unaffected.
The certificate file swap itself is transactional. Installing a renewed certificate backs up the previous generation first (a .cyphers-rollback.json marker alongside the files records what to restore), and the swap is ordered so a process killed mid-write leaves enough evidence for the next run to tell whether to finish the commit or roll back — never a half-written pair of files silently serving.
A failed deploy hook is reported as a rollback, not silently swallowed. If installing the new certificate goes through but the local reload/deploy hook fails, the Scout restores the previous certificate and reports that as a rollback result rather than leaving a broken install in place — which is why a rollback is treated as a renewal failure worth an Error alert (see the alerts that now fire).

The alerts that now fire¶

Three activity-feed alerts are new. All three used to be silent — the underlying condition existed before, but nothing told the operator.

When	Severity	What it says	Where else it shows
A renewal's timeout watchdog closes it out as timed out	Error	"Renewal timed out for `<hostname>` - . Open the endpoint to retry."	Counts toward Pipeline health's recent-failures total, and the health card's "Open Activity" link now actually points at this entry — previously the count and the referral could disagree, because a renewal that started outside the count's lookback window but only just timed out was invisible to both.
A renewal's timeout watchdog instead parks it awaiting verification	Warning	"Renewal for `<hostname>` is awaiting verification - the certificate installed but has not been confirmed live. Open the endpoint to check."	Same Renewal category as above.
A dispatched command expires undelivered (the Scout never acknowledged it inside its 1-hour delivery window)	Warning	"Command `<verb>` to Scout `<name>` expired undelivered - the Scout was offline past the delivery window, and it will not be retried automatically."	Counts toward Pipeline health's recent-failures total (an expired command counts the same as a failed one).
A Scout crosses the dark-alert threshold (default 2h of silence)	Warning	"Scout `<name>` has gone dark - no report recently. Commands sent to it are not delivered."	A one-time push notification only — it does not itself add to any health-area count. The pre-existing, passive Fleet health staleness count (and the Scouts page "Last seen" column) already tracks the same underlying silence on its own clock; see Scout-went-dark alerts.

None of the three change what happens mechanically — a timed-out renewal still needs the same retry it always did, an expired command still needs re-issuing, and a dark Scout still needs reviving. What changed is that all three are now loud instead of silent: they land on the Activity page, and the first two count into the health picture Pipeline already reported.