Monitoring the estate

Day to day, "is the estate healthy?" has a reading order: the Manage summary strip for the estate, the health indicator in the header for the Hub itself, the Activity page for what happened and why, and the Scouts page for the fleet doing the observing. This page walks that order, then covers the two machine endpoints an external monitoring system can poll.

First read: the Manage summary strip

The top of Manage is the Estate summary — one card per state, each showing a live count:

Card	What the count means
Healthy	Serving, verified, nothing due
Partial	Only one vantage has current evidence (one-sided validation)
Attention	A weakness in what the endpoint serves (findings, expiry)
Not serving	The certificate on disk is not the one being served — the remedy is on the host, not in the certificate
Renewing	A renewal is in flight
Failed	A renewal or deployment failed
Action needed	Blocked on an operator decision (e.g. no profile)
Untracked	Served proof exists, not under management
Covered	An unmanaged name on a certificate the Hub already renews
Unproven	Known name, never seen serving
Stale	No active Scout has observed it in 30 days

Every count is derived from the same rows the table renders, so a nonzero card always has matching rows: click a card to filter the list to that state, click it again to clear the filter. A morning glance is therefore one question: are all the counts you expect to be zero actually zero? Anything in Failed, Action needed, or Not serving is yours to act on; what each state means and which action it offers is covered in Endpoints, proof, and management.

The Hub's own health: the header indicator

The button at the top of every page reads the Hub's one health calculation — Healthy, Degraded, Action required, or Unavailable, with a count of operations in flight appended when there are any (e.g. "Healthy · 2 active"). Clicking it opens the Events panel, which breaks health into five areas, each with a state, a reason, and — when something is wrong — a suggested action:

Area	Covers
Core	Hub API, database, migrations, CA storage, crypto
Fleet	NATS and Scout fleet freshness
Coverage	Served-proof freshness, internal/external coverage
Pipeline	Command, renewal, install, rollback, verification pipeline
Backup	Backup schedule, last success/failure, age

The panel updates live — the Hub pushes a signal whenever something changes, so you are never reading a stale snapshot.

The Activity page

Activity in the sidebar ("Recent Hub operations, renewals, scans, verification and system events.") lists the most recent events — message, the hostname it concerns (or its category), and when. Events carry a severity (info, success, warning, error) and fall into six categories: command, scan, renewal, lifecycle, verification, and system.

The entries worth a daily scan:

• Renewal lifecycle events — orders started, deployments verified, failures. A failed renewal is loud here, not just a row state, and so is one the timeout watchdog closes out as timed out or parks awaiting verification — both raise their own activity event the moment the watchdog acts, not only when the operator happens to look at the row. See When the link drops.
• Refusals, with remedies. When a renewal is refused by pre-flight — the Scout is dark, the ACME name does not resolve, another renewal is already in flight on that Scout — the refusal names its finding codes and carries the full findings report, and each finding pairs the problem with a remedy (e.g. "Re-enroll or restart the scout"). The same report is what you see when a Renew Now click is refused in the UI.
• An expired, never-delivered command also raises a warning naming the Scout and the command — see When the link drops.
• Scout-went-dark alerts — see below.

Scout-went-dark alerts

A Scout that stops reporting is flagged automatically — this needs no configuration to be on. Once a Scout has gone unreported for longer than CYPHERS_SCOUT_DARK_ALERT_HOURS (default 2 hours, matching the Fleet staleness line below), the Hub raises a warning on the activity feed: "Scout web-01 has gone dark - no report recently. Commands sent to it are not delivered." That last clause is the point: command delivery is at-most-once, so an offline Scout silently misses every command sent while it is down (see Scout commands).

The alert fires once per dark episode — it does not repeat on every sweep while the Scout stays down — and clears itself the moment the Scout reports again, so a later outage raises a fresh alert. It never disables the Scout or touches its certificates. It is a notification only.

The destructive auto-disable is a separate, opt-in setting

AGENT_STALENESS_HOURS=<n> turns on a second, independent sweep that auto-disables a Scout silent for n hours and revokes its certificates. It is off unless you set it, and setting it does not change when the warning above fires — the two no longer share a threshold or a code path; the disable sweep used to raise the "gone dark" warning itself and no longer does, since that warning is now unconditional. Choose n well above your longest expected maintenance window before enabling auto-disable, since it is destructive and not automatically reversed. Checks are skipped for the first n hours after a Hub restart, so a rebooted Hub does not mass-disable a fleet that was healthy all along.

Vantages: two views of every endpoint

The Hub sees an endpoint from up to two positions: internal — the Scout on or near the host, scanning what it actually serves — and external — a probe from the Hub's own network position. Manage compares them, and two readings are routine:

• "Waiting on re-check" right after a renewal is normal. A freshly installed certificate is newer than the last scan, so nothing current has looked at the endpoint yet; the Hub deliberately issues no verdict rather than accusing the endpoint of misserving. The label clears on the next observation — post-install verification re-probes the renewed names itself, and the regular Scout cycle covers the rest.
• A missing vantage is a coverage gap, not a mismatch. "No external observation" means nothing looked, not that something disagreed. A mismatch verdict is reserved for two current observations that genuinely differ — see coverage gaps.

Staleness: the 30-day window

An endpoint's served proof is only treated as current while an active Scout serves or has scanned it within the last 30 days. Past that, its rows read Stale on Manage — the endpoint is still known to have served once, but nothing is watching it now. Hub-side probes do not reset the clock; staleness specifically means "no agent is watching this."

When Stale counts appear, the fix is on the fleet side, not the endpoint: go to Scouts, find the Scout that used to cover those endpoints, and revive it (restart the service on its host) or replace it (enroll a new Scout that covers the same targets — see Scout install).

Scout fleet health

The Scouts page opens with two cards — Total and Active — and the Enrolled Scouts table: Scout, Status, Endpoints, and Last seen for each. A connected Scout heartbeats every 30 s (see Transport defaults), so Last seen on a healthy Scout reads as moments ago; a value drifting into minutes or hours means the Scout is disconnected or its host is down, and every endpoint it covers is aging toward the 30-day staleness window.

Each Scout can also expose its own liveness/readiness endpoints on its host (opt-in via --health-port); its /readyz reflects whether it is enrolled and currently connected to the Hub's NATS. Details: Transport & health.

Machine endpoints for external monitoring

An external monitoring system should poll the Hub's REST port (default 8443; no authentication, nothing sensitive in the responses; plain HTTP unless your deployment terminates TLS on that port):

• GET http://hub.example.internal:8443/health — liveness. Returns 200 with the body ok whenever the process is up and serving.
• GET http://hub.example.internal:8443/readyz — readiness. Returns a JSON report and 200 when every check passes, 503 when any fails, with "overall": "ok" | "degraded" and a per-check breakdown:

Check	Passes when
database	a test query against PostgreSQL succeeds
migrations	the applied migration ledger exactly matches the binary — filename and checksum, not a count
ca_storage	the CA directory exists and is a directory
crypto_posture	the crypto backend initialised in the configured mode

/readyz is the right upgrade/deploy gate: a binary rolled out against a database it has not migrated (or rolled back past an applied migration) reads 503 rather than serving wrong answers.

What /readyz does not cover

The Hub's /readyz does not check NATS or Scout connectivity — a Hub can be fully ready while the fleet is dark. Fleet reachability surfaces in the header health widget (the Fleet area) and on the Scouts page, and each Scout's own /readyz reflects its NATS connection from its side.

The enrollment port (default 7443, TLS) also answers GET /health with ok, if you want to confirm the enrollment listener specifically is up.