Scout — commands & remediation

When the Hub needs Scout to act — issue, deliver, revoke, reload — it dispatches a command on cyphers.hub.{id}.command. Scout strict-decodes the envelope, dispatches by verb on its own goroutine, and acks on cyphers.scout.{id}.ack: first received, then a terminal completed / failed carrying a typed CommandResult. Each ack is published with up to 3 attempts, 50 ms apart (see Defaults at a glance).

The verbs

Verb	What it does	End-to-end status
scan_now	Full scan cycle; returns a report	Live
scan_target	Scan one Hub-named host:port	Live
acme_http01	Scout runs the entire ACME HTTP-01 flow locally (lego)	Live
acme_dns01_direct	Gen key+CSR → Hub (Hub publishes DNS TXT via its provider)	Live (Hub needs a DNS provider configured)
acme_dns01_manual	Gen key+CSR → Hub returns challenges; operator adds TXT; Hub watchdog finishes	Live
acme_dns01_delegated	Gen key+CSR → Hub (Hub publishes DNS TXT via its provider — same mechanism as acme_dns01_direct)	Live (Hub needs a DNS provider configured)
internal_ca	Gen key+CSR → Hub's built-in CA signs directly (also break-glass)	Live
deliver_cert	Install a Hub-issued cert — the DNS-01 second leg	Live
adopt_key	One-time copy of an existing endpoint-served private key into managed storage so a future renewal can reuse it — offered from Manage when Scout reports an adoptable key	Live
revoke_cert	ACME-revoke and/or destroy the key (destroy_key is a payload field)	Live
abandon_renewal	Operator cancels an in-flight renewal leg — rolls the matching active deployment lifecycle back to its pre-rotation state, reloads, and clears any pending DNS-01 record	Live
renewal_preflight	Read-only dry-run of deploy resolution for a renewal candidate — no issuance, no delivery, no mutation	Live

acme_dns01_delegated needs a DNS provider configured

There is no delegation service — the verb is a legacy name from the removed Master/Satellite split and now runs the same Hub-side code as acme_dns01_direct, publishing the TXT record through whichever DNS provider is configured on the Hub (Route 53, Cloudflare or Google Cloud DNS; only Cloudflare can also be set from the environment). If the Hub has no DNS provider configured, the verb fails with a clear error naming the fix (configure a DNS provider, or switch the profile to manual DNS-01), rather than silently degrading.

destroy_key is not a verb

The old standalone destroy_key command was superseded by revoke_cert{destroy_key:true} and is absent from the dispatch switch — a Hub that sent it would get "unknown command."

ACME HTTP-01 is special

For acme_http01, Scout drives the whole flow itself: it loads/creates a persistent ACME account, listens on port 80 to answer the challenge, obtains the cert (bounded by a 3-minute timeout since lego's obtain can't be cancelled), and installs it. It even probes the currently-served cert before running the local pre hook (which typically stops the web server), so it can find and re-point the right on-disk symlinks afterward. There's a one-shot ARI retry for the common "issued under a different ACME account" mismatch. The private key is generated locally and never leaves the host.

The other issuance verbs

acme_dns01_* and internal_ca share one handler that generates the key+CSR locally and submits the CSR to the Hub over NATS request/reply (cyphers.scout.{id}.csr, 60 s timeout). The Hub replies with the issued cert, a dns_challenge_pending (for manual DNS-01 — Scout keeps the key and writes a stall-detection record), or an error.

⚠️ That 60-second window bounds the whole Hub-side order. On the automatic DNS-01 lane the Hub's propagation wait, the CA's validation poll, finalize and the certificate download all have to complete inside it — which is why the propagation budget is capped at 35 seconds rather than made generous.

When the reply carries a certificate, the handler installs the bundle atomically: fullchain.pem, cert.pem, chain.pem, ca.pem alongside the privkey.pem it already wrote.

⚠️ fullchain.pem was missing from this set before 2026-08-22, and it is the file web servers are pointed at (our own installer writes an nginx ssl_certificate line to it). A synchronous DNS-01 renewal therefore rotated privkey.pem while leaving the previous generation's certificate in fullchain.pem, and the server refused the pair with key values mismatch. The Scout's own post-install validation read cert.pem, privkey.pem and ca.pem — exactly the files that were written — so it passed, and nothing caught it until an operator ran nginx -t. On a Scout older than that, expect this on the automatic DNS-01 lane; the manual lane installs through deliver_cert, which always published fullchain.pem.

Where a key-reuse renewal lands

When a renewal profile has Reuse private key across renewals enabled, the Hub-driven lanes work entirely in the Scout's managed delivery directory, {data-dir}/certs/<domain>/ — the reused private key is loaded from there and the renewed certificate is installed there. Any external certificate paths the endpoint serves from (say a web-server-configured path outside the managed dir) are re-pointed as symlinks to the managed files during delivery, so the web server follows the renewal without a config change.

The key gets into the managed dir via the one-time adopt_key copy (see the verb table above): it copies the endpoint's served private key to {data-dir}/certs/<domain>/privkey.pem — the only path reuse loads from. The key never leaves the endpoint. Operator walkthrough: How-to → Adopt a private key.

Renewal pre-flight

Every renewal — a dashboard "Renew" click or an autopilot dispatch — runs a pre-flight before any ACME order is created, so the Hub can refuse or warn on a renewal it already knows can't succeed or can't take effect, instead of finding out after the order exists.

What Scout checks. renewal_preflight is read-only: it never issues, delivers, or touches any identity material.

• Served-cert probe — dials the candidate primary_domain:port and fingerprints the leaf that's actually being served right now.
• Cert-dirs fingerprint search — walks the same merged cert-dir surface deliver_cert and scanning use (configured --cert-dir set plus the dynamic cert-dirs file) looking for a file matching the served fingerprint.
• Deploy-hook presence — stats <data-dir>/hooks/deploy; does not run it.

The result rides the standard terminal ack on cyphers.scout.{id}.ack as a CommandResult with "type":"preflight" — no new subject, same transport as every other verb.

Hub-side grades. The Hub also runs Hub-side checks (ACME client resolvable, a live scout bound to the target, DNS-01 provider configured) before ever dispatching the scout round-trip, then grades everything together into one of three outcomes:

Grade	Conditions	Operator renew	Autopilot
Fatal	ACME client unresolvable (no renewal profile and no ACME_EMAIL); no live scout bound (no active agent, or last_seen older than CYPHERS_PREFLIGHT_SCOUT_FRESH_SECS); an internal-CA (builtin_ca) profile on a name the Hub classifies as public (builtin_ca_not_internal — the same refusal the CA would give at signing time, raised before anything reaches the Scout)	Refused — no order created; reasons + exact remedies shown	Skip + alert
Cannot take effect	Served cert found in no configured cert-dir and no deploy hook — delivery provably cannot reach the service	Warned with remedies + an explicit "Proceed anyway" override	Skip + alert
Informational	No deploy hook but cert-dirs are armed (file replacement will work; reload is the operator's business); manual DNS-01 (no DNS provider configured); the scout doesn't answer pre-flight while otherwise heartbeat-fresh (older binary — the verb didn't exist yet)	Proceeds, notice shown	Proceeds

Deploy-hook absence never blocks by itself. Whether to run a reload hook at all is the operator's choice (see Safety mechanisms below). A missing hook only escalates to Cannot take effect when it's paired with "no cert-dir has the served file either" — i.e. delivery genuinely has nowhere to land. If the file would land somewhere, a missing hook is Informational at worst.

Config.

Env var	Default	Meaning
CYPHERS_PREFLIGHT_SCOUT_FRESH_SECS	600 (10 min)	How old an agent's last_seen may be before pre-flight treats the scout as dark (Fatal scout_dark)
CYPHERS_RENEWAL_VERIFY_TIMEOUT_SECS	1800 (30 min)	How long an installed renewal may sit unconfirmed before the lifecycle watchdog parks it in action_needed instead of leaving it silently in-flight forever

action_needed renewal state. A renewal enters action_needed — a distinct status, not "renewing" and not the terminal failed/timed_out — on any of three triggers:

1. A reload-ineffective deliver_cert ack (cert valid and installed, but the post-reload served-probe still shows the previous cert).
2. A failed deploy hook reported on the install ack.
3. Verification timeout — an installed renewal with no verified_at after CYPHERS_RENEWAL_VERIFY_TIMEOUT_SECS.

action_needed auto-closes with no reissue: passive closure (a routine scan observing the issued serial actually being served) verifies the renewal from action_needed exactly as it would from installed — fixing the environment (adding the cert-dir, repairing the hook, reloading manually) self-heals the renewal without the Hub ever creating a new ACME order. An operator may also cancel it as with any other in-flight renewal.

Safety mechanisms (the important part)

Scout treats the Hub as semi-trusted and defends the host filesystem:

• Backup before key rotation, rollback on failure. The existing cert+key are backed up before the key is overwritten, so a validation or submit failure restores a matched pair rather than orphaning the host.
• 5-point post-install validation — cert parses, key↔cert public keys match, chain present, SAN covers the hostname, and the local deploy hook succeeds. Any failure triggers a rollback (and re-runs the local deploy hook).
• Cert↔key pairing guards — checked before writing on deliver_cert and before destroying a key on revoke_cert, so a wrong (but in-bounds) key_path can't irreversibly destroy the wrong key.
• deliver_cert SAN + post-reload served re-probe — the delivered cert must cover every requested domain (pre-write, rejected before any write); after the reload Scout dials the local endpoint (serve_port, default 443) and compares the served leaf to the delivered cert. Three outcomes, and the Hub treats them differently:
  • Match → success.
  • Still serving the previous cert → reload_ineffective. The new cert is valid and installed; only the reload didn't take, so the good cert is not reverted. Rides a completed ack; the Hub parks the renewal in action_needed and does not re-issue (re-issuing can't fix a flaky reload).
  • Serving a third, unexpected cert → served_wrong. A genuine failure: the previous cert is restored. Rides a failed ack; the Hub enters the bounded reissue chain (max 3 retries).

Best-effort: an unreachable endpoint never fails the install. - Path containment — every Hub-supplied path (install_path, key_path, …) must resolve under the data dir or a configured cert directory — the --cert-dir flags and the directories listed in the cert-dirs file carry equal authority here — with symlink-escape rejected — fail-closed against a Hub naming /etc/shadow. - Hooks are operator-local, never Hub-supplied — Scout runs <data-dir>/hooks/{pre,deploy,post} at each lifecycle point; the Hub cannot ship a hook. See Security. - Per-domain locking serializes file mutations for one domain; different domains run concurrently. - Graceful drain — on shutdown Scout waits for in-flight commands (and their acks) before disconnecting; a command arriving mid-drain is dropped cleanly, never raced.

At-most-once delivery

Commands travel over core NATS with no redelivery. A command published while a Scout is offline is not delivered and is not re-sent — the Hub's commands row for it expires after 1 hour, at which point the Hub raises a warning naming the Scout and the command and counts it toward Pipeline health, but nothing retries it automatically: the operator must re-issue the action once the Scout is back. The offline Scout itself is also surfaced without any configuration: the Hub tracks live NATS state (last_seen and /readyz reflect real reachability), and an always-on, decoupled "scout went dark" alert fires the first time it crosses the dark-alert threshold — independently of whether the separate, opt-in, destructive auto-disable (AGENT_STALENESS_HOURS) is configured. See Monitoring → Scout-went-dark alerts and When the link drops.

DNS-01 stall detection

For manual DNS-01, if the Hub never sends the follow-up deliver_cert, the preserved key would sit forever with no signal. A reconcile ticker (every 5 min) flags any pending record older than 30 min with a warning. It is detection only — it does not re-drive the Hub or re-issue.