Scout — commands & remediation

When the Hub needs Scout to act — issue, deliver, revoke, reload — it dispatches a command on cyphers.hub.{id}.command. Scout strict-decodes the envelope, dispatches by verb on its own goroutine, and acks on cyphers.scout.{id}.ack: first received, then a terminal completed / failed carrying a typed CommandResult. Each ack is published with up to 3 attempts, 50 ms apart (see Defaults at a glance).

The verbs

Verb	What it does	End-to-end status
scan_now	Full scan cycle; returns a report	Live
scan_target	Scan one Hub-named host:port	Live
acme_http01	Scout runs the entire ACME HTTP-01 flow locally (lego)	Live
acme_dns01_direct	Gen key+CSR → Hub (Hub publishes DNS TXT via its provider)	Live (Hub needs a DNS provider configured)
acme_dns01_manual	Gen key+CSR → Hub returns challenges; operator adds TXT; Hub watchdog finishes	Live
acme_dns01_delegated	Gen key+CSR → Hub publishes TXT via its delegation provider (Mode A local / Mode B to a master)	Live (Hub needs delegation configured)
internal_ca	Gen key+CSR → Hub's built-in CA signs directly (also break-glass)	Live
deliver_cert	Install a Hub-issued cert — the DNS-01 second leg	Live
revoke_cert	ACME-revoke and/or destroy the key (destroy_key is a payload field)	Live

acme_dns01_delegated needs delegation configured

The Hub publishes the TXT record through whatever delegation provider select_provider installed — Mode A (local zone remap) or Mode B (mTLS client to a master Hub's /api/delegation/txt). If the Hub has no DNS provider configured, the verb fails with a clear error naming the fix (configure delegation, or switch the profile to local_dns), rather than silently degrading.

destroy_key is not a verb

The old standalone destroy_key command was superseded by revoke_cert{destroy_key:true} and is absent from the dispatch switch — a Hub that sent it would get "unknown command."

ACME HTTP-01 is special

For acme_http01, Scout drives the whole flow itself: it loads/creates a persistent ACME account, listens on port 80 to answer the challenge, obtains the cert (bounded by a 3-minute timeout since lego's obtain can't be cancelled), and installs it. It even probes the currently-served cert before running the local pre hook (which typically stops the web server), so it can find and re-point the right on-disk symlinks afterward. There's a one-shot ARI retry for the common "issued under a different ACME account" mismatch. The private key is generated locally and never leaves the host.

The other issuance verbs

acme_dns01_* and internal_ca share one handler that generates the key+CSR locally and submits the CSR to the Hub over NATS request/reply (cyphers.scout.{id}.csr, 60 s timeout). The Hub replies with the issued cert, a dns_challenge_pending (for manual DNS-01 — Scout keeps the key and writes a stall-detection record), or an error.

Safety mechanisms (the important part)

Scout treats the Hub as semi-trusted and defends the host filesystem:

• Backup before key rotation, rollback on failure. The existing cert+key are backed up before the key is overwritten, so a validation or submit failure restores a matched pair rather than orphaning the host.
• 5-point post-install validation — cert parses, key↔cert public keys match, chain present, SAN covers the hostname, and the local deploy hook succeeds. Any failure triggers a rollback (and re-runs the local deploy hook).
• Cert↔key pairing guards — checked before writing on deliver_cert and before destroying a key on revoke_cert, so a wrong (but in-bounds) key_path can't irreversibly destroy the wrong key.
• deliver_cert SAN + post-reload served re-probe — the delivered cert must cover every requested domain (pre-write, rejected before any write); after the reload Scout dials the local endpoint (serve_port, default 443) and compares the served leaf to the delivered cert. Three outcomes, and the Hub treats them differently:
  • Match → success.
  • Still serving the previous cert → reload_ineffective. The new cert is valid and installed; only the reload didn't take, so the good cert is not reverted. Rides a completed ack; the Hub raises a "manual reload needed" alert and does not re-issue (re-issuing can't fix a flaky reload).
  • Serving a third, unexpected cert → served_wrong. A genuine failure: the previous cert is restored. Rides a failed ack; the Hub enters the bounded reissue chain (max 3 retries).

Best-effort: an unreachable endpoint never fails the install. - Path containment — every Hub-supplied path (install_path, key_path, …) must resolve under the data dir or a --cert-dir, with symlink-escape rejected — fail-closed against a Hub naming /etc/shadow. - Hooks are operator-local, never Hub-supplied — Scout runs <data-dir>/hooks/{pre,deploy,post} at each lifecycle point; the Hub cannot ship a hook. See Security. - Per-domain locking serializes file mutations for one domain; different domains run concurrently. - Graceful drain — on shutdown Scout waits for in-flight commands (and their acks) before disconnecting; a command arriving mid-drain is dropped cleanly, never raced.

At-most-once delivery

Commands travel over core NATS with no redelivery. A command published while a Scout is offline is not delivered and is not re-sent — the Hub's commands row for it expires after 1 hour. The offline Scout is still surfaced to the Hub rather than silently missing work: the Hub tracks live NATS state (last_seen and /readyz reflect real reachability), and when agent staleness checking is enabled (AGENT_STALENESS_HOURS) it raises a distinct "scout went dark" alert in the activity feed.

DNS-01 stall detection

For manual DNS-01, if the Hub never sends the follow-up deliver_cert, the preserved key would sit forever with no signal. A reconcile ticker (every 5 min) flags any pending record older than 30 min with a warning. It is detection only — it does not re-drive the Hub or re-issue.