Scout — transport & health¶
After enrollment, NATS over mTLS is the only interactive channel — there is
no HTTP fallback, and Scout refuses to start without --hub-nats.
Defaults at a glance¶
Every operational default in one place. Elsewhere in these docs each value is also stated where it applies; this is the quick reference.
Ports. These are defaults, not fixed ports — the actual value is whatever the Hub was configured with and reaches the Scout baked into the installer URLs.
| Port | Default | Purpose |
|---|---|---|
| Enrollment (TLS) | 7443 | one-time enrollment and identity-cert rotation |
| NATS (mTLS) | 4222 | commands, reports, acks, heartbeats, CSRs |
| Hub PKI reads (HTTP) | 8443 | CA chain/root, CRL, OCSP — public read-only |
| Health endpoints | off | enabled only with --health-port |
deliver_cert served re-probe |
443 | local re-probe after a cert reload (serve_port) |
Timers and intervals.
| Setting | Default |
|---|---|
Scan interval (--interval) |
6 h |
| Heartbeat | every 30 s |
| Identity-rotation check | every 6 h |
| Identity-rotation window | clamp(lifetime ÷ 4, 7 d, 30 d) |
| DNS-01 stall reconcile | every 5 min |
| DNS-01 stall threshold | 30 min |
| Acceptance-policy staleness tolerance | 1 h |
| Acceptance-policy republish (Hub) | every 10 min |
Timeouts and retries.
| Setting | Default |
|---|---|
| Enrollment token validity | 1 h |
| Command-row validity (Hub-side) | 1 h |
| ACME HTTP-01 obtain | 3 min |
| CSR request/reply | 60 s |
| CRL fetch | 5 s |
| Ack publish retries | 3 attempts, 50 ms apart |
| NATS reconnect backoff | base 1 s, cap 60 s, retries forever |
Logging.
| Setting | Default |
|---|---|
Log level (--log-level) |
info |
stderr format (--log-format) |
text |
Log file (--log-file) |
/var/log/cyphers-scout/scout.log |
Rotation threshold (--log-max-size-mb) |
20 MB |
Backups kept (--log-max-backups) |
5 |
Full detail: Logging.
Subjects¶
| Subject | Direction | Payload |
|---|---|---|
cyphers.hub.{id}.command |
Hub → Scout | command envelope |
cyphers.scout.{id}.report |
Scout → Hub | AgentReport |
cyphers.scout.{id}.ack |
Scout → Hub | CommandAck (+ typed result) |
cyphers.scout.{id}.heartbeat |
Scout → Hub | empty, every 30 s |
cyphers.scout.{id}.csr |
Scout → Hub | CSR request/reply |
cyphers.scout.{id}.verification_decision |
Scout → Hub | acceptance-policy verdicts |
cyphers.scout.{id}.policy_status |
Scout → Hub | policy apply-status |
Per-agent isolation — the ACL wall¶
One of Scout's most important security properties. Every Scout connects to NATS with its own enrollment certificate (mTLS both ways), and its identity and permissions come from the server, never from anything the Scout claims:
| Step | Mechanism |
|---|---|
| Certificate → identity | The Hub's NATS server verifies the client certificate and maps its CN to a user (verify_and_map). A Scout is whoever its certificate says — nothing self-declared. |
| Identity → permissions | A Hub-generated per-agent ACL limits that user to exactly two subject patterns: publish cyphers.scout.{id}.> and subscribe cyphers.hub.{id}.> — its own channel, nothing else. |
Because the enforcement is server-side, a modified or compromised Scout binary cannot opt out. Concretely, a compromised Scout:
- cannot publish as another agent — no forged reports, acks, or heartbeats under someone else's id;
- cannot subscribe to another agent's subjects — no reading other hosts' commands or telemetry;
- is confined to its own channel — the blast radius of one compromised host is that host.
Resilience¶
MaxReconnects(-1) (retries forever) with exponential backoff +
equal jitter (base 1 s, cap 60 s) to avoid a fleet-wide thundering herd
against a restarted Hub. A graceful drain waits for in-flight commands before
disconnecting, and cert rotation triggers a clean reconnect with fresh creds.
Acceptance-policy verification¶
The acceptance policy is a second, independent check of the Hub's identity on every NATS connection, layered on top of the always-on pinned-mTLS baseline. What a policy is, who signs it, and how the Scout comes to trust it are introduced in Security → Independent verification of the Hub's identity. This section documents how it behaves on the wire.
Enforcement is mandatory and policy-before-connect. There is no ratchet, no
observe-only or baseline mode, and no fleet-wide enforcement flip — a Scout
never dials NATS without a verified signed policy already in hand, from its
very first connection. Enrollment always delivers the signer's public key and
a signed policy in the same reply, and Scout verifies and applies both before
attempting its first NATS connect, so a fresh install has no bootstrap gap. If
a Scout ever finds itself holding valid credentials but no cached policy (for
example the local cache file was deleted), it re-enrolls over --hub-rest to
re-seed a fresh signed policy before connecting — it never falls back to
an unverified connection. That re-seed needs a still-valid re-enrollment token
(persisted in state.json); if that token has also been lost or expired, the
fix is re-running the installer with a fresh --token (and
--policy-signer-fp) from the Hub UI.
Every connection runs the verifier in enforce mode. A handshake whose Hub
identity fails the policy — an unaccepted URI SAN, a revoked peer, or (under
required) an undetermined one — is refused, as is a missing policy or
one older than the 1 h staleness tolerance. Enforcement is fail-closed: a
bad or stale policy keeps a Scout from connecting, which is the accepted cost
of the guarantee; the refusal log names the reason and the fix (re-enroll over
--hub-rest re-delivers a fresh policy).
The Hub keeps every enrolled agent's cached policy fresh by re-signing and republishing on a fixed cadence (every 10 min) — several refreshes per staleness window — so a scout that stays connected never ages into a stale refusal.
Mechanism.
- The signer public key is pinned by fingerprint via
--policy-signer-fp, required on first enrollment alongside--ca-fingerprint. The scout fetches the signer's PEM in the enroll reply, verifies its fingerprint against the pin, and persists the verified key locally — the pin is never taken on faith from the reply, only the fingerprint match is trusted. - A TLS peer-verification callback extracts the Hub certificate's identity names (URI SANs) and checks them against this agent's entry in the signed policy (ES256).
- The cached policy is anti-rollback (its version can never regress) and
staleness-bounded (
generated_atolder than the tolerance is refused). - The first policy arrives in the pin-verified enrollment reply and is applied
through the same verified path; identity rotation re-delivers it; steady-state
updates arrive over NATS. Every apply publishes a
policy_status, and every connection publishes the policy decision (which also gates the handshake).
Peer certificate revocation¶
The verifier resolves the peer certificate's revocation status
(revocation_check) and applies the configured policy mode:
- Primary — stapled OCSP. The scout reads the OCSP response the Hub NATS
server staples into the handshake (
cs.OCSPResponse) — no extra network call. It's verified against the pinned issuer; a delegated responder cert must carryid-kp-OCSPSigning(RFC 6960); out-of-date orNextUpdate-less responses are rejected. - Fallback — CRL. With no valid staple, the scout fetches the leaf's CRL from the URL in the certificate itself (its CRL Distribution Point — the Hub's PKI port, default 8443). The fetch is bounded at 5 s and cached; if that port is unreachable the status is simply "undetermined" — it never hangs or breaks the agent (see which ports identity operations need).
- Modes:
required→ revoked or undetermined ⇒ reject (fail-closed);soft-fail→ revoked ⇒ reject, undetermined ⇒ accept (fail-open);none⇒ skip. The Hub generatesrequiredby default — the weaker modes are an explicit per-resource opt-out. Every error path is fail-safe undetermined — a revoked or unverifiable cert can never read as good.
Stapling is mandatory
The Hub's OCSP responder is always mounted, CYPHERS_CA_URL is required
at startup (every issued cert carries CRL + OCSP pointers), and the
generated NATS conf always sets ocsp { mode: always } — so a staple is
present on every handshake. Two operational consequences:
- A NATS server cert minted before
ca_urlwas enforced has no OCSP AIA URL, and nats-server refusesmode: alwayswith such a cert — re-runnats-bootstrapto re-mint it when migrating an existing deployment. - nats-server fetches its staple from the Hub's OCSP responder, so on a cold boot NATS only comes up once the Hub is reachable — container restart policies retry until it is.
In enforce mode a revoked peer (or, under required, an undetermined one)
is blocked, and the rejection is still published so the Hub can show why
the scout refused. The mandatory staple is what makes that decision sound.
Decisions are buffered during the handshake (no connection exists yet to publish on), then flushed once per connect and again only when the decision changes (deduped), advancing only on a successful publish so a transient failure retries.
Health endpoints¶
Enabled with --health-port (default off):
/healthz(liveness) — always 200; body reportshub: "healthy" | "unreachable", uptime, agent id, last scan./readyz(readiness) — 200 only if enrolled and NATS connected, else 503.
Both derive their hub/ready view from the live NATS connection state
(*nats.Conn.IsConnected()), so they reflect real, current Hub reachability.