Skip to content

Scout — transport & health

After enrollment, NATS over mTLS is the only interactive channel — there is no HTTP fallback, and Scout refuses to start without --hub-nats.

Defaults at a glance

Every operational default in one place. Elsewhere in these docs each value is also stated where it applies; this is the quick reference.

Ports. These are defaults, not fixed ports — the actual value is whatever the Hub was configured with and reaches the Scout baked into the installer URLs.

Port Default Purpose
Enrollment (TLS) 7443 one-time enrollment and identity-cert rotation
NATS (mTLS) 4222 commands, reports, acks, heartbeats, CSRs
Hub PKI reads (HTTP) 8443 CA chain/root, CRL, OCSP — public read-only
Health endpoints off enabled only with --health-port
deliver_cert served re-probe 443 local re-probe after a cert reload (serve_port)

Timers and intervals.

Setting Default
Scan interval (--interval) 6 h
Heartbeat every 30 s
Identity-rotation check every 6 h
Identity-rotation window clamp(lifetime ÷ 4, 7 d, 30 d)
DNS-01 stall reconcile every 5 min
DNS-01 stall threshold 30 min
Acceptance-policy staleness tolerance 1 h
Acceptance-policy republish (Hub) every 10 min

Timeouts and retries.

Setting Default
Enrollment token validity 1 h
Command-row validity (Hub-side) 1 h
ACME HTTP-01 obtain 3 min
CSR request/reply 60 s
CRL fetch 5 s
Ack publish retries 3 attempts, 50 ms apart
NATS reconnect backoff base 1 s, cap 60 s, retries forever

Logging.

Setting Default
Log level (--log-level) info
stderr format (--log-format) text
Log file (--log-file) /var/log/cyphers-scout/scout.log
Rotation threshold (--log-max-size-mb) 20 MB
Backups kept (--log-max-backups) 5

Full detail: Logging.

Subjects

Subject Direction Payload
cyphers.hub.{id}.command Hub → Scout command envelope
cyphers.scout.{id}.report Scout → Hub AgentReport
cyphers.scout.{id}.ack Scout → Hub CommandAck (+ typed result)
cyphers.scout.{id}.heartbeat Scout → Hub empty, every 30 s
cyphers.scout.{id}.csr Scout → Hub CSR request/reply
cyphers.scout.{id}.verification_decision Scout → Hub acceptance-policy verdicts
cyphers.scout.{id}.policy_status Scout → Hub policy apply-status

Per-agent isolation — the ACL wall

One of Scout's most important security properties. Every Scout connects to NATS with its own enrollment certificate (mTLS both ways), and its identity and permissions come from the server, never from anything the Scout claims:

Step Mechanism
Certificate → identity The Hub's NATS server verifies the client certificate and maps its CN to a user (verify_and_map). A Scout is whoever its certificate says — nothing self-declared.
Identity → permissions A Hub-generated per-agent ACL limits that user to exactly two subject patterns: publish cyphers.scout.{id}.> and subscribe cyphers.hub.{id}.> — its own channel, nothing else.

Because the enforcement is server-side, a modified or compromised Scout binary cannot opt out. Concretely, a compromised Scout:

  • cannot publish as another agent — no forged reports, acks, or heartbeats under someone else's id;
  • cannot subscribe to another agent's subjects — no reading other hosts' commands or telemetry;
  • is confined to its own channel — the blast radius of one compromised host is that host.

Resilience

MaxReconnects(-1) (retries forever) with exponential backoff + equal jitter (base 1 s, cap 60 s) to avoid a fleet-wide thundering herd against a restarted Hub. A graceful drain waits for in-flight commands before disconnecting, and cert rotation triggers a clean reconnect with fresh creds.

Acceptance-policy verification

The acceptance policy is a second, independent check of the Hub's identity on every NATS connection, layered on top of the always-on pinned-mTLS baseline. What a policy is, who signs it, and how the Scout comes to trust it are introduced in Security → Independent verification of the Hub's identity. This section documents how it behaves on the wire.

Enforcement is mandatory and policy-before-connect. There is no ratchet, no observe-only or baseline mode, and no fleet-wide enforcement flip — a Scout never dials NATS without a verified signed policy already in hand, from its very first connection. Enrollment always delivers the signer's public key and a signed policy in the same reply, and Scout verifies and applies both before attempting its first NATS connect, so a fresh install has no bootstrap gap. If a Scout ever finds itself holding valid credentials but no cached policy (for example the local cache file was deleted), it re-enrolls over --hub-rest to re-seed a fresh signed policy before connecting — it never falls back to an unverified connection. That re-seed needs a still-valid re-enrollment token (persisted in state.json); if that token has also been lost or expired, the fix is re-running the installer with a fresh --token (and --policy-signer-fp) from the Hub UI.

Every connection runs the verifier in enforce mode. A handshake whose Hub identity fails the policy — an unaccepted URI SAN, a revoked peer, or (under required) an undetermined one — is refused, as is a missing policy or one older than the 1 h staleness tolerance. Enforcement is fail-closed: a bad or stale policy keeps a Scout from connecting, which is the accepted cost of the guarantee; the refusal log names the reason and the fix (re-enroll over --hub-rest re-delivers a fresh policy).

The Hub keeps every enrolled agent's cached policy fresh by re-signing and republishing on a fixed cadence (every 10 min) — several refreshes per staleness window — so a scout that stays connected never ages into a stale refusal.

Mechanism.

  • The signer public key is pinned by fingerprint via --policy-signer-fp, required on first enrollment alongside --ca-fingerprint. The scout fetches the signer's PEM in the enroll reply, verifies its fingerprint against the pin, and persists the verified key locally — the pin is never taken on faith from the reply, only the fingerprint match is trusted.
  • A TLS peer-verification callback extracts the Hub certificate's identity names (URI SANs) and checks them against this agent's entry in the signed policy (ES256).
  • The cached policy is anti-rollback (its version can never regress) and staleness-bounded (generated_at older than the tolerance is refused).
  • The first policy arrives in the pin-verified enrollment reply and is applied through the same verified path; identity rotation re-delivers it; steady-state updates arrive over NATS. Every apply publishes a policy_status, and every connection publishes the policy decision (which also gates the handshake).

Peer certificate revocation

The verifier resolves the peer certificate's revocation status (revocation_check) and applies the configured policy mode:

  • Primary — stapled OCSP. The scout reads the OCSP response the Hub NATS server staples into the handshake (cs.OCSPResponse) — no extra network call. It's verified against the pinned issuer; a delegated responder cert must carry id-kp-OCSPSigning (RFC 6960); out-of-date or NextUpdate-less responses are rejected.
  • Fallback — CRL. With no valid staple, the scout fetches the leaf's CRL from the URL in the certificate itself (its CRL Distribution Point — the Hub's PKI port, default 8443). The fetch is bounded at 5 s and cached; if that port is unreachable the status is simply "undetermined" — it never hangs or breaks the agent (see which ports identity operations need).
  • Modes: required → revoked or undetermined ⇒ reject (fail-closed); soft-fail → revoked ⇒ reject, undetermined ⇒ accept (fail-open); none ⇒ skip. The Hub generates required by default — the weaker modes are an explicit per-resource opt-out. Every error path is fail-safe undetermined — a revoked or unverifiable cert can never read as good.

Stapling is mandatory

The Hub's OCSP responder is always mounted, CYPHERS_CA_URL is required at startup (every issued cert carries CRL + OCSP pointers), and the generated NATS conf always sets ocsp { mode: always } — so a staple is present on every handshake. Two operational consequences:

  • A NATS server cert minted before ca_url was enforced has no OCSP AIA URL, and nats-server refuses mode: always with such a cert — re-run nats-bootstrap to re-mint it when migrating an existing deployment.
  • nats-server fetches its staple from the Hub's OCSP responder, so on a cold boot NATS only comes up once the Hub is reachable — container restart policies retry until it is.

In enforce mode a revoked peer (or, under required, an undetermined one) is blocked, and the rejection is still published so the Hub can show why the scout refused. The mandatory staple is what makes that decision sound.

Decisions are buffered during the handshake (no connection exists yet to publish on), then flushed once per connect and again only when the decision changes (deduped), advancing only on a successful publish so a transient failure retries.

Health endpoints

Enabled with --health-port (default off):

  • /healthz (liveness) — always 200; body reports hub: "healthy" | "unreachable", uptime, agent id, last scan.
  • /readyz (readiness) — 200 only if enrolled and NATS connected, else 503.

Both derive their hub/ready view from the live NATS connection state (*nats.Conn.IsConnected()), so they reflect real, current Hub reachability.