Fleet upgrades¶

A Cyphers estate is one Hub and a fleet of Scouts, and upgrading either is mechanically simple — each is a single static binary you replace and restart. What makes an upgrade an operation is the ordering, the rollback story, and the verification between steps. This page is the altitude view; the component pages carry the exact commands.

The whole procedure, in order:

Back up the Hub's database.
Upgrade the Hub; verify it.
Roll the Scouts, one at a time, verifying each.
Run the post-upgrade checks across the fleet.

The one ordering rule: Hub first, then Scouts¶

The Hub↔Scout wire contract is strict on both sides — every wire message rejects fields it does not know (deny_unknown_fields in the Rust wire types, DisallowUnknownFields() in the Go mirror). A newer Scout may report fields an older Hub has never heard of, and the older Hub then rejects that report outright rather than ignoring the extras. A Scout that is newer than its Hub is not degraded — it is deaf.

Never run a Scout newer than the Hub

Upgrade order is always Hub, then Scouts. The reverse skew is safe: an older Scout simply omits newer fields, which the Hub treats as "not reported". A fleet part-way through a rollout is therefore fine — for as long as it takes — provided every Scout is no newer than the Hub.

This is also why there is no pressure to rush step 3: once the Hub is upgraded and verified, the old Scouts keep reporting normally.

Before touching the Hub: back up¶

Hub upgrades run database migrations, and migrations can be irreversible — a downgraded binary cannot necessarily run against an upgraded schema. Rolling back a Hub upgrade therefore needs both the old binary and a database backup taken before the new binary first started. Missing either one, there is no rollback — only forward fixes.

Take the backup with Back up now in the dashboard, or — purpose-built for this moment — from the Hub host's CLI:

$ cyphers-hub backup --label pre-upgrade
/mnt/backup/cyphers/pre-upgrade-20260822_140301.sql.enc (2469888 bytes)

The backup subcommand is deliberately wired to run before the binary applies any migrations, so it is safe to run it with the new binary against the still-live old database — the dump reflects the pre-upgrade schema either way. Copy the file (and keep the old Hub binary) somewhere off the Hub host before proceeding.

Backups must already be configured (directory + passphrase) for either path to work — set that up once via Back up and restore the Hub.

Upgrading the Hub¶

A Hub upgrade is: stop, replace the binary (or load the new images), start. Migrations are not a separate step — the Hub applies them itself:

Bare install: the server runs its migrations at startup, before it begins serving. Replace cyphers-hub, systemctl restart cyphers-hub, done.
Packaged Docker install: the container entrypoint waits for the database, runs doctor --apply-migrations with visible pass/fail output, then starts the Hub. Load the new images and docker compose up -d.

Install paths, units, and the Compose layout are the install page's territory — see Hub → Install procedure; an upgrade touches the same locations it describes.

Then verify before going anywhere near a Scout:

$ curl http://hub.example.internal:8443/health     # liveness — expect: ok
$ curl http://hub.example.internal:8443/readyz     # readiness — see below

/readyz reports four checks — database, migrations, ca_storage, crypto_posture — and answers 503 if any fails. All four ok, plus a successful dashboard login, is the green light for step 3.

Rolling the Hub back¶

Hub rollback = old binary + pre-upgrade backup

If the upgraded Hub must come back down, reinstalling the old binary is not sufficient on its own once migrations have run — the old binary may refuse or misbehave on the new schema. The rollback is: stop the Hub, reinstall the old binary, restore the pre-upgrade backup, start. And restoring replaces the entire database — everything recorded after the backup is lost — so the earlier this decision is made, the cheaper it is. Restore mechanics: Back up and restore the Hub.

Rolling the Scouts¶

With the Hub upgraded and verified, upgrade the Scouts one at a time, verifying each before starting the next. A Scout is the only thing observing its endpoints, so a broken upgrade rolled out in parallel blinds the whole estate at once instead of one host.

The per-Scout procedure — record the running version, keep the old binary as the rollback, install, restart, confirm the reconnect in the journal — is documented step-by-step in Scout → Install procedure → Upgrade. Per Scout, the verification that matters at fleet altitude:

the service is active on the host and the Scout reconnected to the Hub (a NATS connected line in its journal);
on the dashboard's Scouts page, that Scout's Last seen advances within a minute of the restart.

A Scout that fails to reconnect gets rolled back immediately from its saved binary (rolling a fleet) — and the rest of the rollout pauses until you know why.

Post-upgrade checks¶

When the last Scout is done, sweep the estate once:

http://hub.example.internal:8443/health answers ok, and /readyz reports all four checks ok.
Scouts shows every Scout's Last seen advancing — none stuck at its pre-upgrade timestamp.
Activity is clean: no new stream of errors that began at the upgrade. One-off noise from restarts is normal; a repeating error is not.
On Manage, spot-check that rows look as they did before — an upgrade should change the software, not the estate's posture.

Keep the pre-upgrade backup and old binaries until the fleet has been stable through at least one full scan cycle; after that, the labelled dump ages out with normal retention.