Host updates and rollback

Apply signed releases without widening root access

Host updates and rollback is the signed native-release lifecycle exposed to instance administrators through a narrow root-owned agent.

Operators need a visible, auditable upgrade path without granting the portal or API general root or shell execution.

Use it after a native install, before each server upgrade, when choosing an automatic check policy, or when a verified release must be rolled back.

Requirements for Host updates and rollback

• Prerequisite — A packaged native installation with the updater path unit and check timer active.
• Prerequisite — Off-host backup custody, working readiness checks, and an owner or instance-administrator platform account.
• Required value — The exact announced release version, canonical signature identity, compatibility state schema, and retained job ID.
• Required value — For automatic apply, an explicit UTC maintenance window and operational coverage.

Use the native lifecycle boundary

A packaged native installation includes a root-owned, path-activated updater. The portal and Axum API can request only six typed operations: inspect, check, install an exact version, roll back to an installed exact version, read a job, or update the bounded schedule. They cannot supply a URL, filesystem path, command, or shell argument. The updater accepts only canonical releases whose checksum list has a valid GitHub Actions Sigstore identity and whose requested asset is named by the release manifest.

• FFDB releases live side by side under /opt/ffdb/releases, and /opt/ffdb/current changes atomically only after verification and backup succeed.
• Caddy and the compiled portal stay available while Axum and the sync worker restart, so the progress screen can reconnect to the persisted job.
• Every install and rollback takes a coordinated host backup, holds one global update lock, records an instance audit event, and requires readiness after restart.
• The API runs as ffdb and only writes a bounded request into the updater queue. It never receives root, sudo, or general process-execution authority.

Verify lifecycle services

The native installer enables the request watcher and periodic release check with the application services. Automatic checks are on by default; automatic application is off. Inspect the service boundary after a fresh install or upgrade before relying on portal controls.

sudo systemctl --no-pager --full status \
  ffdb-update-agent.path \
  ffdb-update-check.timer
sudo -u ffdb /usr/local/bin/ffdb-update inspect
sudo journalctl -u ffdb-update-agent.service --since today --no-pager

Check and install from the portal

Open Global administration, then Updates. Check for updates reads the stable channel without restarting services. Install is shown only for a newer compatible signed release. Owners and instance administrators may inspect and check; install, rollback, and schedule changes require a platform session issued within the previous 15 minutes. If the session is older, the portal asks for the account password through the normal sign-in endpoint, replaces the session, and retries the exact pending action. Passwords are never forwarded to the updater.

• Review the target version, signature identity, compatibility state schema, release notes link, and backup requirement before confirming.
• The API returns a job immediately. Keep the progress view open; temporary connection failures during Axum restart are expected and are retried with a bounded backoff.
• Success requires the selected version to be active and both direct API and gateway readiness to pass. A failed health check restores the previous active release and leaves the failure record available for diagnosis.
• Use the job ID and request ID when correlating the portal state with the immutable audit log and system journal.

Configure checks and maintenance

The stable channel is the only production channel. Choose the check interval and, only if unattended maintenance is acceptable, explicitly enable automatic application with a UTC maintenance window. A check outside the window records availability but does not restart the host. Disabling automatic checks stops network discovery without hiding already installed versions or job history.

Roll back a compatible installed release

Rollback selects an already verified release stored on the host; it never downloads an arbitrary older binary. FFDB compares the active and target state-schema metadata and the release rollback floor before enabling the action. A rollback that would cross an incompatible control-plane or durable-state boundary is rejected. Restore the pre-update backup with the documented recovery procedure instead of bypassing that guard.

• A rollback creates another backup before changing the active symlink.
• API, database-worker, sync-worker, web assets, units, and gateway configuration move as one versioned set.
• The previous failed release remains installed for investigation, but it is not selected automatically again.
• If the portal cannot reconnect, inspect the persisted job and journal locally; do not repeatedly submit the same operation.

Recover from an interrupted job

Jobs and the active-release pointer survive API and host restarts. If power is lost before the atomic switch, the old release stays active. If it is lost after the switch, systemd starts the selected release and the retained job records the last completed phase. Resolve the current state before submitting a new install or rollback.

sudo -u ffdb /usr/local/bin/ffdb-update inspect
sudo -u ffdb /usr/local/bin/ffdb-update job "$JOB_ID"
sudo systemctl status ffdb-api.service ffdb-sync-worker.service ffdb-gateway.service
sudo journalctl -u ffdb-update-agent.service -u ffdb-api.service \
  -u ffdb-sync-worker.service --since today --no-pager
curl --fail http://127.0.0.1:8080/readyz
curl --fail http://127.0.0.1:5173/readyz

Host updates and rollback workflow

• 1. Inspect the installed and available versions plus updater capabilities.
• 2. Check the stable release channel and review compatibility and release notes.
• 3. Reauthenticate if required, confirm the mandatory backup, and submit the exact install version.
• 4. Follow the persisted job through verification, backup, activation, restart, and readiness.
• 5. If acceptance fails, select only a compatible installed release or restore the coordinated backup.

Verify host updates and rollback

One signed release is active atomically, the complete versioned service set is ready, and the audit/job record identifies every lifecycle phase.

Troubleshoot host updates and rollback

• Signature, checksum, or manifest verification fails — quarantine the download and do not activate it.
• The compatibility guard rejects rollback — restore the coordinated backup rather than bypassing the state boundary.
• The portal disconnects during restart — keep the job ID and let bounded reconnect polling resume before submitting anything else.

Continue from Host updates and rollback

• Run release acceptance against auth, RLS, storage, sync, observability, and restore.
• Review retained jobs and updater timer policy during normal operations.