RootTrace

On-prem upgrade guide

Supported upgrades and rollbacks for Compose and air-gapped installations.

Get and verify the new release

Same as a fresh install: download the five versioned URLs named by https://packages.roottrace.io/releases/latest.json, then

sha256sum -c roottrace-deployment-<version>.tar.gz.sha256
cosign verify-blob \
  --key roottrace-cosign.pub \
  --insecure-ignore-tlog=true \
  --signature roottrace-deployment-<version>.tar.gz.sig \
  roottrace-deployment-<version>.tar.gz
tar -xzf roottrace-deployment-<version>.tar.gz
cd roottrace-deployment-<version>
./verify-images.sh

Stop if any of the four checks fails. On macOS use shasum -a 256 -c for the first line.

Deploy the digest from release/release-manifest.json in that verified archive, not a numbered tag and never :latest, which moves. Record the source digest, target digest, backup identifier, and rollback decision point in your change record.

The three rules

  1. Preserve the application secret. JWT signing keys and encrypted integration secrets are unrecoverable without it. If the installation predates the secret file, export it first; see below.
  2. Back up MongoDB and test the restore. A successful dump is not restore evidence.
  3. Change one thing. Do not combine a RootTrace upgrade with a MongoDB, Elasticsearch, ingress, certificate, or OS upgrade.

Schema migrations are automatic, tracked, and forward-only. Keep the previous image available until the observation window closes.

One-way notes

The changelog for a release is authoritative. The hardened on-prem release introduced these operator-visible changes:

  • ROOTTRACE_SECURITY_PROFILE defaults to regulated in production. Startup now rejects unsafe combinations such as wildcard CORS/proxy trust, disabled audit chaining or rate limiting, plaintext Elasticsearch, partial direct-TLS configuration, and an invalid FIPS claim. Validate the effective posture in the Flutter administration screens before rollout.
  • The vendor cloud email relay defaults off. Configure direct SMTP if the site needs email, or leave email disabled. Do not re-enable the cloud relay at an air-gapped site.
  • Elasticsearch defaults to disabled. When configured under a regulated profile, its endpoint must use HTTPS. The current Compose deployment does not start an unauthenticated Elasticsearch container.
  • /metrics requires ROOTTRACE_METRICS_BEARER_TOKEN outside development. Update the monitoring secret and scraper together.
  • Web sessions survive a page reload: the dashboard restores tokens from the browser's secure storage. Sign-out, token expiry, and an administrator's session revocation all still end a session immediately.
  • FIPS mode accepts only RS256 or ES256 and refuses startup unless the active OpenSSL provider is actually operating in FIPS mode.
  • JWT signing now defaults to post-quantum ML-DSA-65 when FIPS mode is off. On the first restart, an installation without an explicit database or environment selection creates a new ML-DSA-65 signing key. Existing tokens remain verifiable during the key overlap, but the larger signatures may require higher reverse-proxy or gateway header limits. Platform admins can select a compatibility algorithm in Flutter before a later restart.
  • Existing email-based owners are not automatically treated as on-prem platform admins. Before the first upgraded startup, set ROOTTRACE_ADMIN_EMAILS to their exact comma-separated email addresses. Fresh local-username owners do not need this bridge.
  • The collector no longer has a default ingest destination. Earlier versions fell back to RootTrace Cloud when ROOTTRACE_API_URL was unset; the new collector exits with an error instead. Before upgrading a fleet, confirm the variable is set in /etc/roottrace/collector.env, the container environment, or the DaemonSet ConfigMap on every host. A collector that was relying on the old default was sending diagnostics off-site, so treat a host that fails to start after upgrade as a finding, not just a configuration gap.

Test these behaviors in staging. An upgrade must not silently work around a regulated-profile startup failure by selecting the default profile.

Before you start

  1. Check the current install is healthy: curl --cacert <ca.pem> https://roottrace.example.com/readyz. Don't use curl -k as a production health check.
  1. Record the running image so you can roll back to it:
   docker inspect --format '{{.Config.Image}}' \
     "$(docker compose --env-file deploy/.env -f deploy/docker-compose.yml ps -q roottrace)"
  1. Back up MongoDB. The helper runs mongodump inside the internal Mongo service and pauses the API briefly, so no host port or host tools are needed:
   scripts/mongo_backup.sh /secure/roottrace-backups

For an external MongoDB, install the official Database Tools and set ROOTTRACE_MONGO_URI first. Managed MongoDB should use its own application-consistent snapshot. Record the backup id and test the restore somewhere else.

  1. Diff the shipped config against the new bundle before editing anything live: deploy/.env.example, deploy/docker-compose.yml, deploy/nginx/reverse-proxy.conf, plus anything changed under deploy/mongo/ and scripts/. Carry your existing values across; do not adopt .env.example defaults wholesale.
  1. Rehearse in staging against a restored copy of production data: login and MFA, workspace access, collector ingest, SSO/SCIM if used, notifications, repository links, /readyz, and authenticated /metrics.

Preserve an older database-managed application secret

Skip this section if the existing installation already supplies ROOTTRACE_APP_SECRET or ROOTTRACE_APP_SECRET_FILE. Write that exact value to the target secret file through the site's approved secret manager. Do not generate a replacement.

Older Compose installations could generate the application secret in MongoDB. While the old API and MongoDB services are still healthy, run the helper from the signature-verified target release root:

scripts/migrate_compose_app_secret.sh \
  /srv/roottrace/current/deploy \
  "$PWD/deploy/secrets/app_secret"

Use the site's current deployment directory as the first argument. If ROOTTRACE_APP_SECRET_HOST_PATH points elsewhere in the target configuration, use that path as the second argument.

The helper uses the source deployment's deploy/.env, reads the existing value inside its MongoDB container, and writes a mode 0600 target file without printing the secret. It refuses to overwrite a file or export a database value when the old API already uses an external secret. Target startup also refuses an external value that does not match an existing database-managed secret.

Migrate an older Compose MongoDB volume

Current Compose installations use a bootstrap administrator and a separate least-privilege application user. The initialization script runs automatically only for an empty MongoDB volume. Before starting the new API against a volume created by an older stack:

  1. Preserve the existing root username and password as MONGO_ROOT_USER and MONGO_ROOT_PASSWORD in deploy/.env.
  2. Set a new hexadecimal MONGO_APP_PASSWORD, then recreate only Mongo so it receives the new environment:
   docker compose --env-file deploy/.env -f deploy/docker-compose.yml up -d mongo
  1. Run the shipped idempotent user-creation script. mongosh prompts for the root password; it is not placed on the command line:
   docker compose --env-file deploy/.env -f deploy/docker-compose.yml \
     exec mongo sh -c \
     'exec mongosh --quiet --username "$MONGO_INITDB_ROOT_USERNAME" \
       --authenticationDatabase admin \
       /docker-entrypoint-initdb.d/10-roottrace-user.js'

The script creates MONGO_APP_USER with readWrite and dbAdmin only on the RootTrace database. Rerunning it leaves an existing user unchanged.

Docker Compose upgrade

Run the upgrade from the new release directory, so the Compose definition, proxy config, Mongo init script, and scripts all move together as one reviewed release. Copy your existing deploy/.env, TLS material, and application secret into it, keeping .env and the secret at mode 0600 (the secret inside its 0700 directory).

Merge any new required setting from the target .env.example, then write the new digest into .env, because a shell export silently falls back to the old value on the next operator's run:

ROOTTRACE_IMAGE=public.ecr.aws/byteaffinity/roottrace/server@sha256:REPLACE_WITH_DIGEST

Then, from the target release root:

docker compose --env-file deploy/.env -f deploy/docker-compose.yml config -q
docker compose --env-file deploy/.env -f deploy/docker-compose.yml pull roottrace
docker compose --env-file deploy/.env -f deploy/docker-compose.yml \
  up -d --no-deps roottrace
docker compose --env-file deploy/.env -f deploy/docker-compose.yml ps
docker compose --env-file deploy/.env -f deploy/docker-compose.yml \
  logs --since 10m roottrace

--no-deps leaves MongoDB and nginx alone during the application step. If the release also changes nginx, MongoDB, networks, or volumes, stage that separately once the API is healthy.

Startup will be slower than a restart: the server runs pending migrations after MongoDB connects and before it reports ready. Do not kill it. A migration failure is loud and stops the API from serving at all.

Verify through the same HTTPS endpoint users reach:

curl --fail --silent --show-error \
  --cacert /path/to/roottrace-ca.pem \
  https://roottrace.example.com/readyz

curl --fail --silent --show-error \
  --cacert /path/to/roottrace-ca.pem \
  -H "Authorization: Bearer $ROOTTRACE_METRICS_BEARER_TOKEN" \
  https://roottrace.example.com/metrics

The readiness JSON must report ready: true, mongo: true, no pending schema migrations, and Elasticsearch healthy when it is configured.

Air-gapped upgrade

Use the release identifier and image references in the supplied bundle. No repository checkout or version.txt is needed.

  1. Confirm the transfer record identifies the signature-verified RootTrace deployment archive used as the builder source, including its version, checksum, signature-verification output, and image-verification output. Then verify the imported offline bundle checksum. The outer checksum detects transfer damage; it does not replace that signed-source record. images/IMAGE_REFS.txt records source digests but is not itself signature evidence.
  2. From the bundle's images directory, verify every archive:
   sha256sum -c SHA256SUMS

On macOS, use shasum -a 256 -c SHA256SUMS.

  1. Load server.tar and inspect IMAGE_REFS.txt for the source digest and exact offline tag restored by the archive:
   docker load -i images/server.tar
   cat images/IMAGE_REFS.txt
  1. If required, mirror that image into the site's private registry and record the resulting digest.
  2. Complete the normal backup and staging checks, then perform the Compose procedure above using the private-registry digest.

Keep the previous server archive and datastore backup inside the boundary until the upgrade observation window closes.

Schema migrations

There is no customer-run migration command.

  • Pending migrations execute automatically before traffic is served.
  • Applied migrations are recorded in MongoDB's schema_migrations collection.
  • A lease in system_locks with _id: "schema_migrations" prevents concurrent replicas from applying the same migration.
  • /readyz reports applied and pending migration state.
  • A failed migration aborts startup. Fix the cause or restore the pre-upgrade backup; do not delete the lock or ledger entry while another process may be running.
  • There are no down-migrations.

Operators with authorized database access can inspect the ledger:

db.schema_migrations.find().sort({_id: 1})

The recorded build identifier is diagnostic metadata baked into the released image. Customers do not supply a version file.

Rollback decision

First determine whether the target release applied a migration or wrote data that the previous release cannot read.

No migration or data incompatibility

Write the exact previous image digest back to ROOTTRACE_IMAGE in deploy/.env, then recreate the application workload with the unchanged configuration and datastores.

For Compose:

docker compose --env-file deploy/.env -f deploy/docker-compose.yml config -q
docker compose --env-file deploy/.env -f deploy/docker-compose.yml \
  up -d --no-deps roottrace

Migration or incompatible writes occurred

Stop every API replica. Restore the pre-upgrade MongoDB backup and any required external Elasticsearch snapshot, preserve the same application secret, then start the previous image digest. Do not run old and new images against the restored database at the same time.

The included restore helper is destructive and prompts before using mongorestore --drop. For Compose it stops the API first and restarts it only after a successful restore:

ARCHIVE=/secure/roottrace-backups/REPLACE_WITH_ARCHIVE
scripts/mongo_restore.sh "${ARCHIVE}"

Set ROOTTRACE_MONGO_URI only when restoring an external database reachable from the host. If a Compose restore fails, the helper deliberately leaves the API stopped; inspect the restore error before retrying or starting the service.

Restore into an isolated environment first whenever time permits. If a migration partially failed, treat the live database as incompatible until the release owner confirms otherwise.

Post-upgrade observation

Keep the change open through the agreed observation window. Watch:

  • API and migration errors;
  • login, MFA, SSO, and SCIM failures;
  • collector authentication and ingest lag;
  • notification retries and SMTP delivery;
  • MongoDB and optional Elasticsearch latency;
  • /readyz, authenticated /metrics, and ingress error rates;
  • audit export and hash-chain continuity.

Upgrade collectors separately through their signed package repository or immutable container image. The server upgrade does not modify customer hosts. The collector has no in-process updater.

Retain the target digest, verification output, backup/restore evidence, health results, migration state, approver, and rollback outcome as the upgrade record.