On-prem hardening guide

A CIS-style baseline for an on-premises or air-gapped deployment. Each control names the setting that enforces it, so the sections map onto a CIS Benchmark or STIG checklist.

The assumed threat model: no outbound egress from the application tier, and mutual-TLS client authentication at the edge. If yours is looser, the controls still apply: the recommended values are just stricter than you need.

Set variables in deploy/.env (Compose maps them into the container) and restart the API. There is a verification checklist at the end.

1. TLS termination

RootTrace supports two TLS topologies. Pick one; do not run cleartext HTTP on any listener reachable outside localhost.

1.1 TLS at the shipped Compose proxy

The supported Compose stack mounts deploy/nginx/reverse-proxy.conf. It terminates TLS before traffic reaches the API and enables server authentication by default:

Operator responsibilities:

• Install the server keypair under deploy/certs/ and keep the host private key mode 0600.
• For mutual TLS, add ca.cert.pem and ca.crl.pem to that directory, uncomment the four ssl_client_* directives and the $ssl_client_verify gate in the shipped proxy, then test valid, expired, revoked, and missing client certificates. Keep the CRL current.
• Client-certificate verification is a transport gate, not a RootTrace role or user allowlist. Keep application authentication and authorization enabled.
• The proxy forwards X-Forwarded-For and X-Real-IP. Keep ROOTTRACE_FORWARDED_ALLOW_IPS and ROOTTRACE_TRUSTED_PROXY_CIDRS aligned with ROOTTRACE_NGINX_IP.
• The Compose proxy uses the nginx image's normal access/error logs. Configure the site's container log driver or log collector to retain and forward them; nginx/nginx.conf is not mounted by the Compose stack.

Keep the app's own TLS (§1.2) disabled in this topology. Compose exposes only nginx and keeps the API on its private networks.

The repository also carries nginx/reverse-proxy.conf, a standalone mTLS reference for a proxy on the same host as the API. It uses different certificate paths and a localhost upstream, so translate both deliberately rather than copying it over the shipped config.

1.2 Direct TLS termination in the app (uvicorn)

For installs without a reverse proxy, terminate TLS in the app itself. main.py passes these straight to uvicorn's --ssl-certfile / --ssl-keyfile when both are set:

Note: direct termination does not provide mutual-TLS client-cert enforcement or a CRL gate. For an air-gapped, high-assurance install, prefer §1.1. If you must use §1.2, enforce network reachability and client authentication by other means (IP allowlisting §9, service-account scoping §10).

2. Disable all egress (no call-home)

An air-gapped install must make no outbound connection. Configure and verify each path explicitly.

The air_gapped profile rejects an enabled vendor email relay and online embedding mode. It does not classify every arbitrary URL as internal or external. Review the effective configuration and enforce the boundary below.

On-prem password flows never call HaveIBeenPwned, regardless of ROOTTRACE_BREACHED_PASSWORD_CHECK. That variable controls only hosted-cloud screening and is not an on-prem egress control.

The table above covers the server. The agents that report into it have their own destination, and it is the one an air-gapped install is most likely to get wrong:

Belt-and-braces: enforce egress denial at the network layer too (default-deny egress firewall / no default route out of the enclave). Application settings are the second line of defense, not the only one.

3. SMTP-only email

With the cloud relay off (§2), outbound notification/auth email must go through your own SMTP server inside the enclave. Both the global sending switch and a configured SMTP host are required.

Do not configure SendGrid (ROOTTRACE_SENDGRID_API_KEY) on an air-gapped install: that is a cloud egress path. Always require TLS to the relay (ROOTTRACE_SMTP_STARTTLS=true or ROOTTRACE_SMTP_SSL=true); never send credentials over cleartext SMTP.

4. FIPS posture

For a FIPS-validated cryptographic posture:

Run the host on a FIPS-validated OpenSSL module (OS-level FIPS mode enabled) so that the Python crypto stack and TLS both operate within the validated boundary. ROOTTRACE_FIPS_MODE=true verifies that boundary instead of merely advertising it. A bad provider or algorithm stops the service.

Outside that validated-provider mode, leave JWT signing at the default ML-DSA-65. It is the post-quantum signature scheme standardized by NIST FIPS 204 and registered for JOSE by RFC 9964. Account for its larger token signatures when setting proxy and gateway header limits.

5. Master-secret externalization (KMS / Vault)

The master application secret must never be baked into an image or committed. Source it externally:

Recommended pattern:

• Store the secret in Vault (or your KMS) and project it into the container as a mounted file, then point ROOTTRACE_APP_SECRET_FILE at that path. The value never appears in the environment, image layers, or process listing.
• For a custom orchestrator projection, restrict the mounted file to mode 0400, owned by the app user.
• Rotate via your secret manager; the app derives JWT-key encryption material from the app secret (app_secrets), so plan a rotation window.
• Prefer ROOTTRACE_APP_SECRET_FILE over ROOTTRACE_APP_SECRET: an env-var value is visible to anything that can read /proc/<pid>/environ.

The supported Compose stack copies ROOTTRACE_APP_SECRET_HOST_PATH through a network-disabled initializer and mounts the private Docker-volume copy at /run/secrets/app_secret. Keep the host source mode 0600 inside the mode 0700 deploy/secrets directory. Do not make it world-readable, and do not set a container-only path that is not mounted.

6. STIG session and access controls

These all ship in the product. Enable them for a hardened install. The SRG identifiers are DISA Application Security Requirements Guide references.

The per-org settings (max_concurrent_sessions, session_idle_timeout_minutes, login_lockout_*, account_inactivity_disable_days, logon_banner_*) are set through the Flutter workspace security settings and override/refine the global env defaults. The page also includes security_profile and privileged-MFA enforcement.

How the two session controls interact

A session is terminated at exactly two points: when it exceeds the idle window, and when a new sign-in pushes the account past its concurrent cap. Nothing else ends an active session.

The idle timeout is enforced when a token is next presented, which means a device someone simply stopped using is never seen again and its session would otherwise sit there indefinitely. Those sessions are therefore swept at the next sign-in and do not count toward the concurrent cap. Otherwise three laptops closed last week would fill a 3-session cap and signing in on a fourth device would evict the phone in your hand. The cap bounds concurrent sessions; a session inactivity already ended is not one.

A session with no recorded activity at all is never swept on that basis: absent metadata is unknown age, not proven idleness.

When either control ends a session, the reason is recorded and returned on the next request from that device ("Session terminated due to inactivity", or the concurrent-session-limit message) rather than a generic token rejection. Both are also audit events. Eviction always removes the least-recently-used session, never the one signing in, so a user cannot be locked out by the cap.

FIPS provider state, JWT/WebAuthn policy, token lifetimes, secrets and metrics authentication, rate limits, audit chaining, request limits, offline embeddings, email paths, Elasticsearch TLS, proxy/CDN trust, origins, egress exceptions, collector-token lifetime, and direct API TLS are deployment boundaries. Workspace admins can verify their effective values in the same Flutter page, but cannot mutate process-wide runtime state for neighboring workspaces. Platform admins can select the deployment security profile and JWT signing algorithm in Flutter without restarting the API. The profile is accepted only when the running process configuration satisfies it. JWT changes rotate the database signing key immediately and are observed by every API replica.

6.1 Workspace-mandated SSO (sso_required): per-workspace enforcement

Setting a workspace's login mode to sso_required (workspace login settings; requires at least one enabled SSO method: OIDC, SAML, or LDAP) is enforced at two layers:

• At sign-in: password login, password reset, and password-based invite acceptance are refused for accounts whose every workspace mandates SSO (account-wide block, kept as defense in depth).
• At workspace access: every token records how the session authenticated (an RFC 8176 amr claim: pwd, otp, mfa, or federated for OIDC/SAML/LDAP sign-ins; refresh carries it forward unchanged). Every org-scoped request re-checks it: a session that did not come through an identity provider gets 403 on an sso_required workspace, even if the account also belongs to password-capable workspaces. This closes the mixed-workspace gap where a password session could previously reach an SSO-mandated workspace.

Tokens minted before the amr claim existed are treated as password-only, so sso_required workspaces reject them immediately; affected SSO users simply sign in again through their IdP. Machine credentials (service-account tokens, collector tokens, SCIM) are exempt: they never authenticate through an IdP and their access is bounded by their own scopes.

6.2 MFA recovery (backup) codes

Enrolling the account's first MFA factor (TOTP, email, or WebAuthn) issues 10 single-use recovery codes, returned in plaintext exactly once; only HMAC hashes are stored. A recovery code can complete an MFA challenge in place of the TOTP/email/passkey proof (same attempt limits), consuming it permanently. POST /api/auth/mfa/backup-codes/regenerate (fresh authenticator, email, or passkey proof required; an existing recovery code also works for a passkey-only account) invalidates the old set and mints a new one; disabling the last MFA factor clears the stored codes. Every generation, consumption (with codes_remaining), and regeneration is audit-logged.

7. Password policy

Passwords are hashed with PBKDF2-SHA512 and a per-password salt (see docs/security.md). On-prem breached-password screening is disabled in code; use complexity, length, MFA, lockout, and password history as the local controls.

8. Audit hash-chain and SIEM/syslog export

RootTrace maintains a tamper-evident audit trail and can forward it to your SIEM.

Forward audit events to a SIEM that is independent of the RootTrace host so the chain and the copy cannot be altered together. Combine with the nginx access log (§1.1) for full request-level attribution.

9. IP allowlisting

The roottrace_ip_allowlist module enforces per-organization CIDR/host allowlists (GET/mutation endpoints under /api/organizations).

• Configure each organization's allowlist with the CIDR ranges (or bare host addresses) that legitimately reach the workspace. A non-empty allowlist causes enforce_ip_allowlist to return 403 for any client IP outside it. The check runs again after tenant resolution, so routes that infer a workspace from a resource ID or default membership cannot bypass it.
• The same policy covers user JWTs, service-account tokens, SCIM requests, and collector/OTLP ingest.
• Allowlist database, import, or enforcement failures return 503. The control fails closed instead of silently widening access.
• Default is deliberately permissive for backward compatibility: an empty or absent allowlist means allow-all. For a hardened install, set a non-empty allowlist on every organization.
• This relies on the true client IP, so run behind the reverse proxy with --proxy-headers (§1.1) or the allowlist will see the proxy's address.
• Treat IP allowlisting as defense-in-depth behind mTLS (§1.1) and network segmentation, not as the sole access gate.

Outbound destination allowlisting

Notification webhooks and repository SSH links are server-side network clients. Their destinations are public-only by default. Private, loopback, link-local, reserved, and metadata addresses are rejected, including addresses reached after an HTTP redirect.

If an on-prem integration must reach an internal service, list only its service subnet:

ROOTTRACE_OUTBOUND_ALLOWED_PRIVATE_CIDRS=10.40.8.0/24

This setting does not enable plaintext HTTP. Webhooks additionally require ROOTTRACE_NOTIFICATION_ALLOW_INSECURE_WEBHOOKS=true for HTTP or unverified TLS. Repository SSH resolves before each operation and pins the checked address to close the DNS-rebinding window.

10. Service-account scoping

Machine-to-machine access uses scoped service-account tokens (roottrace_service_accounts) rather than user credentials.

• Tokens are org-scoped and explicitly scoped to a capability set: read, write, scim (VALID_SERVICE_ACCOUNT_SCOPES). The default is read only. Grant write/scim only when required.
• Grant least privilege: a telemetry-ingest integration needs write; a read-only exporter needs only read; reserve scim for the identity provisioning integration that drives roottrace_scim (/scim/v2).
• Set an expiry (expires_in_days) on every token; 0/omitted means non-expiring. Avoid non-expiring tokens in a hardened install.
• Tokens are hashed at rest (HMAC), returned once, and independently revocable. Rotate on a schedule and revoke on personnel/integration change.
• An unknown scope string is rejected (422) rather than silently ignored, so typos fail loudly.

11. Collector-token TTL and rotation

Collector enrollment tokens ingest telemetry from agents; bound their lifetime.

Operational guidance:

• Collector tokens are scoped to one organization and one environment, returned once, hashed at rest, and revocable (docs/security.md).
• Collectors are outbound-only and never accept inbound command execution; default mode is read_only.
• Rotate on the TTL cadence: issue the new token, let the overlap window elapse while agents pick it up, then revoke the old one.

12. Health and metrics endpoints

The roottrace_metrics module exposes /metrics (scrape) and /readyz (readiness/health) with no third-party dependencies. Set a high-entropy ROOTTRACE_METRICS_BEARER_TOKEN; Prometheus must send it as a bearer token. Outside development, /metrics returns 404 when the setting is empty and 401 for a missing or incorrect token.

/readyz and /healthz/ stay unauthenticated for orchestrator probes. Restrict them to the monitoring subnet at the reverse proxy or firewall.

The API also rejects oversized bodies before parsing. Keep ROOTTRACE_API_MAX_REQUEST_BODY_BYTES at its 16 MiB default unless a documented API payload requires more. The two NDJSON collector streams use ROOTTRACE_API_MAX_STREAM_BODY_BYTES (64 MiB by default).

Quick verification checklist

• [ ] Profile: ROOTTRACE_SECURITY_PROFILE=regulated (or air_gapped) and startup validation passes.
• [ ] TLS: shipped Compose proxy (deploy/nginx/reverse-proxy.conf), with its optional mTLS controls enabled when required, or ROOTTRACE_SSL_CERTFILE/ROOTTRACE_SSL_KEYFILE set; no cleartext listener.
• [ ] Egress: private destinations blocked or narrowly listed in ROOTTRACE_OUTBOUND_ALLOWED_PRIVATE_CIDRS; ROOTTRACE_ON_PREM_EMAIL_RELAY_ENABLED=false, ROOTTRACE_EMBEDDING_OFFLINE=true, no ROOTTRACE_CLAUDE_API_KEY/ ANTHROPIC_API_KEY; egress firewall default-deny.
• [ ] Agents: every collector and every instrumented service points at this deployment's own /api base. The SDKs need this set explicitly: their default is RootTrace Cloud.
• [ ] Email: either disabled, or ROOTTRACE_EMAIL_SENDING_ENABLED=true with ROOTTRACE_SMTP_HOST set and TLS enabled; SendGrid unset.
• [ ] FIPS: ROOTTRACE_FIPS_MODE=true, ROOTTRACE_JWT_ALGORITHM=RS256 or ES256, and the validated OpenSSL FIPS provider is active.
• [ ] Secret: no inline value; Compose host source 0600 inside a 0700 directory, or custom-orchestrator projection 0400 owned by the app user.
• [ ] STIG: ROOTTRACE_REQUIRE_PRIVILEGED_MFA=true, idle timeout, lockout, concurrent-session cap, inactivity disable, logon banner all set.
• [ ] Audit: hash chain on, syslog export to SIEM, retention ≥365d, /api/audit/export reachable to auditors.
• [ ] Access: non-empty IP allowlist per org; scoped, expiring service-account tokens; finite collector-token TTL with rotation overlap.
• [ ] Metrics: ROOTTRACE_METRICS_BEARER_TOKEN stored as a secret; /metrics, /readyz, and /healthz/ restricted to the monitoring subnet.