Sizing and high availability

Hardware targets and reference topologies for a self-hosted deployment: the API server, MongoDB, the optional Elasticsearch vector index, plus embeddings, load balancing, health probes, and backup/restore.

Just trying it out? 4 vCPU, 8 GB RAM, 20 GB SSD runs the whole Compose stack. Come back here when you are provisioning for real.

Tiers map to fleet size (monitored hosts and concurrent operators), not to plan names:

Assumptions throughout: SSD or NVMe everywhere (Elasticsearch on spinning disk is unsupported at these numbers), retention at tier defaults, and per-node sizes quoted at steady state with ~30% headroom for compaction and GC spikes. Provision for peak, not average. Heavier retention scales MongoDB storage roughly linearly.

A self-hosted LLM inference server is sized by whoever runs it; RootTrace only calls it (ROOTTRACE_LLM_OPENAI_BASE_URL).

1. API server

The API server is a stateless FastAPI/uvicorn process listening on 8090 (ROOTTRACE_MONGO_URI, ROOTTRACE_ELASTICSEARCH_HOST point it at the data tiers). Because it is stateless, you scale it horizontally: add replicas behind the load balancer (section 5). The launcher also manages a separate embedding worker process and, when enabled, a local model-service process. Those children share the pod or container CPU and memory budget.

Notes:

• Disk is scratch only (logs, temp, the embedding lock file at /tmp/roottrace_embedding_model.lock). No durable state lives on the API node. The Compose deployment uses a writable container filesystem plus a named VCS-cache volume. Custom orchestrator manifests should make only /tmp and the VCS cache writable.
• The RAM floor is driven by the embedding model. Loading all-mpnet-base-v2 (the default embedding_model) resident costs ~1–2 GB before request working set. If you set embedding_offline with a bundled model, budget the same. The managed model-service process remains in the API container or pod, so budget this memory on every replica.
• Replica count for HA: run at least 2 on Medium and 3 on Large so a node loss or rolling restart never drops the tier to zero. The shipped Compose file is a single-host topology; multi-host HA requires a site-managed deployment and load balancer.
• Set worker/uvicorn concurrency to leave room for the reserved embedding cores (below). A good starting point is workers = vCPU - embedding_reserved_cpu_cores.
• Set ROOTTRACE_RATE_LIMIT_BACKEND=mongo whenever more than one API worker or replica serves traffic. The default in-memory limiter is process-local and otherwise multiplies the effective limit.

2. Embedding CPU reservation

Semantic incident search embeds text with a CPU sentence-transformers model (all-mpnet-base-v2, with embedding_dimensions of 768). Embedding is CPU-heavy and bursty; if it competes freely with request handlers, p99 latency and readiness both suffer. RootTrace therefore lets you fence off cores for it.

Relevant settings (src/roottrace_settings.py, all overridable by env):

Guidance:

• Reserve ~25% of the replica's vCPU for embedding, minimum 1 core.
  • Small (4 vCPU): embedding_reserved_cpu_cores=1, embedding_cpu_threads=1.
  • Medium (8 vCPU): reserve 2, embedding_cpu_threads=2, embedding_worker_threads=2.
  • Large (16 vCPU): reserve 4, embedding_cpu_threads=4, embedding_worker_threads=2–4.
• Keep embedding_cpu_threads * embedding_worker_threads <= embedding_reserved_cpu_cores so torch never oversubscribes the fenced cores.
• Interop threads are pinned to 1 internally to avoid thread storms; do not fight that.
• The managed model service is loopback-only and launched with the API. It is not a supported remote-offload protocol. Large replicas still need local embedding CPU and memory; disable semantic search instead when a tier cannot reserve that capacity.
• If the model can't load, search runs in a deterministic hashed-fallback degraded mode: correct but lower quality. Watch logs for the "DEGRADED mode" warning; it usually means the model or its CPU budget is missing.

3. MongoDB

MongoDB stores telemetry, configuration, workspace/incident/audit data, embedding job state, and license data. It is the source of truth and must be durable and highly available; Elasticsearch is an optional, rebuildable search index over data that originates here.

Sizing

• RAM should cover the working set (hot indexes + frequently read docs). WiredTiger uses ~50% of RAM for its cache by default; the recommended RAM column keeps the working set resident.
• Size disk from measured telemetry ingest, retention, indexes, and audit volume. The table is a starting point, not a substitute for load testing.

Running an external replica set / sharded cluster / Atlas

RootTrace connects with a single ROOTTRACE_MONGO_URI and does not manage the database lifecycle, so any externally operated MongoDB works:

• Replica set (recommended default for HA): run a 3-member replica set (primary + 2 secondaries, or primary + secondary + arbiter for smaller footprints). Point RootTrace at all seeds: `` ROOTTRACE_MONGO_URI=mongodb://mongo-a:27017,mongo-b:27017,mongo-c:27017/roottrace?replicaSet=rs0 ` The driver handles primary failover automatically. Use w=majority for write durability across a node loss (append &w=majority` if not already the default).
• Sharded cluster (Large tier / very high metadata volume): connect to the mongos routers instead of mongod. Most deployments do not need sharding. Reach for it only when a single replica set can no longer hold the working set in RAM. RootTrace shards itself; see below.
• MongoDB Atlas: paste the Atlas SRV string directly: `` ROOTTRACE_MONGO_URI=mongodb+srv://user:pass@cluster0.example.mongodb.net/roottrace?retryWrites=true&w=majority `` Allowlist the API server egress IPs in Atlas, and prefer a region colocated with the API tier to keep round-trips low. Atlas handles replica-set management, backups, and failover for you.

Whatever the topology, keep MongoDB on a low-latency network to the API replicas (same VPC/subnet); metadata reads are on the request hot path.

Sharding, when you use it

In production, sharding is enabled and strict by default. Startup creates indexes and then shards every managed collection: auth, organization, collector, diagnostic, incident, recommendation, postmortem, integration, embedding reverse-mapping, application audit, and Linux audit.

Shard keys are chosen per collection so unique indexes stay valid:

• Global auth collections shard by a globally unique lookup field such as email or token_hash.
• Tenant-owned collections shard by organization_id where possible, so organization-scoped reads stay targeted rather than scatter-gather.
• High-volume tenant collections with no unique-index constraint use hashed organization_id keys and initial chunks, so the balancer can spread them.
• Non-system collections outside the managed list are inspected and sharded with a compatible fallback key.

Set ROOTTRACE_ENABLE_MONGO_SHARDING=false for single-node MongoDB. The shipped Compose file does exactly that.

4. Elasticsearch

Elasticsearch backs semantic incident search. It holds embedding documents and does the vector + keyword query work. It is set with ROOTTRACE_ELASTICSEARCH_HOST; leaving that empty cleanly disables search (and drops the ES readiness check; see section 6). Indices are created per workspace as roottrace_<organization_id> with a 768-dim dense_vector mapping. Elasticsearch is a rebuildable mirror of MongoDB (the system of record), so it does not need its own backup; see section 7.

Sizing

Rules of thumb:

• Heap = 50% of node RAM, never above ~30 GB (compressed-oops boundary). The rest is OS page cache, which vector/keyword search depends on heavily.
• Disk from first principles: daily_embedding_documents * measured_doc_bytes * (1 + replicas) * retention_days / node_count, then add at least 30% free space. Measure indexed document size with representative content.
• Dense vectors are RAM-hungry: 768-dim float vectors are ~3 KB each and are memory-mapped for HNSW search. Keep vector-bearing indices' working set in page cache; this is the main reason ES RAM outpaces Mongo.
• Replicas for HA: use number_of_replicas: 1 on Medium/Large (already reflected in the disk math) so any single data node can fail without data loss or search downtime. Small/single-node runs at 0 replicas, acceptable only because ES is rebuildable from source data.
• Dedicated master-eligible nodes (3, small) once you pass ~6 data nodes, to keep cluster state stable.

ILM and the retention model

Search retention follows the same workspace retention window used by other consumers. On-prem admins pick from the presets in src/roottrace_plans.py:

ONPREM_RETENTION_PRESET_DAYS = (30, 90, 180, 365, 730)   # days
UNLIMITED_RETENTION_SENTINEL = 0                          # stored value = "keep forever"
UNLIMITED_RETENTION_DAYS     = 36500                      # ~100y stand-in for date math

How it flows into Elasticsearch:

• RootTrace keeps one roottrace_<organization_id> index per workspace, attaches a lifecycle policy, stamps each document with expires_at, and excludes expired documents from searches.
• Unlimited retention is stored as the sentinel 0. Consumers translate it to 36500 days so date computations remain bounded. The sentinel must survive license and catalog refreshes.
• The application does not configure time-based rollover aliases. Operators with large vector indexes should test their own shard and snapshot policy against the exact supported Elasticsearch version before adding one.

The current lifecycle delete phase is based on the age of the whole workspace index, not each document. Because the application does not roll that index over, the policy can delete newer vector documents when an old index reaches its retention age. Search remains rebuildable from MongoDB, but this is a known availability defect. The shipped Compose default keeps ROOTTRACE_EMBEDDING_LIFECYCLE_ENABLED=false; leave it off until it is replaced with a document-expiry purge or proper rollover design. Perform a tested expires_at-based cleanup under an operator-controlled retention procedure. Search already excludes expired documents, but disabling ILM alone does not remove them from storage.

5. Multi-replica API behind the shipped nginx

The supported Compose proxy is deploy/nginx/reverse-proxy.conf. It provides server-authenticated TLS, includes commented optional client-certificate controls, and starts with this single Compose upstream:

upstream upstream_backend {
    least_conn;
    server roottrace:8090;
}

Compose is a single-host topology. For a site-managed multi-host deployment, adapt that shipped file and its certificate paths:

1. Add each replica as an upstream server. For replicas on other hosts: ``nginx upstream upstream_backend { least_conn; server api-1:8090 max_fails=3 fail_timeout=15s; server api-2:8090 max_fails=3 fail_timeout=15s; server api-3:8090 max_fails=3 fail_timeout=15s; keepalive 32; } ` least_conn suits the mixed short/long requests here better than round-robin. max_fails/fail_timeout eject an unhealthy replica automatically; keepalive reuses upstream connections (works with the existing proxy_http_version 1.1`).
2. Keep the API stateless. It already is: sessions are not held in-process, so any replica can serve any request. The real client IP reaches the app via the X-Forwarded-For / X-Real-IP headers nginx already sets (consumed by uvicorn --proxy-headers and get_client_ip), which the IP allowlist and audit logging depend on. Preserve those headers on every hop.
3. Health-gate the pool with /readyz (section 6). The shipped open-source nginx configuration provides passive upstream failure handling, not active readiness polling. Put active readiness checks in the orchestrator or load balancer. In Kubernetes, point the readiness probe at /readyz and let the platform remove unready endpoints before nginx balances across the Service.
4. Run nginx itself HA for true end-to-end availability (two proxy nodes behind a virtual IP / L4 load balancer), so the proxy is not a single point of failure. Each proxy carries the same server/client-certificate config derived from deploy/nginx/reverse-proxy.conf.

Preserve HSTS, CSP, security headers, timeouts, and access/error logging when adapting the shipped configuration. If client certificates are required, enable and test its optional mTLS controls. The separate nginx/reverse-proxy.conf file is a standalone localhost example with different certificate paths; Compose does not mount it.

6. Orchestration: readiness and metrics

Both endpoints are dependency-free and cheap by design (src/roottrace_metrics.py), so you can scrape/probe them on tight intervals.

/readyz: readiness probe

• Checks MongoDB connectivity, and Elasticsearch connectivity only when elasticsearch_host is set (an empty host disables search and its check; a readiness gate never fails just because search is intentionally off).
• Checks that required schema migrations have completed.
• Returns HTTP 200 with {"ready": true, "checks": {...}} when all wired dependencies are up, or HTTP 503 with the same shape (ready: false) when any is down.
• Wire it as the container/pod readiness probe and the nginx/LB upstream health check. A replica returning 503 is pulled from rotation until Mongo (and ES, if configured) recover, which is exactly the behavior you want during a database failover.
• For a liveness probe, use /healthz/. Do not tie liveness to database health, or a transient DB blip will cause needless container restarts.

Example (Kubernetes):

readinessProbe:
  httpGet: { path: /readyz, port: 8090 }
  periodSeconds: 10
  failureThreshold: 3
livenessProbe:
  httpGet: { path: /healthz/, port: 8090 }
  periodSeconds: 20

/metrics: Prometheus scrape

• Prometheus text exposition, per-worker (each replica reports its own counters/gauges). Scrape every replica and aggregate in Prometheus.
• Set a high-entropy ROOTTRACE_METRICS_BEARER_TOKEN on every replica. Outside development, an unset token disables the endpoint with HTTP 404; a missing or incorrect bearer token returns HTTP 401.
• Exposes at least: roottrace_build_info, roottrace_mongo_up, roottrace_elasticsearch_configured, and roottrace_metrics_scrapes_total.
• Alert on roottrace_mongo_up == 0, on replicas dropping out of /readyz, and on ES disk watermark from your Elasticsearch exporter.

Prometheus example:

scrape_configs:
  - job_name: roottrace
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/secrets/roottrace-metrics-token
    static_configs:
      - targets: ["roottrace-1:8090", "roottrace-2:8090"]

Keep the token in the orchestrator's secret store, not in a checked-in Prometheus configuration file.

7. Backup and restore

Back up the source of truth (MongoDB) on a schedule; Elasticsearch can be rebuilt from it, but snapshotting ES too shortens recovery time.

MongoDB

• Use the shipped helper scripts/mongo_backup.sh (mongodump-based point-in-time export) on a cron schedule; keep encrypted copies off-box. ``sh scripts/mongo_backup.sh ARCHIVE=/secure/roottrace-backups/REPLACE_WITH_ARCHIVE scripts/mongo_restore.sh "${ARCHIVE}" ``
• On a replica set, dump from a secondary (readPreference=secondary) to avoid loading the primary.
• On Atlas, use Atlas continuous/cloud backups (PITR) instead of scripting your own; test restores into a scratch cluster regularly.
• Test restores, not just backups. Restore into a throwaway environment, point a staging API at it, and confirm /readyz is green.

Elasticsearch

• Elasticsearch is a rebuildable mirror of MongoDB, so a dedicated ES backup is optional. If you want faster recovery than a full re-index, use the native snapshot/restore API into a registered repository (shared filesystem or S3-compatible object store) on a schedule aligned to your RPO.
• If ES is lost entirely, you can rebuild the search index from MongoDB by re-running embedding/backfill (embedding_job_worker_enabled), at the cost of reindex time; snapshots exist to avoid that wait.

Recovery order

1. Restore/confirm MongoDB primary; /readyz will still be 503 until it is reachable.
2. Restore or reindex Elasticsearch; once elasticsearch_host responds, /readyz includes the ES check again and flips to 200.
3. Bring API replicas back into the nginx/LB pool as each returns 200 on /readyz.

Quick reference (recommended tier)

Numbers are steady-state recommendations with ~30% headroom; validate against your own ingest rate, retention window, and query concurrency before rollout.