What RootTrace is
RootTrace is a read-only diagnostics, incident memory, and AI postmortem platform. It runs as a hosted service or a self-hosted Docker Compose deployment.
It is not a replacement for your monitoring stack, and it does not change production. It sits between monitoring and incident workflow, and answers the questions an engineer actually asks at 3 a.m.:
- What is broken?
- What changed?
- Has this happened before?
- What systems are affected?
- What should we check next?
- What should go into the postmortem?
Existing tools answer the first question well. RootTrace exists for the other five.
The shape of an incident
Every part of the product sits somewhere on one arc, from a machine noticing something to a human writing it down afterwards.
1. Something is collected. A read-only collector runs on each host and reports disk, CPU, memory, network, systemd, container, and service health. Applications report their own latency, errors, and traces through the APM SDKs. Both are outbound-only. Neither accepts inbound commands.
2. Failures become one issue. Related failing checks group by service rather than arriving as fifty separate alerts. An issue carries evidence, a timeline, severity, and the hosts and services it touches.
3. The issue explains itself. The issue detail view holds evidence, a timeline, similar past issues, recommended next checks, an AI summary, and an audit trail. Change tracking adds the version pair, the commits in the deploy, and the suspect commits whose changed files intersect new error stack frames. A deploy-shaped symptom with a disk-shaped cause then points at the disk, not the release.
4. Memory makes it cheaper the second time. Similar-incident search is vector-backed and scoped to your workspace. "Has this happened before" stops being a question someone has to remember the answer to.
5. A human closes it out. Incidents link issues, track impact, and draft a postmortem you can export as Markdown or PDF. Recommendations are suggestions for a person to approve. AI output never executes commands.
What AI is allowed to do
This boundary is the product, so it is worth stating flatly.
- Diagnostic payloads are redacted before storage and again before any AI use.
- AI output always carries its evidence, a confidence value, and either a suggested next action or an explicit "not enough data".
- AI output never executes anything.
- Remediation is off by default and stays human-approved.
The security model covers the rest: tenant isolation, token handling, and the network boundaries around it all.
Architecture
| Piece | What it is |
|---|---|
| Dashboard | Flutter, bundled with the server image |
| API | FastAPI, stateless, horizontally scalable |
| System of record | MongoDB: every tenant record carries organization_id |
| Vector mirror | Elasticsearch, optional; rebuildable from MongoDB |
| Auth | ML-DSA-65 post-quantum JWTs by default, keys held in MongoDB |
| Agents | Read-only collector, plus in-process APM SDKs |
Elasticsearch is used only as a workspace-scoped vector mirror, in indexes named roottrace_<organization_id>, with MongoDB holding the reverse mapping. Leave ROOTTRACE_ELASTICSEARCH_HOST empty and RootTrace runs without semantic search; nothing else changes.
Availability
Every check result is folded into a durable state history, so RootTrace can answer how often something breaks, for how long, and whether it always breaks at the same hour, not just whether it is broken now.
Availability is reported as three numbers rather than one, each labelled with what it assumes: the raw figure, the figure over measured time, and the figure with declared maintenance and upstream host outages removed. Time when no collector was reporting is recorded as not measured, never counted as uptime. An availability target can also be set as an SLO, which is what lets a host check carry an error budget.
Profiling
Tracing stops where instrumentation stops. When a request is slow inside your own code, a trace shows one span and no explanation. Work that belongs to no request at all, like garbage collection or a background thread, never appears in a trace.
Continuous profiling samples what the process is actually executing, so the answer is a function name. RootTrace ranks functions by self time (the time inside the function itself rather than in what it called) and translates CPU time into vCPU-hours per month, which is the literal answer to "why are we paying for this many instances".
The part worth having is the deploy diff. RootTrace already knows which deploy preceded an issue and which commits it carried; open the issue and it also names the functions whose share of the profile grew afterwards. That extends "what changed" from commits to code.
Profiling is off until you enable it per service in the dashboard. The SDKs ask the API what to do and start nothing on their own, so turning it off during the incident it is causing is a toggle, not a redeploy.
Maintenance windows
Settings → Maintenance Windows. While a window is active (optionally scoped to specific environments), brand-new issues are suppressed and counted against the window. Already-open issues keep updating, and a condition still failing when the window closes opens an issue normally on the next evaluation. Collector heartbeat pages are suppressed too, so planned reboots stop paging people.
A window also removes its time from the adjusted availability figure and from any availability error budget. That exclusion is computed when a report is read, so a window recorded after the work (including one that has already ended) still corrects the report, within your retention window.
Self-hosted licensing
Licenses are bought through the /self-hosted portal in the dashboard, and each one is bound to an installation id that must match the ROOTTRACE_INSTALLATION_ID in your deployment. Keys appear only in the portal: the activation email deliberately carries no token.
A license carries fixed counts, not a meter: hosts, APM services, users, integrations, and a retention ceiling. Both capacity axes matter because one of them describes a fleet you run and the other describes a system you deploy into infrastructure you do not run. Nothing measures volume, so no figure in your deployment feeds a bill. Licenses issued before APM services became a priced axis carry no services claim and stay uncapped on it.
Verification is local, against a public key compiled into the server image, so your install needs no AWS identity, no license-key variable, and no mounted key file. It never phones home, including to renew: renewals reissue the license automatically with a 14-day grace window on top of the period end, but an install that passes its displayed expiry simply expires. Renew before that date.
Where to go next
- Standing it up: sizing, then install, or air-gapped install
- Getting data in: collector, APM SDKs, Linux audit
- Reporting on reliability: availability, change tracking
- Asking your own questions: dashboards and the metric explorer, database queries, telemetry volume
- Satisfying an auditor: security model, hardening guide
- Embedding RootTrace in your own product: OEM integration