Change tracking: what changed before it broke

Most outages follow a change. RootTrace can tell you which one, but only if it knows what you deployed and where the code lives.

This is optional and off until configured. It works as a three-step ladder, and each step is useful without the next: you can stop after step 1 and still get value.

1. Bind versions to commits

The cheapest step. Either set ROOTTRACE_APM_COMMIT_SHA where the agent runs (GITHUB_SHA is picked up automatically on GitHub Actions), or post a deploy event from CI using the collector token the fleet already has:

ROOTTRACE_API_URL already ends in /api (for example https://roottrace.example.com/api), so the path below does not repeat it:

curl -X POST "$ROOTTRACE_API_URL/apm/deploys" \
  -H "Authorization: collector $ROOTTRACE_COLLECTOR_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"service\": \"api\", \"version\": \"$VERSION\", \
       \"commit_sha\": \"$GITHUB_SHA\", \"deployed_by\": \"$GITHUB_ACTOR\"}"

With versions flowing, deploys become vertical markers on latency and throughput charts, and RootTrace can compare the window before a deploy with the window after it.

2. Link a repository

Settings → Change Tracking. RootTrace generates an Ed25519 deploy key; add the public half to the repository read-only. The private half is AES-GCM encrypted at rest and never leaves the server.

GitHub, GitLab, Gitea, and any plain SSH remote work with deploy keys. AWS CodeCommit works too: paste the console SSH URL (ssh://git-codecommit.<region>.amazonaws.com/v1/repos/<name>) and RootTrace generates an RSA key instead, because CodeCommit's IAM SSH auth is RSA-only. Upload it to a read-only IAM user, then save the returned SSH key id on the link. FIPS endpoints are recognized.

Which repository gets diffed for a service

In order: a link whose optional service list names the APM service wins, then a link literally named after the service, then the oldest link with no service list. Single-repo workspaces need none of this. Multi-repo workspaces should scope or name their links.

What the server actually does

Fetches are bare, blobless (--filter=blob:none), and outbound-only: the collector's read-only posture applies here too. git and ssh must be present in the API image; the default Dockerfile installs both.

Repository hosts are resolved and checked before every network Git operation. Public addresses are allowed by default. Private, loopback, link-local, and other non-public addresses require an explicit CIDR in ROOTTRACE_OUTBOUND_ALLOWED_PRIVATE_CIDRS, and SSH is pinned to the checked address so a second DNS lookup cannot redirect the connection.

Mirror cache

Mirrors live under ROOTTRACE_VCS_CACHE_DIR (default /var/tmp/roottrace-vcs) and manage their own disk use:

Everything in the cache is rebuildable. An evicted mirror re-clones on the next finding that needs it.

3. Read the finding

When a deploy regresses, the issue's What changed panel shows:

• the version pair,
• the commits in the deploy,
• the suspect commits whose changed files intersect new error stack frames, and
• any failing host checks on the service's hosts in the same window.

That last line is the one that earns the feature. A deploy-shaped symptom with a disk-shaped cause points at the disk, not the release.

Everything in the panel is deterministic evidence. Where a step has no data, the panel says exactly what to enable rather than guessing.

Automatic regression detection

Once versions flow, RootTrace compares each new version against the one before it without being asked. Comparisons use equal windows either side of the deploy (60 minutes of baseline, a 5-minute warm-up skipped, up to 60 minutes of candidate) and judge only groups with at least 100 requests and 5 minute-buckets on both sides.

Verdicts are deliberately dual-threshold, so noise does not page anyone:

• Latency regresses at ≥30% relative and ≥100 ms absolute.
• Error rate regresses at ≥50% relative and ≥1 point absolute and ≥10 failed candidate requests.

Each flagged group carries attribution: the span type whose milliseconds per request moved most, classified as "more calls per request (N+1?)" versus "slower calls".

The same computation runs server-side on the issue-monitor cadence and opens one apm.regression issue per (service, version), titled Possible faulty deployment: <service> <version>, through the normal issue pipeline (PagerDuty on first open, stale-sweep auto-resolve). Turn it off per workspace with the apm_regression_alerts_enabled setting.

The read API behind the dashboard panel is GET /api/apm/regressions, documented in the APM read API reference (apm/READ-API.md in the source release, and on the RootTrace documentation site).