RootTrace

RootTrace APM read API

The endpoints the dashboard calls to read back what the APM wrappers sent, including the profiling endpoints under /api/profiles. Authentication is the normal dashboard JWT, not a collector token.

Organization and environment resolve the standard way: organization_id query param or X-Organization-Id header, environment_id query param or X-Environment-Id header. time_range accepts the platform's standard keys (15m 1h 6h 12h 24h 7d 30d 90d 1y) and defaults to 24h.

Every data endpoint (metrics, metrics/series, transactions, transactions/series, errors, traces, dependencies, graph, graph/impact) also accepts an optional deployment query param filtering to one Kubernetes deployment.

Percentiles, in one place

Percentile fields on these endpoints come from the log2 duration histograms described in the protocol. Range-level percentiles aggregate every histogram in the range; series percentiles merge the histograms inside each downsampled bucket, so a chart's percentiles follow its own resolution. All of them are null when nothing in scope reported buckets. avg and max are unaffected either way.

Profiling

Five endpoints reading what the profiling protocol stored. All accept either a preset time_range or an explicit start/end.

GET /api/profiles/services

Which services have profiles and of what type, newest first.

{"services": [{"service": "checkout-api", "profile_type": "cpu",
               "last_seen_at": "2026-07-30T12:05:00Z", "buckets": 288}]}

GET /api/profiles/flamegraph

service, profile_type (cpu/wall/heap), optional deployment, optional limit (100–20000, default 4000).

Returns the merged call tree. Each node is {name, value, self, children} where value is total time (the function and everything it called) and self is what stayed in the function itself. A wide bar is ambiguous by construction, so both numbers are always present. Children are sorted widest-first.

{"time_range": "24h", "value_unit": "nanoseconds", "total": 10000000000.0,
 "stacks_total": 12400, "stacks_shown": 4000, "truncated": true,
 "cost": {"cpu_seconds": 10.0, "average_vcpus": 2.4, "vcpu_hours_per_month": 1752.0},
 "tree": {"name": "all", "value": 10000000000.0, "self": 0.0, "children": [...]}}

truncated is explicit: silent truncation would read as "this is the whole profile". cost is null for anything but a CPU profile. Wall clock counts time spent waiting, so billing it as vCPUs would charge for blocking on a socket, and a heap profile is bytes rather than a duration.

GET /api/profiles/top-functions

limit 1–200, default 25. Ranked by self time, because ranking by total would put main at the top of every profile ever taken.

{"functions": [{"function": "encode_json", "total": 8000000000.0,
                "self": 8000000000.0, "self_pct": 80.0,
                "cost": {"vcpu_hours_per_month": 1401.0}}]}

A function appearing several times in one stack (recursion) contributes that stack's time to total once, not once per frame. Counting each occurrence would let a recursive function report more total time than the profile contains.

GET /api/profiles/diff

service, plus before_start/before_end and after_start/after_end.

{"regressed": [{"function": "serialise", "before_pct": 10.0,
                "after_pct": 60.0, "delta_pct": 50.0}],
 "improved": [...]}

Both sides are normalised to shares of their own total before comparing. Comparing raw values would mark every function "worse" whenever the second window simply served more traffic: the failure that makes naive profile diffs useless.

GET /api/profiles/for-issue/{issue_id}

window_minutes 5–1440, default 30. Derives the service and window from the issue and diffs it against the same length of time before it, trying cpu then wall.

{"available": true, "service": "checkout-api", "profile_type": "cpu",
 "before_window": {"start": "...", "end": "..."},
 "after_window": {"start": "...", "end": "..."},
 "regressed": [{"function": "serialise", "delta_pct": 50.0}]}

When there is nothing to show, available is false with a reason of issue_has_no_service, no_profiles_in_window, or service_not_profiled. The last two are deliberately distinct: "profiled, but not over this window" is worth a note in the UI, while "never profiled" would put a nag about an unused feature on every issue in the workspace.

GET /api/apm/summary

GET /api/apm/summary?time_range=24h
{
  "time_range": "24h",
  "totals": {"services": 3, "metric_names": 27, "data_points": 15420},
  "services": [
    {
      "id": "665f...",
      "name": "checkout-api",
      "language": "python",
      "hostnames": ["web-1", "web-2"],
      "runtime": {"language_version": "3.12.1"},
      "interval_seconds": 30,
      "first_seen_at": "...",
      "last_seen_at": "...",
      "status": "reporting",
      "metric_count": 12,
      "deployments": [
        {"name": "checkout-api", "namespace": "prod",
         "pods": ["checkout-api-7d9f8b6c5d-x2k4p"], "last_seen_at": "..."}
      ]
    }
  ]
}

deployments is empty for services not reporting Kubernetes context.

status is reporting when last_seen_at is within max(3 * interval_seconds, 300) seconds, else stale.

GET /api/apm/metrics

GET /api/apm/metrics?time_range=24h&service_id=<id>

The dynamic metric catalog lists every metric name seen in the range:

{
  "metrics": [
    {
      "name": "http.request.duration",
      "kind": "timer",
      "unit": "ms",
      "service_id": "665f...",
      "service_name": "checkout-api",
      "count": 240000,
      "sum": 9800000.0,
      "min": 1.0,
      "max": 2500.0,
      "avg": 40.8,
      "p50": 32.0, "p90": 78.8, "p95": 93.7, "p99": 264.0,
      "latest_value": 38.2,
      "latest_at": "...",
      "tag_keys": ["endpoint=/checkout", "endpoint=/cart"]
    }
  ]
}

p50/p90/p95/p99 aggregate the histogram buckets over the whole range and are null for metrics with no histogram data (any non-timer metric, and timers from wrappers not sending buckets).

GET /api/apm/metrics/series

GET /api/apm/metrics/series?service_id=<id>&name=<metric>&time_range=24h&tags_key=<optional>

Time series for one metric, downsampled to a bucket size fitting the range (minute up to 6h, then 5m/10m/1h/6h/1d as the range grows):

{
  "name": "http.request.duration",
  "kind": "timer",
  "unit": "ms",
  "service_id": "665f...",
  "bucket_seconds": 600,
  "series": [
    {"bucket_at": "...", "count": 1200, "sum": 48000.0, "min": 2.0,
     "max": 300.0, "avg": 40.0, "p50": 32.0, "p99": 264.0,
     "last_value": 38.2}
  ]
}

Omitting tags_key aggregates across all tag combinations.

Each bucket's p50/p99 merge the histograms of every rollup minute downsampled into it, so percentiles follow the chart's own resolution rather than being computed once over the range. Both are null in buckets with no histogram data.

GET /api/apm/transactions

GET /api/apm/transactions?time_range=24h&service_id=<id>

Transaction groups seen in the range:

{
  "transactions": [
    {
      "name": "GET /checkout", "type": "request",
      "service_id": "665f...", "service_name": "checkout-api",
      "count": 240000, "sum": 9800000.0, "min": 1.0, "max": 2500.0,
      "avg": 40.8,
      "p50": 32.0, "p90": 78.8, "p95": 93.7, "p99": 264.0,
      "success": 239100, "failed": 900, "failure_rate": 0.00375,
      "latest_at": "...",
      "breakdown": [
        {"type": "db", "subtype": "postgresql", "count": 480000, "sum": 3200000.0}
      ]
    }
  ]
}

GET /api/apm/transactions/series

GET /api/apm/transactions/series?service_id=<id>&name=<n>&type=<t>&time_range=24h

Same downsampling rules as metric series; buckets carry {bucket_at, count, sum, min, max, avg, p50, p99, success, failed} plus top-level bucket_seconds. p50/p99 merge the histograms within each downsampled bucket and are null where there is no histogram data. Per-group p50/p90/p95/p99 on /api/apm/transactions follow the same rule.

GET /api/apm/errors

GET /api/apm/errors?time_range=24h&service_id=<id>

Error groups sorted by last_seen_at descending, capped at 200:

{
  "errors": [
    {
      "id": "665f...", "fingerprint": "a1b2c3d4e5f6a7b8",
      "service_id": "665f...", "service_name": "checkout-api",
      "type": "ValueError", "message": "invalid order id",
      "culprit": "orders.checkout.validate",
      "transaction_name": "POST /checkout",
      "count": 3421, "first_seen_at": "...", "last_seen_at": "...",
      "stack": [{"function": "handle", "file": "...", "line": 41}]
    }
  ]
}

GET /api/apm/traces

GET /api/apm/traces?time_range=24h&service_id=<id>&transaction_name=<optional>&trace_id=<optional>

Sampled traces, slowest first, capped at 50. Each document is the stored trace sample (trace_id, transaction fields, duration_ms, started_at, outcome, spans, spans_dropped, service_id, service_name, hostname). Querying by trace_id returns every service's samples for that trace.

GET /api/apm/overview

GET /api/apm/overview?time_range=24h&service_id=<optional>&deployment=<optional>

The four golden headline series from transaction rollups: latency (p50 and p99 from the duration histograms, plus avg and max), throughput, error rate, and time-per-request by span type (top 5 types, an other fold, and an app remainder = transaction time minus instrumented span time, clamped at 0). Response: totals (requests, per_minute, avg_ms, p50_ms, p90_ms, p95_ms, p99_ms, max_ms, error_rate, span_types), series (per bucket: count, avg, p50, p99, max, failed, error_rate, spans map of ms/request), and, when service-scoped, versions (see below).

totals percentiles aggregate every histogram in the range; series percentiles merge the histograms inside each downsampled bucket. All are null when no transaction in scope reported buckets; avg/max are unchanged either way.

GET /api/apm/regressions

GET /api/apm/regressions?time_range=24h&service_id=<optional>

Deploy-anchored pre/post comparisons for every service version first seen inside the window, powered by the version history the server records at ingest (one document per (service, version) with first/last seen). Each comparison uses equal windows either side of the deploy (60 min baseline, 5 min warm-up skipped, up to 60 min candidate; "collecting" until 10 candidate minutes exist) and judges only groups with ≥100 requests and ≥5 minute buckets on both sides. Verdicts are dual-threshold: latency regresses at ≥30% relative AND ≥100 ms absolute; error rate at ≥50% relative AND ≥1 point absolute AND ≥10 failed candidate requests. Each flagged group carries attribution: the span type whose ms/request moved most, classified "more calls per request (N+1?)" vs "slower calls". adoption_share (trailing 6h) marks low-traffic versions as early_signal. The response also carries has_version_history so the dashboard can distinguish "no service reports a version yet" (setup guidance) from "versions flowing, no deploy in this window".

The same computation runs server-side on the issue-monitor cadence and opens one apm.regression issue per (service, version), titled "Possible faulty deployment: <service> <version>", through the normal issue pipeline (PagerDuty on first open, stale sweep auto-resolve). Disable per workspace with the apm_regression_alerts_enabled setting.

Version markers: GET /api/apm/metrics/series, /api/apm/transactions/series, and service-scoped /api/apm/overview responses include versions: [{version, first_seen_at}] inside the window, rendered as vertical deploy markers on charts. Wrappers opt in simply by setting service_version; transaction rollups are also stamped with the flushing wrapper's version (service_version field) so traffic can be sliced by app version.

GET /api/apm/clients

GET /api/apm/clients?time_range=24h&group_by=agent_type|client_ip&service_id=<optional>

Sampled requests grouped by user-agent family or origin IP, built from the http context on trace samples. agent_type is derived server-side at ingest from http.user_agent (coarse families: Chrome, Firefox, Safari, Edge, Opera, Bot, curl, wget, Python, Go, Java, Node.js, Dart, Postman, RootTrace, Other, Unknown). Wrappers never send it. Because trace samples are the slowest transactions per flush, these numbers describe the sampled slice, not total traffic; the response carries "sampled": true. Sorted by count descending, capped at 500 groups.

{
  "group_by": "agent_type",
  "time_range": "24h",
  "sampled": true,
  "clients": [
    {"key": "Chrome", "count": 412, "avg": 233.1, "max": 2210.0,
     "failed": 3, "last_seen_at": "...", "unique_ips": 57}
  ]
}

client_ip rows carry agent_types (up to 5 families seen from that IP) instead of unique_ips. transaction_name=<name> narrows the view to the clients that hit one endpoint.

GET /api/apm/clients/series

GET /api/apm/clients/series?key=<group key>&group_by=agent_type|client_ip&time_range=24h&service_id=<optional>

Time series of sampled requests for one group, for charting. Same downsampling buckets as the other series endpoints; each bucket carries {bucket_at, count, avg, max, failed} plus top-level bucket_seconds. The response also carries transactions: the top 10 API calls this group hit, each {name, count, avg, failed}.

GET /api/apm/dependencies

GET /api/apm/dependencies?time_range=24h&service_id=<id>

Outbound destinations aggregated from the http.client.duration rollups:

{
  "dependencies": [
    {"service_id": "665f...", "service_name": "checkout-api",
     "destination": "payments.internal:8443",
     "count": 120000, "avg": 7.5, "max": 900.0, "failures": 320}
  ]
}

failures counts requests whose status tag was 4xx, 5xx, or error.

GET /api/apm/graph

GET /api/apm/graph?time_range=24h&deployment=<optional>

The service dependency graph for the whole workspace scope, built from the same http.client.duration rollups as /dependencies, but with each destination resolved back to the service that answers it, so the result is a graph rather than a per-service list.

{
  "time_range": "24h",
  "nodes": [
    {"id": "665f...", "service_id": "665f...", "name": "checkout-api",
     "kind": "service", "language": "python",
     "requests": 240000, "failed": 900, "error_rate": 0.00375, "avg_ms": 40.8},
    {"id": "ext:api.stripe.com:443", "service_id": null,
     "name": "api.stripe.com:443", "kind": "external", "language": null,
     "requests": 0, "failed": 0, "error_rate": null, "avg_ms": null}
  ],
  "edges": [
    {"source": "665f...", "target": "667a...",
     "count": 120000, "avg_ms": 7.5, "max_ms": 900.0,
     "failures": 320, "failure_rate": 0.00267,
     "destinations": ["payments.internal:8443"]}
  ]
}

Destination resolution matches a destination's host (the port is ignored) against every APM service in the same organization and environment, on:

  • the service name,
  • any hostname in the service's reported hostnames list,
  • any Kubernetes deployment name on the service record, with and without its namespace suffixes: <deployment>, <deployment>.<namespace>, <deployment>.<namespace>.svc, and <deployment>.<namespace>.svc.cluster.local.

Matching is case-insensitive. A destination resolving to nothing becomes an external node ({"id": "ext:<destination>", "kind": "external"}), so third-party APIs and unmonitored services stay visible instead of being dropped. Several destinations resolving to the same service fold into one edge, which lists the destinations behind it (up to 10). Self-calls are not edges. Node request totals come from transaction rollups over the same window; a service that is silent in the window and touches no edge is left off the graph.

GET /api/apm/graph/impact

GET /api/apm/graph/impact?service_id=<id>&time_range=24h

The blast radius of one service, computed as a breadth-first walk of the graph edges in both directions:

{
  "time_range": "24h",
  "service_id": "665f...",
  "service_name": "payments",
  "upstream": [
    {"id": "667a...", "service_id": "667a...", "name": "checkout-api",
     "kind": "service", "depth": 1, "requests": 240000, "failed": 900,
     "error_rate": 0.00375, "avg_ms": 40.8}
  ],
  "downstream": [
    {"id": "ext:api.stripe.com:443", "name": "api.stripe.com:443",
     "kind": "external", "depth": 1, "requests": 0, "failed": 0,
     "error_rate": null, "avg_ms": null}
  ]
}

upstream is everything that transitively calls this service: who breaks if it breaks. downstream is everything it transitively depends on: what could be breaking it. depth is the hop count, so immediate neighbours are depth: 1. Cycles terminate (each node is visited once). 404 when the service id is not an APM service in scope.