RootTrace

RootTrace APM Protocol

How the RootTrace APM wrappers (Python, Java, Node.js, Go, and PHP) report application performance metrics to the RootTrace API. Metric names are dynamic: applications may report any metric name at any time, and the API and UI discover them from the data.

This document is the write side: the wire contract a wrapper must satisfy, covering metric/transaction ingest, log shipping, and continuous profiling. The endpoints the dashboard reads back are in the APM read API reference. If you are instrumenting an application rather than writing a wrapper, start with the APM SDK guide instead.

Authentication

Wrappers use the same environment-scoped collector token as the RootTrace collector (rtc_...), sent on every request:

Authorization: Collector <token>

The Bearer scheme is also accepted. Tokens are minted from the dashboard during collector onboarding.

Ingest

POST /api/apm/ingest

One request per flush interval, containing everything the wrapper aggregated since the previous flush.

{
  "service": "checkout-api",
  "language": "python",
  "hostname": "web-1",
  "runtime": {
    "language_version": "3.12.1",
    "pid": 1234,
    "wrapper_version": "0.1.0"
  },
  "interval_seconds": 30,
  "metrics": [
    {
      "name": "http.request.duration",
      "kind": "timer",
      "unit": "ms",
      "count": 120,
      "sum": 5400.0,
      "min": 3.0,
      "max": 250.0,
      "tags": {"endpoint": "/checkout"}
    },
    {
      "name": "orders.processed",
      "kind": "counter",
      "count": 42,
      "sum": 42.0
    },
    {
      "name": "process.memory.rss_bytes",
      "kind": "gauge",
      "unit": "bytes",
      "value": 183500800.0
    }
  ]
}

Beyond metrics, a flush may carry three more optional arrays: transactions, errors, and trace_samples. Each is described in its own section below.

Field rules:

  • service: required, 1–160 chars. Identifies the application.
  • service_version: optional, ≤64 chars. Deployed version of the app; the latest one wins on the stored service record.
  • language: required, e.g. python, java.
  • hostname: required. The reporting instance.
  • runtime: optional free-form dict; the latest one wins.
  • kubernetes: optional object {deployment, namespace, pod}, each ≤253 chars (DNS-1123). See "Kubernetes context" below.
  • interval_seconds: the wrapper's flush interval (used for staleness). Default 30, minimum 5, maximum 3600; the server rejects values outside that range, so wrappers clamp to it.
  • metrics: up to 500 entries per request, including the automatic runtime metrics (wrappers reserve headroom for them under their series cap). Each entry:
    • name: required, 1–200 chars. Dynamic; any dotted name.
    • kind: counter, gauge, or timer.
    • unit: optional display unit (ms, bytes, %, ...).
    • tags: optional flat string map, at most 8 keys. Low cardinality (endpoint names or queue names, never user ids).
    • Aggregate fields, by kind:
      • counter: count = number of increment calls, sum = total amount added since the last flush.
      • gauge: value = last observed value.
      • timer: count, sum, min, max over the observed durations since the last flush. sum/min/max are in unit (default ms). May also carry buckets; see "Duration histograms".

Response 200:

{"accepted": 3, "rejected": 0, "rejected_buckets": 0, "transactions": 2,
 "errors": 1, "traces": 2, "service_id": "665f...",
 "server_time": "2026-07-07T12:00:00+00:00"}

rejected_buckets counts entries whose buckets object was malformed and therefore dropped; the entry itself still counted toward accepted.

Error responses: 401 invalid or revoked token, 422 malformed payload, 429 flushing faster than the server allows (honor Retry-After).

Duration histograms

count/sum/min/max support an average and an extreme, and nothing in between. A mean hides the tail that pages people. Timer metrics and transaction groups may therefore also carry buckets: a fixed-layout log2 histogram of the durations aggregated since the last flush, which the server accumulates per rollup bucket and reads back as real percentiles.

buckets maps a stringified integer bucket index to a positive integer count:

{
  "name": "GET /checkout",
  "type": "request",
  "count": 120, "sum": 5400.0, "min": 3.0, "max": 250.0,
  "success": 118, "failed": 2,
  "buckets": {"53": 90, "66": 29, "79": 1}
}

The layout is fixed, so wrappers and server never negotiate it. For a duration d_ms, the bucket index is:

i = min(127, max(0, floor(log2(max(d_ms, 0.001)) * 4) + 40))

and the representative duration of bucket i (what a percentile landing in that bucket reports) is:

representative_ms(i) = 2 ** ((i - 40 + 0.5) / 4)

That is 128 buckets at 4 per power of two: ~19% relative width, index 40 is 1 ms, index 0 absorbs everything at or under ~1 µs, index 127 absorbs everything over ~5 days. Percentiles are therefore accurate to within one bucket width, which is the point: the tail's magnitude, not its exact value.

Validation, per buckets object:

  • keys must parse as integers in [0, 127],
  • values must be positive integers,
  • at most 128 entries.

A buckets object failing any of these is dropped silently and the rest of the entry is accepted as normal (its count/sum/min/max still roll up); the flush response counts the drop in rejected_buckets. A malformed histogram never fails an entry, and never fails a flush.

buckets is optional everywhere. Wrappers that don't send it, and rollups written before it existed, behave exactly as before: every percentile field on every read endpoint is null, and avg/max are unaffected.

Server-side, buckets accumulate with $inc on buckets.<i> alongside count/sum/min/max, on both metric and transaction rollups, in the same per-minute documents, so percentiles are available per time bucket, not only over a whole range.

Transactions and spans

A transaction is one unit of work in the app: an HTTP request being served, a job run, a scheduled task. Spans are timed operations inside a transaction (a DB query, an outbound HTTP call). Wrappers aggregate completed transactions per flush into groups keyed (name, type) and send them in the ingest payload's transactions array (at most 250 groups per flush; at most 40 span-breakdown rows per group):

"transactions": [
  {
    "name": "GET /checkout",
    "type": "request",
    "count": 120, "sum": 5400.0, "min": 3.0, "max": 250.0,
    "success": 118, "failed": 2,
    "spans": [
      {"type": "db", "subtype": "postgresql", "count": 240, "sum": 1800.0},
      {"type": "http", "subtype": "payments.internal:8443", "count": 120, "sum": 900.0}
    ]
  }
]
  • name: required, 1–200 chars. Route template or job name, never a raw URL (cardinality).
  • type: request, task, or any label ≤40 chars.
  • count/sum/min/max: duration aggregates in ms, like a timer.
  • buckets: optional duration histogram, exactly as on timer metrics (see "Duration histograms"). This is what makes the percentile fields on the transaction and overview endpoints non-null.
  • success/failed: outcome counts; success + failed == count.
  • spans[].type: span category (db, http, cache, custom, ...); subtype (≤200 chars) narrows it (driver name, destination host:port). count/sum aggregate the spans of that type across the group's transactions in this flush. This is the breakdown metric: the dashboard shows where transaction time went.

Errors

Wrappers capture exceptions (from transaction bodies, or via an explicit capture call), group them by fingerprint since the last flush, and send at most 50 distinct errors per flush:

"errors": [
  {
    "fingerprint": "a1b2c3d4e5f6a7b8",
    "type": "ValueError",
    "message": "invalid order id",
    "culprit": "orders.checkout.validate",
    "count": 5,
    "transaction_name": "POST /checkout",
    "stack": [
      {"function": "handle", "file": "app/orders/checkout.py", "line": 41}
    ]
  }
]
  • fingerprint: required, ≤64 chars. The first 16 hex chars of SHA-256 over type + culprit + the top ≤5 stack frames (file:function each). Stable across restarts.
  • message: ≤1000 chars, truncated by the wrapper.
  • culprit: where it happened, module.function (Python) / Class.method (Java), ≤300 chars.
  • stack: outermost frame first, at most 50 frames, each {function, file, line}.
  • count: occurrences since the last flush.

The server keeps one document per (service, fingerprint) accumulating count, first_seen_at, last_seen_at, and the latest message/stack. It also folds a per-minute errors.count counter into the metric rollups so error rate charts over time. The metric name errors.count is therefore reserved: user metrics with that name are skipped at ingest.

Trace samples (distributed tracing)

Each flush may carry up to 2 sampled transactions (the slowest completed since the previous flush) with their full span trees, so the dashboard can render a waterfall without storing every event:

"trace_samples": [
  {
    "trace_id": "0af7651916cd43dd8448eb211c80319c",
    "transaction_name": "GET /checkout",
    "transaction_type": "request",
    "duration_ms": 812.4,
    "started_at": "2026-07-07T12:00:00+00:00",
    "outcome": "success",
    "spans_dropped": 0,
    "spans": [
      {"name": "SELECT orders", "type": "db", "subtype": "postgresql",
       "start_offset_ms": 1.2, "duration_ms": 320.0}
    ],
    "http": {
      "method": "GET",
      "path": "/checkout?step=2",
      "status_code": 200,
      "client_ip": "203.0.113.7",
      "remote_ip": "10.0.0.2",
      "user_agent": "Mozilla/5.0 ..."
    }
  }
]
  • trace_id: 32 lowercase hex chars (W3C format). At most 100 spans per sample; excess spans are dropped and counted in spans_dropped.
  • http: optional request context, every field optional. Because samples are individual requests (never aggregated), this is where per-request debugging detail lives; transaction names and metric tags stay low-cardinality. Fields: method (≤40); path, the real path with its query string, unnormalized (≤1024; the server masks the values of sensitive-looking query parameters before storing); status_code (int, 0–999; wrappers drop out-of-range values so a bad code can't get the whole flush rejected); client_ip, the claimed origin of the connection, which is the first X-Forwarded-For hop when present, else the socket peer (≤64); remote_ip, always the direct socket peer (≤64), the one address the transport vouches for; and user_agent (≤300). X-Forwarded-For is set by whoever sent the request, so treat client_ip as client-controlled unless a trusted edge proxy overwrites the header. The Python WSGI middleware and Node middleware fill it automatically; Python Transaction.set_http(...), Node Transaction.setHttp({...}), and Java Transaction.setHttp(method, path, statusCode, clientIp, remoteIp, userAgent) set it from custom instrumentation.

Trace context propagation

Outbound HTTP calls made inside a transaction carry a W3C traceparent header (00-<trace_id>-<span_id>-01). Inbound instrumentation (the Python WSGI middleware; startTransaction(name, type, traceparent) in Java) adopts the incoming trace_id, so samples of one distributed request share a trace id across services and can be found together via the traces API.

Kubernetes context

When a wrapper runs inside Kubernetes it reports its workload identity so the dashboard can filter and sort at the deployment level:

"kubernetes": {
  "deployment": "checkout-api",
  "namespace": "prod",
  "pod": "checkout-api-7d9f8b6c5d-x2k4p"
}

Detection, in order of precedence:

  1. Explicit configuration: deployment=/namespace= init options, or the ROOTTRACE_APM_DEPLOYMENT / ROOTTRACE_APM_NAMESPACE env vars (set them via the Downward API for exact values).
  2. In-cluster auto-detection, only when KUBERNETES_SERVICE_HOST is set: pod is the hostname; namespace is read from /var/run/secrets/kubernetes.io/serviceaccount/namespace; deployment is derived from the pod name: <name>-<replicaset-hash>-<suffix> collapses to <name> (Deployments), <name>-<ordinal> to <name> (StatefulSets). Names that match neither pattern are sent as-is.

Outside Kubernetes the object is omitted entirely.

Server-side, deployment is a dimension on metric and transaction rollups (empty string when absent) and a field on error groups and trace samples, so every read endpoint accepts an optional deployment filter. The service record accumulates a deployments map, from deployment name to {namespace, pods, last_seen_at} with the pod list capped at 50. GET /api/apm/summary returns that map per service for building filter and grouping UI. Deployment counts are low-cardinality by nature (one per workload), so the dimension does not meaningfully grow storage; pods are deliberately NOT a rollup dimension.

Outbound HTTP monitoring

Wrappers time every outbound HTTP call and report, as ordinary dynamic metrics: http.client.duration (timer, ms) and http.client.requests (counter), both tagged {"destination": "host:port", "status": "2xx"}. Calls that fail without an HTTP response (connection refused, timeout) are recorded with status error. Inside a transaction the call additionally becomes a span of type http with the destination as subtype. Python instruments the stdlib http.client (which also covers requests/urllib3) and, when installed, httpx (sync and async clients; a redirect chain followed by httpx is one recorded call against the original destination); Java wraps java.net.http.HttpClient via RootTraceApm.wrap(client); Node.js instruments the core http/https modules (which also covers axios, got, and node-fetch).

The destination tag is host:port only, never the path, to keep cardinality bounded.

Server-side storage

Each metric entry is folded into a minute-bucket rollup keyed (organization_id, environment_id, service_id, name, tags_key, bucket_at) where tags_key is the canonical k=v,k=v form of tags sorted by key (empty string when no tags). Before joining, %, =, and , in tag keys and values are percent-encoded (%25, %3D, %2C) so distinct tag maps can never collide. Buckets accumulate count, sum, min, max, last_value. Services are upserted into apm_services keyed (organization_id, environment_id, name).

Tag cardinality is bounded server-side: after 2,000 distinct tag combinations per (service, metric name), new combinations fold into the untagged series, so granularity degrades instead of storage growing without bound.

Alerting

Ingest feeds the platform's existing issue pipeline, so APM alerts dispatch to the workspace's configured PagerDuty integration like any other issue:

  • An error group with no active issue (a fingerprint seen for the first time, or one still occurring after its issue was resolved) opens a check_type: apm.error issue, severity high. While the issue stays open, recurring occurrences refresh it without paging again.
  • A transaction group whose latency in a flush runs hot opens a check_type: apm.latency issue, severity high. The threshold is dynamic by default: each transaction is compared to its own trailing 7-day baseline and alerts when the flush exceeds apm_latency_baseline_multiplier × baseline (floored so trivially fast paths don't page). When both the flush and the baseline window carry duration histograms, the comparison is p99 vs baseline p99, since the tail is what pages are about and a mean hides it; otherwise it falls back to the mean (sum / count) on both sides. The same floors and multipliers apply either way, and the issue evidence records which was used (basis: "p99" or "avg"). A transaction without enough history for a trusted baseline falls back to the static apm_latency_threshold_ms. When a later flush brings latency back under whichever threshold applied, the issue auto-resolves (and PagerDuty receives a resolve event).

Both kinds auto-resolve on their own. The point where "it stopped" is an absence, not an event. A background sweep resolves any apm.error or apm.latency issue whose last occurrence is older than apm_alert_auto_resolve_seconds (default 900): an error that stopped firing, a latency issue whose service went silent, or a recovery a flush never got to report. Resolution emits the same issue.auto_resolved event and PagerDuty resolve as a manual close.

Two organization settings control this, set through the standard organization settings endpoint:

  • apm_error_alerts_enabled (boolean, default true)
  • apm_latency_baseline_enabled (boolean, default true): compare each transaction to its own recent baseline instead of a fixed number
  • apm_latency_baseline_multiplier (number ≥ 1.5, default 2): how many times its baseline a transaction may run before it pages
  • apm_latency_threshold_ms (number, default 2000; 0 disables): the static fallback used for transactions without enough history for a baseline

Alert evaluation is best-effort: an alerting failure is logged server-side and never fails the ingest response.

Continuous profiling

Two endpoints, both authenticated with the same collector token as ingest and both requiring the collector:apm scope.

GET /api/apm/config

What a wrapper should be doing. Profiling is off until somebody enables it in the RootTrace UI, so a wrapper asks rather than deciding for itself.

GET /api/apm/config?service=checkout-api
Authorization: Collector <token>
{"service": "checkout-api", "source": "environment",
 "profiling_enabled": true, "sample_rate_hz": 100,
 "upload_interval_seconds": 60, "max_frames": 128, "max_stacks": 5000,
 "profile_types": ["cpu", "wall"],
 "bounds": {"sample_rate_hz": {"min": 10, "max": 200},
            "upload_interval_seconds": {"min": 15, "max": 900},
            "max_frames": {"min": 16, "max": 256},
            "max_stacks": {"min": 500, "max": 20000}}}

source is service, environment, or default, saying which document answered. bounds is published so a wrapper can clamp against the server's real limits rather than constants that drift out of step with them.

Contract a wrapper must satisfy, all of it deliberate:

  • Scope comes from the token, never the request. The server resolves organization and environment from the token document, so ?service= narrows the answer and cannot widen it.
  • The server clamps every value before returning it. A stored 10kHz comes back as 200.
  • The wrapper clamps again, independently. Trusting the server to bound these would turn a compromised or spoofed control plane into an application-level denial of service inside every customer process. It costs four lines per wrapper; do not skip it.
  • Fail closed. If the config cannot be fetched, profiling does not start. A wrapper already profiling keeps its current settings rather than tearing down for a transient blip.
  • Poll on the background loop, not inside flush(). A caller asking to flush wants its buffered metrics sent, not a second connection opened to the control plane. Once per 60s is the expected cadence.

POST /api/apm/profiles

Two transports, converging on one internal shape: the same pattern as OTLP.

JSON, for wrappers that build stacks themselves:

{"service": "checkout-api", "profile_type": "wall",
 "value_unit": "nanoseconds", "period_ms": 10.0, "duration_ms": 60000.0,
 "sample_count": 5931, "deployment": "checkout-7d9f", "service_version": "1.4.2",
 "overhead_percent": 0.31,
 "stacks": [{"frames": ["main", "handler", "encode_json"], "value": 8000000000}]}

frames are root-first, and are function identities without line numbers. A line number would give a function a new identity every time an edit shifted it down, which breaks diffing a flame graph across a deploy. At most 256 frames per stack and 20,000 stacks per profile; longer stacks are truncated, excess stacks rejected.

value carries whatever value_unit says. Do not convert a sample count into nanoseconds: a count relabelled as a duration becomes a cost estimate in vCPU-hours that was never measured.

profile_type is cpu, wall, or heap, and is a first-class storage dimension. CPU and wall clock are different measurements and the server never merges them. Report what you actually measured.

overhead_percent is optional and should be measured, not asserted. Say in your wrapper's docs what it covers: a sampler the wrapper owns can include its own cost, while a runtime-internal sampler cannot be observed at all and should report nothing rather than a plausible number.

pprof, for anything already profiling:

POST /api/apm/profiles?service=checkout-api&profile_type=cpu&deployment=checkout-7d9f
Content-Type: application/x-protobuf
Authorization: Collector <token>

<gzipped pprof bytes>

Gzipped or not, the magic bytes are sniffed. Go's runtime/pprof emits this format from the standard library, so the Go wrapper posts its bytes unchanged with no conversion step. When a profile carries several sample types (a Go CPU profile carries samples/count and cpu/nanoseconds), the time-valued one wins because it is the one that converts to a cost; failing that the last one does, which is pprof's own convention and gives a heap profile inuse_space.

?service= is required for a pprof upload, since the format does not carry it.

Storage

One document per (organization_id, environment_id, service, deployment, profile_type, five-minute bucket), holding {stack_key: value} merged with $inc, plus a separate symbol dictionary keyed by a hash of the frames. Stack keys are hex, so they are safe as MongoDB field names. Both collections follow the workspace's plan retention; the settings do not, because configuration somebody entered should not expire and silently switch profiling off across a fleet.

Uploads for the same bucket merge rather than replace, so several processes of one service (or several requests, for the PHP wrapper) aggregate into one profile.

Security signals

POST /api/apm/security-events

Reported when instrumented code reaches a call site an attacker needs, inside a transaction. Same collector token and collector:apm scope as ingest and profiling: a deployment that can send telemetry can send these.

POST /api/apm/security-events
Authorization: Collector <token>
{
  "service": "checkout-api",
  "service_version": "4.12.0",
  "deployment": "checkout-api",
  "events": [
    {
      "kind": "process_spawn",
      "target": "/bin/sh",
      "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
      "transaction": "POST /api/v1/upload",
      "method": "POST",
      "path": "/api/v1/upload",
      "frames": ["app/upload.py:convert", "subprocess.Popen"],
      "observed_at": "2026-08-05T14:03:22.512Z"
    }
  ]
}
{"accepted": 1, "dropped": 0}

kind must be one of process_spawn, deserialization, dynamic_eval, file_access, outbound_new_host. An unknown kind fails the whole batch, so a wrapper must not invent one. The kinds the server derives for itself (cpu_shift, profiling_silent) are rejected here, so a compromised process cannot file a finding that reads as the server's own conclusion.

Lengths, all clipped by the wrapper and enforced again on arrival: service 1-160, service_version 64, deployment 200, target 512, trace_id 64, transaction 200, method 10, path 400. At most 32 frames are kept, each clipped to 200. observed_at is optional and RFC 3339; without it the arrival time is used.

Contract a wrapper must satisfy, all of it deliberate:

  • Never send the arguments. Only the shape: the executable that was spawned, the format that was deserialized. An attacker's arguments are the exfiltration, and a wrapper that shipped them would move a customer's breach rather than report it. The server does not store an argument field to put them in.
  • Never send a client address. The server takes the reporting host from the connection, through its trusted-proxy configuration, because the process filing the report is the one that may already be owned. The address the request came from is read off the trace trace_id names, so send the trace id and let the two be joined.
  • Bound the buffer and drop rather than grow. A process being driven by an attacker produces these faster than any flush interval. The reference wrappers hold 100 events between flushes and discard beyond that: an unbounded buffer inside the victim is its own denial of service.
  • Expect a trimmed batch, not a rejected one. At most 200 events per request are recorded; the rest are counted in dropped and logged server-side. Failing the request would lose the whole batch instead of its tail. The body limit for this endpoint is 4MB.
  • Expect a clamped timestamp. An observed_at more than 6 hours behind the server, or more than 5 minutes ahead of it, is moved to the nearest value the server will accept and the occurrence is flagged so the UI can say the clock was not trusted. Buffering between flushes is normal and is not clamped.

Grouping and storage

One document per (organization_id, environment_id, service, fingerprint), where the fingerprint covers kind, service and target and deliberately not the trace or the address. One attacker probing an endpoint a thousand times is one finding with a thousand occurrences rather than a thousand rows. The last 20 occurrences are kept in full.

A service holds up to 500 distinct findings. Past that, further distinct targets fold into one finding per kind, because the target is chosen by whoever spawned it and minting a document per name is a cheaper way to hide a shell than silencing the wrapper. Findings that already exist keep collecting occurrences regardless of the count.

Findings follow the audit trail's fixed retention rather than the workspace's telemetry retention: one that expired alongside the profile that produced it would disappear exactly when somebody came looking for it.

Go and PHP wrappers

Two more wrappers speak this protocol: apm/go and apm/php. They follow the same contract as the others, with the same collector token, the same ingest payload, and the same flush model, and report the same transaction, error, and trace-sample shapes. Language-specific details live in each wrapper's own README.

PHP is the one wrapper without a background loop: apm/php follows the request lifecycle, so its profile covers one request and ships with the shutdown flush. The server's five-minute buckets merge those into the same profile a long-running process would produce.

What each wrapper profiles

Every wrapper reports what its runtime can actually measure, and labels it honestly. A wrapper must never relabel wall clock as CPU to make a chart look comparable.

Wrapperprofile_typeMechanismRate
Gocpu, heapruntime/pprof, posted as pprofFixed 100Hz
Node.jscpuV8 via node:inspectorConfigurable
Pythonwallsys._current_frames() on a daemon threadConfigurable
JavawallThread.getAllStackTraces() on a daemon threadConfigurable
PHPwallext-excimer (optional; no profiling without it)Configurable

Go's rate is fixed because pprof.StartCPUProfile calls runtime.SetCPUProfileRate(100) itself and the runtime refuses to change it mid-profile; the wrapper logs a configured rate as inapplicable rather than ignoring it silently.

Every sampling wrapper weights samples by the period it actually achieved, not the one it requested: elapsed window divided by passes taken. A Python sampler starved by the GIL, a JVM that will not reach a safepoint on time, and an Excimer interrupt that cannot fire inside a blocking C call all take fewer passes than the rate implies. At the nominal period every duration in the profile would be understated by exactly that shortfall, and the totals would quietly fail to add up to the window they cover.

Log shipping

The Node.js, Python, and Java wrappers can also ship application logs to POST /api/logs/ingest, using the same collector token as APM ingest. Log shipping is a separate endpoint with its own payload contract; it is not part of /api/apm/ingest and is documented with the logs API.

Each of the three offers a direct call, and where the language has a dominant logging framework, an adapter that needs no call-site changes:

  • Python: RootTraceLogHandler, a stdlib logging.Handler.
  • Java: RootTraceApm.log(level, message, attrs), plus io.roottrace.apm.logback.RootTraceAppender for Logback (MDC entries ride along as attrs).
  • Node.js: apm.log(level, message, attrs) and apm.logger().

All are opt-in: nothing captures logs until you attach the handler/appender or call log(...). Go and PHP have no log API. For those, use OTLP /v1/logs or the collector's file tailer.

Records buffer in memory (cap 500, drop-oldest) and ship on the wrapper's existing flush interval, to its own endpoint with its own rate-limit backoff, so a throttled log stream never stalls metrics. The record shape is:

{"service": "checkout", "level": "warn", "message": "payment declined",
 "logger": "com.acme.PaymentService", "timestamp": "2026-07-16T01:23:45.678Z",
 "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
 "attrs": {"order_id": "A-91"}}

posted as {"logs": [...]} (the server also accepts {"entries": [...]}, which is what the collector sends). trace_id is attached automatically when a transaction is active on the calling thread, which is what makes a log line clickable from its trace. Levels are free-form and normalized server-side to debug/info/warn/error/fatal, with warning, critical, crit, err and trace aliased; an unrecognized level is rejected rather than guessed at. Messages truncate at 8KB, attrs at 16 keys / 512 chars per value. Attribute keys that look like secrets (password, token, api_key, authorization, cookie, session and friends) are redacted in the wrapper, before the record leaves the process: a key-name heuristic that catches the obvious accident and cannot catch a secret pasted into a message.

Wrapper behavior

The wrappers follow the same contract:

  • Configuration, in order of precedence: explicit init arguments, then environment variables ROOTTRACE_APM_SERVICE, ROOTTRACE_APM_TOKEN (falls back to ROOTTRACE_COLLECTOR_TOKEN), ROOTTRACE_API_URL (default https://api.roottrace.io/api), ROOTTRACE_APM_INTERVAL_SECONDS (default 30, minimum 5). That default targets RootTrace Cloud. A self-hosted deployment must set the URL explicitly on every instrumented service, or its telemetry leaves the network. The collector, by contrast, has no default destination and refuses to start without one.
  • The RootTrace API server URL is a first-class init setting: Python init(api_url=...), Java ApmConfig.builder().apiUrl(...). The wrapper validates it at init (http/https scheme required, fail-fast on misconfiguration) and exposes the resolved values read-only for diagnostics and logging: Python Apm.api_url and Apm.ingest_url properties on the instance init() returns; Java RootTraceApm.apiUrl() and RootTraceApm.ingestUrl(). ingest_url is the full flush target, <api_url>/apm/ingest.
  • Metrics aggregate in memory between flushes; a background daemon thread/executor posts one ingest payload per interval.
  • At most 200 distinct (name, tags) entries per flush; excess recordings are dropped and a warning is logged once.
  • On send failure: log the error and merge the unsent aggregates back into the live buffer (so counter deltas and timer stats aren't lost). If the merged buffer would exceed the cap, drop the oldest data and log.
  • Instrumentation never raises into the host application. The one deliberate exception is init-time misconfiguration, which fails fast: init() with a missing service/token or a non-http(s) api_url, and Java wrap(null).
  • Runtime metrics are reported automatically each flush (gauges unless noted):
    • Python: process.memory.rss_bytes, process.cpu.percent, process.threads, process.gc.collections (counter delta), process.uptime_seconds.
    • Java: jvm.memory.heap_used_bytes, jvm.memory.heap_max_bytes, jvm.gc.collections (counter delta), jvm.gc.time_ms (counter delta), jvm.threads.count, process.uptime_seconds.
    • Node.js: process.memory.rss_bytes, process.cpu.percent, process.uptime_seconds, nodejs.eventloop.lag_ms (mean event-loop delay since the last flush), nodejs.gc.collections and nodejs.gc.time_ms (counter deltas), nodejs.handles.active.
  • User-Agent: roottrace_apm-python/<version> / roottrace_apm-java/<version> / roottrace_apm-node/<version>.
  • Transactions: a contextvar (Python) / ThreadLocal (Java) tracks the active transaction; spans opened while one is active attach to it. Completed transactions fold into per-(name, type) aggregates (cap 250 groups per flush; excess dropped with a logged warning) and compete for the 2 trace-sample slots (slowest wins). An exception escaping a transaction body marks it failed, is captured as an error, and is re-raised unchanged.
  • Errors: grouped by fingerprint between flushes, cap 50 distinct (excess dropped with a warning, counts preserved on the kept ones). An explicit capture call exists for handled exceptions.
  • Outbound HTTP instrumentation is on by default and can be disabled at init. It never raises: any instrumentation failure falls back to the uninstrumented call.
  • Transactions, errors, and trace samples ride the same flush, merge-back, and 4xx/429/5xx handling as metrics.