Continuous profiling

Tracing tells you a request took 380ms and which spans consumed it. It stops at span boundaries, so when the time is inside your own code you get one span reading handle_checkout 380ms and no explanation. And because a trace belongs to a request, work that belongs to no request (garbage collection, background threads, cron, pool churn) is invisible to tracing entirely. That is why a box at 80% CPU can have traces accounting for a third of it.

A sampling profiler answers both. Every few milliseconds the SDK records what the process is executing; thousands of samples aggregate into a picture accurate enough to name the function.

Profiling in the sidebar.

It is off until you turn it on

Profiling runs inside your process, so the switch lives here rather than in an environment variable that needs a redeploy to change:

Profiling → Settings. Enable it for a whole environment, or per service.

The SDKs poll GET /api/apm/config and start nothing on their own. They pick up a change within a minute. Turning it off during the incident it is causing is the same toggle: no deploy, no restart.

Editing these settings needs the engineer role and writes an audit event.

Security properties, all of them deliberate

• Scope comes from the collector token, never from the request. A token cannot read or apply another environment's configuration.
• The server clamps every value (sample rate to 10–200Hz, upload interval to a floor), so a bad edit cannot instruct a fleet to sample at 10kHz.
• The SDK clamps again, independently. Trusting the server to bound this would turn a compromised or spoofed control plane into an application-level DoS inside every customer process. It costs four lines in each SDK.
• Fail closed. If the config cannot be fetched, profiling does not start. An SDK that defaults to on when the control plane is unreachable is the wrong failure direction for code running in somebody else's process.

CPU and wall clock are not the same measurement

They are never merged, relabelled, or charted together.

The dynamic languages get wall clock because their runtimes cannot give a per-thread CPU sample cheaply. In CPython that means setitimer(ITIMER_PROF), whose signal only ever reaches the main thread, which is useless in a threaded web server. On the JVM the CPU-accurate mechanism is JFR's jdk.ExecutionSample, which needs JDK 14 to consume in-process while the artifact targets 11.

This is not a consolation prize. Wall clock is frequently the better answer to "why is my handler slow", because the answer is usually that it was waiting. Every screen says which measurement it is showing.

Two per-language notes worth knowing before you read a number:

• Go's rate is fixed. runtime/pprof hard-codes 100Hz and the runtime refuses to change it while a profile is running, so a rate set in the UI does not apply to Go. The SDK logs that once rather than silently ignoring it.
• Go's heap profile is close to free. The runtime keeps it continuously whether anyone reads it or not. It answers what a CPU profile cannot: what is still holding memory. It is stored under its own profile type, never merged into a CPU flame graph.

PHP needs one extension, and only PHP

Every other SDK profiles with nothing but its standard library, so it installs in an air-gapped environment like the rest of RootTrace. PHP is the exception, because the language genuinely cannot do it: there are no threads and no userland timers, so nothing inside a request can interrupt it. Sampling has to come from the engine.

pecl install excimer          # or: apt install php-excimer

Without it the wrapper reports profiling as unavailable and everything else carries on untouched. Because PHP has no background loop, one request is one profile and it ships with the shutdown flush; the server merges them into five-minute buckets, so a busy service produces the same flame graph a long-running process would.

The config is cached in the system temp directory for a minute and refreshed after the response is sent, so no request ever waits on the control plane.

Reading a flame graph

A bar's width is the time in that function and everything it called. Its colour is a hash of its name. Every profiler follows that convention, because it makes the same function recognisable across two graphs.

The number that identifies the culprit is self time: the time inside the function itself. A wide bar is ambiguous by construction (wide because it is slow, or wide because it called something slow), and only self time tells them apart. Top functions ranks by self time for exactly this reason; ranking by total would put main at the top of every profile ever taken.

Tap a frame to zoom into it; the breadcrumb walks back out.

Frames are functions, deliberately without line numbers. A line number would give a function a new identity every time an edit shifted it down, which would break the deploy diff below.

What a function costs

For a CPU profile, self time is translated into vCPU-hours per month: the literal answer to "why are we paying for this many instances", rather than a chart you have to convert yourself.

average vCPUs   = CPU-seconds / window seconds
vCPU-hours/month = average vCPUs × 730

No cost is shown for a wall-clock or heap profile. Wall clock counts time spent waiting, so converting it to vCPUs would bill you for blocking on a socket, and a heap profile is measured in bytes and is not a duration at all. The API returns no number rather than one that happens to compute.

The deploy diff on an issue

This is the part worth having.

RootTrace already knows which deploy preceded an issue and which commits it carried. Open an issue and, when a profile covers the window, What changed in the code compares the time after the issue started against the same length of time before it, and names the functions whose share of the profile grew.

Both sides are normalised to shares of their own total first. Comparing raw values would mark every function "worse" whenever the second window simply served more traffic, which is the failure that makes naive profile diffs useless.

That extends "what changed" from commits to functions, in the same sheet.

Overhead

Where the SDK can measure its own overhead it reports it on every upload, so the cost is observed rather than asserted here.

The figures measure different things, and it matters:

• Python and Java include the sampling itself, because the sampler is a thread the SDK owns. Typically well under 1% at 100Hz.
• Node reports the SDK's collection and conversion cost only, once per upload window. V8's in-process sampling is not observable from JavaScript, so it is not in that number.
• Go and PHP report no figure. The sampling happens entirely inside the runtime or the extension, and a number the SDK cannot measure is not one it should invent.

Samples are weighted by the period actually achieved

Every sampling SDK here divides the elapsed window by the passes it actually took, rather than assuming each sample cost one nominal interval.

That matters more than it sounds. A Python sampler competing with a hot loop is starved by the GIL; Thread.getAllStackTraces() needs a safepoint a loaded JVM will not reach on time; Excimer's interrupt cannot fire inside a blocking C call. In all three cases the nominal period would understate every duration in the profile by exactly the fraction of ticks that were missed. The totals would quietly fail to add up to the window they cover.

Sending pprof directly

Anything already profiling can post to RootTrace with no RootTrace SDK. Go's runtime/pprof emits pprof from the standard library:

curl -X POST \
  -H 'Authorization: Collector <token>' \
  -H 'Content-Type: application/x-protobuf' \
  --data-binary @cpu.pprof \
  'https://api.roottrace.io/api/apm/profiles?service=checkout-api&profile_type=cpu'

Gzip is optional: the magic bytes are sniffed. When a profile carries several sample types (Go CPU profiles carry samples/count and cpu/nanoseconds), the time-valued one is used, because it is the one that converts to a cost.

Storage and retention

Two collections, following the rollup idiom used everywhere else:

• profile_stacks: the symbol dictionary, one document per distinct stack, written once and read only when rendering.
• profile_rollups: one document per (environment, service, deployment, profile type, five-minute bucket), holding a map of stack → value merged in place.

One document per bucket rather than one row per stack is what makes this affordable: a 30-day read touches buckets rather than millions of rows. Five-minute buckets, not one: profiles do not need metric resolution.

Both follow the workspace's plan retention. The settings do not: configuration somebody entered should not expire and silently switch profiling off across a fleet.

API

Read endpoints take either a preset time_range or an explicit start/end.