RootTrace

Security signals

Your APM SDK already times your code. It can also notice when your code reaches somewhere it should not have. Because that happens inside a request, it can tell you which request, from which address, on which build.

This is not a firewall. It does not inspect or block traffic, and it never captures request bodies. It reports what your application actually did.

What it detects

A dangerous call site was reached. Spawning a process, or deserializing untrusted bytes, inside a web request. Ordinary request handling rarely does either, and both are what an attacker needs after finding a way in. These reports are exact: a timestamp, not a sample.

Code that has never run before is now burning CPU. From your CPU profile, RootTrace learns which call stacks a service's current version normally executes. When a meaningful share of its CPU moves into stacks that version has never run, and stays there across consecutive five-minute windows, that is reported too. Turning CPU profiling off turns this off with it; wall and heap profiles are not used for it.

The second one exists to catch what the first cannot: something that is still running, such as a cryptominer, a resident implant, or a runaway loop. It is corroboration, not accusation, and RootTrace labels it that way.

A service that was profiling has gone quiet, and is still sending metrics. The SDK runs inside the process it watches, so an attacker with code execution can stop it reporting. The absence is harder to hide. When a service that RootTrace has profiled before sends nothing for 45 minutes while its metrics keep arriving, and profiling is still switched on for it, that is reported.

Three things it deliberately does not report, because each of them is ordinary: switching profiling off yourself, a service being retired and going quiet altogether, and a service that never profiled in the first place. The 45 minutes is three times the longest upload interval the SDK will accept, so a service on a slow interval is not mistaken for a silenced one.

An honest limit

The profiler samples. A web shell that runs id finishes in a few milliseconds, and no sampling profiler will reliably see it. That is why the call-site reports exist and why they matter more: they fire every time, not when a sample happens to land. Anyone who tells you a sampling profiler is an intrusion detection system is selling you something.

A baseline also has a size. It holds up to 20,000 distinct call stacks per service version, kept by the CPU time they hold, which covers an ordinary service whole. A version with more diversity than that is not scored for novelty at all, and the server logs that it skipped it. Reporting a version whose baseline was never able to finish learning would mean a finding on every sweep for the life of that version. The call-site reports are unaffected.

Turning it on

Python and Node.js report automatically. There is nothing to enable and no call to add. The SDK wraps the relevant standard-library entry points when it initialises.

roottrace_apm.init(service="checkout-api", token="rtc_...")
# subprocess and pickle.loads now report when reached inside a transaction.
# Disable with security_sinks=False.
roottraceApm.init({ service: "checkout-api", token: "rtc_..." });
// child_process reports when reached. Disable with securitySinks: false.

Java is explicit, at the call sites you already know are risky:

import io.roottrace.apm.SecuritySinks;

// Report the spawn, then start it as usual.
Process p = SecuritySinks.wrap(new ProcessBuilder("/usr/bin/convert", input)).start();

// Anything else.
SecuritySinks.report(SecuritySinks.DESERIALIZATION, "java.io.ObjectInputStream");

Java has no way for a plain library to intercept ProcessBuilder; doing so needs a -javaagent and bytecode rewriting at class load. The RootTrace SDK is a dependency, not an agent, and rewriting ProcessBuilder across every application on a JVM is not a decision to make on your behalf. So the call is yours to place.

Go and PHP are explicit for the same reason, and for reasons of their own.

import roottrace "github.com/byteaffinity/roottrace-apm-go"

// Report the spawn, then run it as usual. Only the executable is recorded.
out, err := roottrace.WrapCmd(tx, exec.Command("/usr/bin/convert", input)).Output()

// Anything else.
roottrace.ReportSink(tx, roottrace.SinkDeserialization, "encoding/gob")
use RootTrace\Apm\SecuritySinks;

// Reports the spawn and hands the command straight back.
$out = shell_exec(SecuritySinks::exec('/usr/bin/convert ' . $file));

// Deserialization, with allowed_classes still closed by default.
$value = SecuritySinks::unserialize($bytes);

Go resolves os/exec at compile time and offers no import hook, so the alternatives are rewriting your build or //go:linkname into the standard library. PHP's exec, shell_exec, proc_open and unserialize are internal functions that userland cannot redefine; the extensions that can (uopz, runkit7) are debugging tools, and the userland alternatives rewrite your source as it is included. None of those is a decision a dependency should make on your behalf, so in both the call is yours to place.

Reading a finding

Findings are grouped: one attacker probing an endpoint a thousand times is one finding with a thousand occurrences, not a thousand rows.

Opening one answers the questions that always follow, on one screen:

  • What ran, the executable or the format. Never the arguments.
  • Which request reached it, the endpoint, method, and trace ID, so you can open the trace.
  • Where it came from, which is two addresses and they are not interchangeable. The client address is where the request came from, read off the trace the finding names. Your SDK recorded it there from the request's X-Forwarded-For header, falling back to the socket peer, so it is only as trustworthy as the proxy in front of your application. The reporting host is the address that posted the report, resolved by the server from the connection through your trusted-proxy configuration. That one cannot be forged with a header, but it is your own application server, not the caller. A finding whose trace carried no request context shows the reporting host alone rather than offering it as the origin.
  • What else happened on that host, Linux audit events recorded around the same moment, if you run the collector with audit ingestion. An in-process report and a kernel audit record agreeing is much stronger evidence than either alone: an attacker would have to defeat both to hide.

This needs exec auditing switched on, which the collector package does by default (ROOTTRACE_LINUX_AUDIT_EXEC). Without it, the host records only privilege-changing execs, so an ordinary shell spawned by a compromised application produces no kernel record to corroborate against. Turning it off costs you this corroboration and nothing else. The finding, the request, and both addresses all come from the SDK.

  • What had just changed, the deploys shortly before, so "this started after the 14:02 release" is visible rather than reconstructed.
  • What else that host reported, every other finding filed by the same reporting host, one click away. That groups by the server that filed them, not by the caller.

Findings are triaged as acknowledged, resolved, or not a threat. False positives are recorded rather than deleted: their rate is how you judge whether a detection is worth keeping switched on.

One service holds up to 500 distinct findings. Past that, further distinct targets fold into a single finding per type, saying so on its row. The executable name is chosen by whoever spawned it, so without a ceiling a compromised process could spawn a thousand uniquely named binaries and push the real shell off the end of a queue nobody can read. Ordinary applications do not come close: these are call sites request handling rarely reaches at all.

Searching

Search by command, endpoint, or service, and filter by severity, type, or status. Two filters matter most during an incident: by trace ID, which ties a finding to the request that caused it, and by reporting host, which shows everything one of your servers filed.

The client address is shown on a finding but is not one of the filters. It lives on the trace rather than on the finding, so filtering by it would mean a join across your trace samples. To see what else one caller did, take the address to the APM page's Clients view and group by IP address. That view is built from trace samples, which are the slowest requests per flush, so it describes the sampled slice rather than all of your traffic.

What is never collected

No request bodies. No command arguments. No SQL text. No file contents.

Only the shape is recorded: the executable that was spawned, the format that was deserialized. An attacker's arguments are the exfiltration, and a monitoring tool that captured them would have moved your breach rather than reported it.

This is also why RootTrace cannot tell you whether a specific user input reached a sink, only that the sink fired inside a specific request. A tool that answers the first question has to hold your customers' request bodies.

What it will not do

It does not block. Nothing here refuses a request. A false positive that takes down your checkout is worse than the attack it prevented, and that judgement belongs to you, not to a monitoring agent.

It is not tamper-proof. The SDK runs inside the process it watches. An attacker who achieves code execution can silence it. Reports leave the host on the normal flush cadence to narrow that window, and the silence itself is reported from the server, where the compromised process cannot reach it. Neither of those makes the SDK trustworthy against someone inside it: nothing in-process can promise to survive the compromise of that process. Treat these signals as one layer, alongside host-level auditing and whatever sits at your perimeter.

It is not a compliance control by itself. It tells you something happened and where it came from. What you do about it is your incident process.

Retention

Findings are kept on the same fixed schedule as the audit trail, independent of your plan's telemetry retention. A finding that expired alongside the profile that produced it would disappear exactly when someone came looking for it.