Collector guide

The RootTrace collector is an outbound-only, read-only diagnostics agent. Each cycle it POSTs a heartbeat and a diagnostics payload to the RootTrace API and closes the connection (Connection: close). It opens no inbound ports, runs no remediation, never restarts services, never mutates Kubernetes resources, never writes customer configuration, and never reads cloud credentials. Linux is the first-class target; the current collector is 0.3.5 on Python 3.12.

None of that has to be taken on trust. The collector is Apache-2.0 and its source is public at byteaffinity/roottrace-collector. It ships as plain Python and is never compiled, on every install method, so the file running on your host is byte-identical to the one you read, which is the point of publishing it rather than describing it.

One scoped collector token (rtc_...) is intentionally reusable across many servers in the same environment: RootTrace identifies each machine from host identity data, so you do not need one token per server. The token maps to the environment server-side; collectors never receive or send an environment id.

Optional third-party libraries (pymongo, psycopg, mysql-connector-python, cassandra-driver, python3-dbus, python3-systemd, playwright, cryptography) are import-guarded: a missing library degrades the matching check to a dependency warning instead of failing the collector or shelling out to unsafe workarounds.

Release channels

The supported release channels are the GPG-signed dnf and apt repositories at https://packages.roottrace.io (dnf repo at /dnf, apt repo at /apt, public signing key at /roottrace.asc) and the RootTrace Agent Amazon ECR Public Gallery for Docker and Kubernetes installs. The Gallery shows the available image tags and pull commands and does not require an AWS account for public pulls. Use a numbered tag to find the release, then deploy its immutable digest:

COLLECTOR_IMAGE='public.ecr.aws/byteaffinity/roottrace/agent@sha256:REPLACE_WITH_DIGEST'
docker pull "${COLLECTOR_IMAGE}"

Do not source the collector from anywhere else; an artifact from any other location is not a RootTrace artifact.

For offline or air-gapped fleets, mirror the package repositories into your internal artifact host and mirror the agent image from the Gallery into your private registry before installing any collectors. Refresh those mirrors on your approved release cadence, together with the published Python and Node.js APM packages and the API server image; the air-gapped install guide shows the full mirror flow.

The Ansible playbook is deployment automation for a trusted release archive or reviewed repository checkout. It is not a download channel and has no updater.

Install

Linux packages (dnf/apt): recommended

Native packages are the recommended install on Linux hosts: the package manager owns every file, upgrades are ordinary dnf update / apt upgrade, and removal takes the collector's files and host integration with it. The packages ship the collector as plain Python source (nothing is compiled), so what runs on the host is byte-identical to what an auditor reads.

RHEL / Rocky / Alma / Amazon Linux (dnf):

sudo curl -fsSL -o /etc/yum.repos.d/roottrace.repo https://packages.roottrace.io/dnf/roottrace.repo
sudo dnf install roottrace-collector

Upgrading later is sudo dnf upgrade roottrace-collector. The repo file sets metadata_expire=1h, so a newly published version appears within the hour. If you are expecting one sooner than that (right after a release, say), refresh the cache rather than waiting:

sudo dnf clean metadata --disablerepo="*" --enablerepo=roottrace

An installation from before that setting existed keeps dnf's 48-hour default until the repo file is fetched again, which is worth doing once.

The repo file enables gpgcheck and repo_gpgcheck; dnf asks once to import the RootTrace signing key from https://packages.roottrace.io/roottrace.asc.

Debian / Ubuntu (apt):

sudo curl -fsSL -o /usr/share/keyrings/roottrace-archive-keyring.gpg https://packages.roottrace.io/roottrace.gpg
echo "deb [signed-by=/usr/share/keyrings/roottrace-archive-keyring.gpg] https://packages.roottrace.io/apt stable main" | sudo tee /etc/apt/sources.list.d/roottrace.list
sudo apt update && sudo apt install roottrace-collector

Then configure and start. Installing deliberately does not start the service, because the token and self-hosted endpoint are not set yet:

sudoedit /etc/roottrace/collector.env
# Set ROOTTRACE_COLLECTOR_TOKEN=rtc_...
# Set ROOTTRACE_API_URL=https://roottrace.example.com/api
sudo roottrace-collector-setup apply         # ACLs, audit plugin, SELinux boolean
sudo systemctl enable --now roottrace-collector.service

Confirm that systemd selected the package-owned unit and that the process starts from /usr:

systemctl show -p FragmentPath roottrace-collector.service
systemctl show -p ExecStart roottrace-collector.service
sudo systemctl status roottrace-collector.service

FragmentPath should be /usr/lib/systemd/system/roottrace-collector.service and ExecStart should use /usr/bin/roottrace-collector.

File locations follow the DISA STIG for RHEL: executables root-owned under /usr/bin, /usr/sbin, and /usr/libexec, never group- or world-writable; configuration root-owned 0600 in /etc; state under /var/lib with nothing executable in it. Nothing installs to /opt, so noexec mounts on /opt, /var, and /tmp never break the collector.

The packaged unit is hardened end to end: collector user with an empty CapabilityBoundingSet, NoNewPrivileges, ProtectSystem=strict, PrivateTmp, kernel protections, RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX, write access only to its state directory, Restart=always, and a planned daily restart via RuntimeMaxSec=86400. It deliberately contains no ExecStartPre updater. Systemd reads /etc/roottrace/collector.env through the unit's EnvironmentFile each time the service starts. Native installs upgrade only through the package manager, and the running service could not modify itself anyway: it runs unprivileged with /usr read-only to it.

roottrace-collector-setup apply performs host integration (read-only log and kubeconfig ACLs, the SELinux nis_enabled boolean, and the auditd realtime plugin), honoring the ROOTTRACE_* toggles read from collector.env.

The broad RHEL 9 STIG audit rules are off by default. Enable ROOTTRACE_LINUX_AUDIT_STIG_RULES=true only after reviewing the generated rules against the host's existing audit policy. The final rule sets audit configuration immutable with -e 2; later changes cannot reach the live kernel rules until the host reboots. Setting the option back to false removes RootTrace's rule files, but it cannot undo an already loaded immutable ruleset before that reboot.

After changing ordinary runtime settings in collector.env (the token, API URL, interval, thresholds, check selection, or service credentials), restart the service so systemd reads the file again:

sudo systemctl restart roottrace-collector.service

Rerun sudo roottrace-collector-setup apply before that restart only when you changed a host-integration setting: log or kubeconfig access, Linux audit rules or the auditd plugin, or the SELinux integration. The helper is idempotent.

Upgrade and remove:

sudo dnf update roottrace-collector        # apt: sudo apt upgrade roottrace-collector
sudo dnf remove roottrace-collector        # apt: sudo apt remove (or purge) roottrace-collector

Removal stops the service and runs roottrace-collector-setup remove before files are deleted, so the ACLs, audit rules, and auditd plugin configuration leave the host with the package. A locally edited config survives dnf remove as /etc/roottrace/collector.env.rpmsave (it can hold credentials you may still need); on Debian the conffile survives apt remove and is deleted by apt purge.

Two host-wide settings are deliberately not reverted on removal, because undoing them could disrupt unrelated services: the SELinux nis_enabled boolean (other software may now rely on it), and the live audit rules on hosts where the STIG finalize rules made the audit config immutable (-e 2). Those rules stay loaded until the next reboot even though the rule files are gone. Reset the boolean by hand with sudo setsebool -P nis_enabled off if nothing else needs it.

Docker

COLLECTOR_IMAGE='public.ecr.aws/byteaffinity/roottrace/agent@sha256:REPLACE_WITH_DIGEST'
docker pull "${COLLECTOR_IMAGE}"
docker volume create roottrace-collector-state
docker run -d --name roottrace-collector \
  --restart unless-stopped \
  --read-only \
  --tmpfs /tmp:rw,nosuid,nodev,size=16m \
  --mount type=volume,source=roottrace-collector-state,target=/var/lib/roottrace-collector \
  --cap-drop ALL \
  --security-opt no-new-privileges \
  -e ROOTTRACE_API_URL=https://roottrace.example.com/api \
  -e ROOTTRACE_COLLECTOR_TOKEN=rtc_... \
  -e ROOTTRACE_STREAMING=false \
  -e ROOTTRACE_MAX_RUNTIME_SECONDS=86400 \
  -e ROOTTRACE_ENABLED_CHECKS=collector_self \
  "${COLLECTOR_IMAGE}"

The image runs as a non-root roottrace user on a read-only filesystem. The named volume preserves acknowledged log and audit offsets across container replacement. ROOTTRACE_MAX_RUNTIME_SECONDS=86400 recycles the collector process once per day, but the restart policy still starts the same container image. To upgrade, pull the next verified digest and recreate the container with that reference; the collector never updates itself.

This minimal Docker command observes the collector container itself. It does not expose the host's /proc, mount table, logs, or service manager. Use the native package for full host diagnostics. If a site builds a host-monitoring container, mount only the required host paths read-only and point ROOTTRACE_PROC_ROOT and ROOTTRACE_DISK_PATHS at those mounts; do not grant a Docker socket or broad write access.

Kubernetes

The following command expects the RootTrace release archive or repository checkout, because the DaemonSet manifest is a shipped file:

The source manifest contains a zero digest so an unreviewed apply fails closed. Render it with the exact verified agent digest from the release manifest:

COLLECTOR_IMAGE=public.ecr.aws/byteaffinity/roottrace/agent@sha256:<agent-digest-from-release-manifest>
kubectl create namespace roottrace --dry-run=client -o yaml | kubectl apply -f -
kubectl -n roottrace create secret generic roottrace-collector \
  --from-literal=ROOTTRACE_COLLECTOR_TOKEN=rtc_... \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl -n roottrace create configmap roottrace-collector-config \
  --from-literal=ROOTTRACE_API_URL=https://roottrace.example.com/api \
  --from-literal=ROOTTRACE_INTERVAL_SECONDS=60 \
  --from-literal=ROOTTRACE_MAX_RUNTIME_SECONDS=86400 \
  --from-literal=ROOTTRACE_STREAMING=false \
  --dry-run=client -o yaml | kubectl apply -f -
sed \
  "s|public.ecr.aws/byteaffinity/roottrace/agent@sha256:0000000000000000000000000000000000000000000000000000000000000000|${COLLECTOR_IMAGE}|" \
  collector/kubernetes/roottrace-collector-daemonset.yaml |
  kubectl apply -f -

The DaemonSet ships a Namespace, ServiceAccount, and a read-only ClusterRole (get/list on pods and events; get on nodes; get/list on HPAs and KEDA ScaledObjects/ScaledJobs; and get on the non-resource /version). The pod runs as uid 65532 with readOnlyRootFilesystem, all capabilities dropped, the host / and /proc mounted read-only, a 16 MiB writable state emptyDir, and a 16 MiB memory-backed /tmp. ROOTTRACE_PROC_ROOT=/host/proc and ROOTTRACE_DISK_PATHS=/host point host checks at the node.

The render replaces only the fail-closed image placeholder and preserves every document in the file, including the ClusterRoleBinding. Do not pass the multi-document file through a workload-only image mutation command; it can discard the non-workload resources.

Linux audit ingestion is disabled in the shipped DaemonSet. Host audit logs are normally root-only, while the collector runs as uid 65532. Enable ROOTTRACE_LINUX_AUDIT_ENABLED only in a site-specific overlay that grants that non-root identity explicit read access to the host audit log; do not make the log broadly readable.

Ansible

This command likewise runs from the release archive or repository checkout:

ansible-playbook -i inventory.ini collector/ansible/install_roottrace_collector.yml \
  -e roottrace_collector_token=rtc_... \
  -e roottrace_api_url=https://roottrace.example.com/api

The playbook copies the exact collector files from that trusted tree and installs the hardened systemd service, ACLs, and audit integration. It never downloads or self-updates the collector, and the token variable is marked no_log.

The STIG rules are also opt-in here:

ansible-playbook -i inventory.ini collector/ansible/install_roottrace_collector.yml \
  -e roottrace_collector_token=rtc_... \
  -e roottrace_api_url=https://roottrace.example.com/api \
  -e roottrace_linux_audit_stig_rules=true

Review the rules first. The same -e 2 reboot constraint described for the native package applies.

Local one-shot test

ROOTTRACE_COLLECTOR_TOKEN=rtc_... \
ROOTTRACE_API_URL=http://localhost:8090/api \
python3 collector/roottrace_collector.py --once

Configuration

Config lives in /etc/roottrace/collector.env (systemd EnvironmentFile format; --env-file loads the same file for non-systemd runs and imports only ROOTTRACE_* and KUBECONFIG keys). Precedence for tunables: host environment variable, then a server-pushed override delivered in the heartbeat response, then the built-in default. Tunables whose names look secret (password, secret, token, key, cred, auth, DSN, URI, URL) are never reported to the server and can never be server-set. Keep ROOTTRACE_CUSTOM_METRICS_PATHS pinned in the host environment for the same reason: a host-set value cannot be overridden remotely.

Core keys

Check selection

ROOTTRACE_ENABLED_CHECKS and ROOTTRACE_DISABLED_CHECKS accept comma-separated check names. An empty enabled list means "everything except the disabled ones". The setup UI defaults to host-safe diagnostics plus Linux audit ingestion; enable service checks only on hosts that actually run those services.

The check reference lists every check name and its tunables: host and threshold checks, Docker/Kubernetes/EKS, the web-server, database, and queue checks, synthetic and browser journeys, secret expiry, and custom Python metrics.

Linux audit ingestion

The linux_audit check ships auditd events for NIST SP 800-171 Rev. 3, CMMC Levels 1-3, DISA STIG/SRG, and ISO/IEC 27001:2022 evidence, either through a realtime auditd plugin (the default) or rotation-safe log tailing. Offsets advance only past batches the API acknowledged, so a transient send failure re-sends those events next cycle instead of losing them.

The RHEL 9 STIG rule set is shipped but off by default, because its final rule makes the running audit configuration immutable until reboot.

Setup, tunables, the STIG opt-in, and the compliance evidence reports are in the Linux audit guide.

Log shipping

ROOTTRACE_LOG_FILES forwards application log lines alongside diagnostics using the service=path list format:

ROOTTRACE_LOG_FILES=api=/var/log/myapp/api.log,worker=/var/log/myapp/worker.log

Tailing is rotation-safe (device/inode/offset state in log_ship_offsets.json under the state directory; ROOTTRACE_LOG_SHIP_STATE_PATH overrides). New files ship from their current end unless ROOTTRACE_LOG_SHIP_READ_FROM_BEGINNING=true; rotated replacements are picked up from the start. Each cycle reads up to 2000 lines or 2 MiB (ROOTTRACE_LOG_SHIP_MAX_LINES_PER_CYCLE, MAX_BYTES_PER_CYCLE). JSON lines contribute level, message, timestamp, and logger plus remaining keys as attributes; plain text gets a level guessed from ERROR/WARN/INFO markers. Batches of at most 500 go to POST /api/logs/ingest with messages truncated at 8 KB.

Redaction happens at source, before anything leaves the host: inline key=value / key: value secrets with sensitive-looking keys have the value replaced (the key stays visible), AWS access key ids and PEM private key blocks are removed, and any line carrying an Authorization/Bearer value is dropped entirely. Ordinary technical text like retry timeout=30s is left intact. Failed batches do not advance the offset, so lines are retried next cycle.

Custom Python metrics

Custom metrics let you collect from services RootTrace ships no built-in check for, using the same ingest path, so dashboards, thresholds, active issues, and incident context work without a separate integration. Enable the custom check and point at local module files or directories:

ROOTTRACE_ENABLED_CHECKS=disk,inode,cpu,memory,collector_self,custom
ROOTTRACE_CUSTOM_METRICS_PATHS=/etc/roottrace/custom_metrics.d

The check reference holds the full programming contract: the collect_metrics(emit) signature, every emit(...) field, the mount patterns for Docker and Kubernetes, and the shipped KEDA monitor.

Host identity

On EC2 the collector uses IMDSv2 (169.254.169.254) to report instance id, availability zone, instance type, region, account id, and instance tags (including Name when metadata tags are enabled). Outside AWS it reports hostname, FQDN, primary IP, architecture, and hashed machine and interface identifiers; raw machine ids and MAC addresses are never sent.

Updates

Updates flow through the same channel the collector was installed from; there is no self-updating downloader in any current install path:

• Native packages: dnf update roottrace-collector / apt upgrade roottrace-collector. Every package and both repos' metadata are GPG-signed, so the package manager verifies each upgrade end to end.
• Containers (Docker, Kubernetes): pull the next verified digest and recreate the Docker container or update the DaemonSet image and wait for its rollout. Restarting an existing container does not switch its image.
• Ansible: obtain and verify a later release archive, then rerun its playbook. The playbook copies from that trusted source tree.

There is no in-process downloader or self-update setting. Release changes remain visible to the package manager or orchestrator and can follow the site's normal approval and rollback process.

Uninstall

sudo dnf remove roottrace-collector        # apt: sudo apt remove (or purge) roottrace-collector

See the removal notes in the install section above: the packages stop the service and run roottrace-collector-setup remove before deleting files.

Transport and security model

Production collector traffic must use outbound HTTPS with certificate and hostname verification. HTTP endpoints are accepted for explicit local development, such as the localhost one-shot example above. A kubeconfig can also weaken verification if it sets insecure-skip-tls-verify; do not use that setting in production. Endpoints: collectors/heartbeat, collectors/ingest, logs/ingest, and collectors/linux-audit/ingest, plus NDJSON streaming variants when streaming is enabled. There is no in-cycle retry: a failed POST is logged and retried next cycle, and log and audit offsets only advance past acknowledged batches.

Every payload passes through redaction before send: secret-looking dict keys, AWS access keys, PEM private key blocks, and strings carrying Authorization:/Bearer values are stripped; URL credentials are masked.

• Unprivileged everywhere: dedicated system user under systemd, uid 65532 in Kubernetes, non-root in Docker; all capabilities dropped.
• Root is used only by the package manager at install/upgrade/remove time and by roottrace-collector-setup; the running service never has it.
• Every check is read-only: databases receive only SELECT/SHOW/PING/INFO class statements; Docker and Kubernetes are queried with GETs.
• Secrets stay host-side: DSNs, passwords, and the token are excluded from tunable reporting, redacted from evidence, and cannot be set or read by the server.

Pairing with the APM SDKs

The collector covers the host and infrastructure side; the APM SDKs instrument your application code. They are designed to be deployed together:

• One token: the SDKs read ROOTTRACE_COLLECTOR_TOKEN as a fallback when ROOTTRACE_APM_TOKEN is not set, so a host that already runs the collector needs no additional credential. Both use the same Authorization: Collector <token> scheme against the same ROOTTRACE_API_URL.
• Set the SDK's URL explicitly. The two differ here, and it matters self-hosted: the collector has no default destination and stops without ROOTTRACE_API_URL, but the SDKs fall back to RootTrace Cloud (https://api.roottrace.io/api). Pass api_url / apiUrl in the SDK's init(...), or export ROOTTRACE_API_URL into the service process, so an application never ships traces anywhere but your own deployment.
• Same ingest plane: SDK payloads go to apm/ingest and logs/ingest next to the collector's collectors/* endpoints; no extra network allowances are needed beyond what the collector already has.
• Correlated dashboards: set the SDK service to the same name you use in ROOTTRACE_SERVICE_NAME on the host, and RootTrace joins application-level latency, errors, and traces with the host's disk, CPU, database, and web-server diagnostics for that service.
• Logs from both sides: collector log shipping (ROOTTRACE_LOG_FILES) tails files on disk, while the SDK log handlers ship in-process records with the active trace_id attached. Use the SDK path when you want logs linked to traces, the collector path for anything that only exists as a file. Both land in the same log store with redaction applied before leaving the host.
• No double counting: the SDKs report application runtime metrics (process.*, nodejs.*, python.* under the service), while the collector reports host metrics; they are attributed separately and do not overlap.

A typical instrumented host runs the collector via systemd or the DaemonSet plus roottrace_apm / roottrace-apm inside each service process, sharing one token and one API URL.