Collector check reference

Every check the collector can run, what it collects, and how to tune it. For installing and operating the agent itself, see the collector guide.

Checks are read-only without exception. Databases receive only SELECT/SHOW/PING/INFO-class statements; Docker and Kubernetes are queried with GETs. Credentials never leave the host, and are redacted out of evidence.

Selecting checks

ROOTTRACE_ENABLED_CHECKS and ROOTTRACE_DISABLED_CHECKS take comma-separated names from this list:

Host       disk  inode  disk_io  swap  memory  cpu  network  tcp  process
           systemd  systemd_timer  kernel  nvidia  ec2  collector_self
Platform   docker  kubernetes  kubernetes_node  eks
Endpoint   http  tls  dns  ping  status_page  synthetic  browser
Service    nginx  apache  haproxy  mongodb  postgres  mysql  redis
           rabbitmq  elasticsearch  clickhouse  cassandra
Other      linux_audit  secret_expiry  custom

An empty ROOTTRACE_ENABLED_CHECKS means "all except the disabled ones". The setup UI defaults to host-safe diagnostics plus Linux audit ingestion. Enable service checks only on hosts that actually run those services.

kubernetes_system_pods is a result type emitted inside the kubernetes check, gated by ROOTTRACE_KUBERNETES_SYSTEM_POD_CHECKS. It is not a selectable check name.

Every check on this page automatically keeps a state history, so each one gets uptime, MTTR, outage patterns, and an optional availability SLO with no extra configuration. A check that runs on a slow interval is fine: the cadence is learned per check, and time when the collector is not reporting is recorded as undetermined rather than counted as uptime. See the availability guide.

Host checks and thresholds

Per-check thresholds follow a ROOTTRACE_<CHECK>_WARN_* / FAIL_* pattern.

Network health tracks TCP socket pressure against the local ephemeral port range, so RootTrace can warn about port starvation before requests start failing.

Platform checks

• Docker: baseline daemon signals (dockerd/containerd visibility, socket presence) always; read-only Docker API inventory (running count, non-zero exits) only when the collector user can already read the socket. RootTrace never changes Docker permissions or executes Docker commands. Tune with ROOTTRACE_DOCKER_API_CHECK, ROOTTRACE_DOCKER_SOCKET, ROOTTRACE_DOCKER_INSPECT_LIMIT, ROOTTRACE_DOCKER_REPORT_LIMIT.
• Kubernetes: uses the in-cluster service account, or a kubeconfig when ROOTTRACE_KUBECONFIG / ROOTTRACE_KUBERNETES_KUBECONFIG / KUBECONFIG is set (kubeconfig wins over the service account; ROOTTRACE_KUBECONFIG_CONTEXT selects a context). Prefer a least-privilege read-only kubeconfig; kubeadm's admin.conf works but normally carries cluster-admin rights. Scans node-local pods across namespaces plus a dedicated kube-system startup check: CrashLoopBackOff, image pull errors, stale ContainerCreating, unscheduled pending pods, failed init containers, and non-running system pods produce a failed kubernetes_system_pods diagnostic that auto-links to one open cluster-level incident.
• kubernetes_node: node conditions and capacity.
• EKS: no AWS credentials involved. When the API version identifies EKS, or ROOTTRACE_EKS_CLUSTER_NAME / ROOTTRACE_EKS_CHECK=true is set, reads version data and kube-system add-on health (aws-node, CoreDNS, kube-proxy, CSI drivers) with the same read-only access.

Service and endpoint checks

Status endpoints behind Basic Auth take matching ROOTTRACE_<SERVICE>_USERNAME / PASSWORD values. Credentials are redacted from evidence.

Nginx log tailing needs read access the installers grant with POSIX ACLs on /var/log/nginx and the configured log paths only. They do not change ownership, chmod logs broadly, or add the collector to web-server groups. Set ROOTTRACE_NGINX_LOG_ACCESS=false before install to skip it.

A missing optional driver degrades that check to a dependency warning. The collector never falls back to a shell workaround.

Synthetic HTTP journeys

ROOTTRACE_SYNTHETIC_JOURNEYS_PATH points at a JSON file of journeys, each with a name and ordered steps. A step takes name, method, url, optional headers and body, an extract map that pulls variables from the response ({"json": "dot.path"} or {"regex": "..."}), and an assert block (status list, body_contains, max_latency_ms). Extracted variables substitute into later steps wherever {var} appears, so a login token can authenticate the next request:

[
  {
    "name": "checkout",
    "steps": [
      {"name": "login", "method": "POST", "url": "https://app.example.com/login",
       "body": {"user": "probe"}, "extract": {"token": {"json": "auth.token"}},
       "assert": {"status": [200]}},
      {"name": "cart", "method": "GET", "url": "https://app.example.com/cart",
       "headers": {"Authorization": "Bearer {token}"},
       "assert": {"status": [200], "body_contains": "items", "max_latency_ms": 800}}
    ]
  }
]

Each journey emits one synthetic diagnostic with total latency_ms as the threshold metric, and per-step latencies, statuses, and assertion failures as evidence. A failure names the failing step. Tune ROOTTRACE_SYNTHETIC_LATENCY_WARN_MS, FAIL_MS, and ROOTTRACE_SYNTHETIC_TIMEOUT_SECONDS. Journeys run on the standard library only.

Browser journeys

ROOTTRACE_BROWSER_JOURNEYS_PATH points at a JSON file of journeys with a name, start_url, and steps of {action, selector?, value?, text?, timeout_ms?} where action is goto, click, fill, wait_for, or assert_text.

Requires Playwright (pip install playwright && playwright install chromium). When it is missing, the collector reports one unknown browser diagnostic instead of failing the cycle. Tune ROOTTRACE_BROWSER_STEP_TIMEOUT_MS, ROOTTRACE_BROWSER_LATENCY_WARN_MS, FAIL_MS.

Secret and key expiry

ROOTTRACE_SECRET_EXPIRY_PATH lists secrets and certificates to track:

[
  {"name": "signing-key", "expires_at": "2026-12-01T00:00:00Z",
   "warn_days": 30, "fail_days": 7},
  {"name": "site-cert", "cert_file": "/etc/ssl/private/site.pem"}
]

Use expires_at for anything with a known ISO 8601 expiry, or cert_file for a local x509 certificate. That is read with cryptography, falling back to openssl x509 -enddate, and reported as unknown when neither is available.

days_remaining is the threshold metric and lower is worse; defaults are warn 30 / fail 7, overridable per entry or globally with ROOTTRACE_SECRET_EXPIRY_WARN_DAYS / FAIL_DAYS.

Only names and expiry dates belong in this file. Never the secret values.

Custom Python metrics

Custom metrics cover services RootTrace ships no built-in check for: internal APIs, queue systems, sidecars, batch processors, appliances. They use the same ingest path as native checks, so dashboards, thresholds, active issues, evidence, and incident context all work without a separate integration.

Enable the custom check and point at local module files or directories:

ROOTTRACE_ENABLED_CHECKS=disk,inode,disk_io,swap,memory,cpu,network,tcp,process,systemd,kernel,ec2,collector_self,custom
ROOTTRACE_CUSTOM_METRICS_PATHS=/etc/roottrace/custom_metrics.d
ROOTTRACE_CUSTOM_METRICS_MAX_FILES=16

Directories load non-recursively; names starting with _ or . are skipped. Modules run in-process as the collector user, so keep them root- or deployment-owned, readable but not writable by the collector user, and limited to read-only APIs. Never store secrets in a custom module. Use least-privilege read-only credentials supplied through the platform's secret mechanism.

A module defines collect_metrics(emit):

# /etc/roottrace/custom_metrics.d/checkout_queue.py
import json
import urllib.request


def collect_metrics(emit):
    with urllib.request.urlopen("http://127.0.0.1:9102/metrics/queue", timeout=2) as response:
        payload = json.loads(response.read().decode("utf-8"))

    emit(
        name="checkout_queue_depth",
        value=payload["depth"],
        label="Checkout queue depth",
        unit="jobs",
        warning=100,
        error=500,
        service="checkout-worker",
        service_type="queue",
        resource="checkout-main",
        resource_label="Queue",
        details={"oldest_job_seconds": payload.get("oldest_job_seconds")},
    )

emit(...) accepts:

A module may instead return a list of such dicts, a plain {metric_name: value} mapping for pass-only metrics, or a fully formed diagnostic result dict with check_type prefixed custom..

One check, several metrics

Keeping check the same and varying name produces an I/O-busy-style drilldown:

def collect_metrics(emit):
    for device, sample in read_storage_backend_metrics().items():
        emit(check="storage_backend", check_label="Storage backend",
             name="io_busy_percent", value=sample["io_util_percent"],
             label="I/O busy", unit="%", warn=70, fail=90,
             resource=device, resource_label="Device")
        emit(check="storage_backend", check_label="Storage backend",
             name="avg_wait_ms", value=sample["avg_wait_ms"],
             label="Avg wait", unit="ms", warn=50, fail=200,
             resource=device, resource_label="Device")

Where modules live

Docker installs bind-mount the module directory read-only:

docker run ... \
  --mount type=bind,source=/etc/roottrace/custom_metrics.d,target=/etc/roottrace/custom_metrics.d,readonly \
  -e ROOTTRACE_ENABLED_CHECKS=...,custom \
  -e ROOTTRACE_CUSTOM_METRICS_PATHS=/etc/roottrace/custom_metrics.d \
  "$COLLECTOR_IMAGE"

Kubernetes installs package modules in a ConfigMap or image layer and mount them read-only. RootTrace's own shipped read-only monitors live under additional_monitors: /usr/libexec/roottrace-collector/additional_monitors for native packages (the path collector.env pins in ROOTTRACE_CUSTOM_METRICS_PATHS), and /app/additional_monitors in containers.

KEDA monitor

The shipped KEDA monitor runs through the custom check and reports ScaledObject/ScaledJob readiness, fallback and pause states, HPA replica gaps, warning events, and operator restarts.

It uses the in-cluster service account, or discovers a kubeconfig via ROOTTRACE_KEDA_KUBECONFIG, ROOTTRACE_KUBERNETES_KUBECONFIG, ROOTTRACE_KUBECONFIG, KUBECONFIG, /etc/roottrace/kubeconfig, then kubeadm/k3s/RKE2/MicroK8s client configs. It auto-starts only when a Kubernetes-family check is enabled and credentials are present. ROOTTRACE_KEDA_ENABLED=true opts in directly and reports a failure when KEDA is expected but unavailable; false keeps it silent. Prefer a dedicated read-only kubeconfig at /etc/roottrace/kubeconfig.