> ## Documentation Index > Fetch the complete documentation index at: https://hilt.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # Agent filtering > How the collector reduces raw syscall volume to a security-relevant event stream. The collector captures kernel-level telemetry (file access, process execution, network flows) via eBPF. On a busy host, raw syscall volume can exceed **100k events/second**, the vast majority of which is OS housekeeping noise with low governance value. The filtering system reduces this to a manageable stream of security-relevant events through a two-tier architecture. ## Architecture overview | Tier | Where it runs | Purpose | | :- | :- | :- | | **Kernel-space** | Inside eBPF programs on syscall tracepoints | Drops obvious noise before it ever leaves the ring buffer, at zero userspace cost | | **Userspace** | In the collector process after ring buffer read | Applies richer logic (string matching, CIDR math) that isn't feasible in BPF | Events that survive both tiers are serialized and either written to disk or shipped to the ingest endpoint. *** ## What gets filtered The filtering pipeline removes events that have no governance or security value: | Category | Examples | Why filtered | | :- | :- | :- | | **Self-events** | The collector's own I/O | Prevents feedback loops | | **OS housekeeping** | Locale loads, shared library opens, procfs reads | High volume, zero security signal | | **Infrastructure noise** | Kubernetes node-agent file reads, systemd transient units | High volume from well-understood system components | | **Loopback traffic** | localhost and link-local network connections | Internal-only, not governance-relevant (configurable) | | **Repeat reads** | The same process re-reading the same file within 60s (interpreter imports, config polls, log tailers) | The first read ships with full fidelity; only the duplicate edge is deduplicated (see below) | ### Repeat-read coalescing On container-dense hosts most shipped volume is the *same* process re-reading the *same* file. The collector deduplicates these: a successful, read-only open of a path already reported by the same process instance within the coalescing window (default **60 seconds**) is not re-shipped. The first open of every distinct (process, file) pair always ships with its original timestamp — coalescing removes repeats of an already-reported access, never an access edge itself. Never coalesced: write opens, failed opens (access attempts), sensitive-path reads (credentials, tokens, secrets — including customer-defined sensitive paths and detection-critical files such as `/etc/shadow` and `/etc/sudoers`), anything under `/home`, `/root`, or removable/network-share mounts, and network/exec/process events (which are never subject to any volume reduction). Three additional fidelity guards: after a process **writes** any file, its subsequent reads ship per-row again (so read-copy-read sequences keep full evidence); a file **replaced** at the same path is treated as new (inode identity is part of the deduplication key); and an event dropped under queue backpressure never arms the window, so retries remain independent. A keepalive re-emits one coalesced event if the host has been otherwise silent, so quiet hosts never appear offline. On Kubernetes workers this typically removes **30–60%** of shipped rows with no loss of distinct access edges. *** ## What is never filtered Certain event categories bypass all filtering regardless of source: * **All process executions**: the collector always records what was executed, even from filtered processes * **All write operations**: file writes to any path are always security-relevant * **Container lifecycle events**: required for maintaining container identity mapping * **Sensitive path access**: reads of credential files, tokens, secrets, and security-relevant configuration are preserved even from infrastructure processes *** ## Filtering behavior guarantees | Guarantee | Detail | | :- | :- | | **Write-through** | Any file open with write flags always passes; writes are never filtered by path | | **Exec-through** | Process execution events are never filtered by process name | | **Defense in depth** | Kernel and userspace filters are layered; userspace re-checks as a backstop | | **No content inspection** | Filters operate on metadata (path prefix, process name, CIDR), never on file or packet contents | Opt-in **content inspection** (`DAG_CONTENT_INSPECT`) is a separate capture path after a public `DATA_FLOW`; it does not change how events are filtered. See [Data handling](https://hilt.ai/docs/trust-security/data-handling.md#content-inspection). *** ## Metrics and observability The collector tracks filter hit rates and reports them periodically: | Metric | Description | | :- | :- | | `events_total` | Total events received from the ring buffer | | `filtered_self` | Dropped: event came from the collector itself | | `filtered_comm` | Dropped: process name matched infrastructure filter | | `filtered_path` | Dropped: path matched noise category | | `filtered_net` | Dropped: destination matched excluded CIDR | | `passed` | Events that survived all filters (before repeat-read coalescing) | | `ringbuf_drops` | Events the kernel couldn't submit (ring buffer full) | | `dag_agent_open_coalesced_total` | Repeat reads deduplicated by the coalescing window | | `dag_agent_open_coalesce_window_seconds` | Live coalescing window (0 = disabled) | On shutdown, a summary is printed: ```text theme={null} --- Event Stats --- Total: 482031, Filtered: 459218 (95.3%), Passed: 22813 ``` *** ## Configuration | Variable | Default | Description | | :- | :- | :- | | `DAG_EXCLUDE_NET_CIDRS` | *(none)* | Comma-separated CIDRs to exclude from network flow events | | `DAG_DISABLE_DEFAULT_LOCALHOST_FILTER` | `0` | Set to `1` to keep loopback and link-local traffic | | `DAG_OPEN_COALESCE_SECONDS` | `60` | Repeat-read coalescing window; `0` disables (max 600). Also the config key `open_coalesce_seconds`, reloadable via SIGHUP. The environment variable always wins over the config key, including across reloads | | `DAG_OPEN_KEEPALIVE_SECONDS` | `25` | Re-emit one coalesced event after this much output silence; `0` disables (max 59) | | `DAG_CONTENT_INSPECT` | off | Opt-in copy of the source file on public `DATA_FLOW` (100 MiB cap). Never started under `DAG_HOST_PROFILE=latency_critical` | | `DAG_SAMPLE_RATE` | *(none)* | Optional additional 1-in-N sampling of routine non-sensitive file opens (e.g. `FILE_OPEN=8`, max 16). Unlike coalescing, sampling is lossy — distinct paths can be missed — so prefer coalescing first | Coalescing is disabled automatically under `DAG_STRICT_SENSITIVE_CAPTURE=1` (strict sensitive capture) and when a configured local sensitive-path file is missing or invalid (fail-closed), the same gates that govern sampling. The built-in noise filters are compiled into the collector and are not user-configurable. They target well-known system paths and infrastructure processes that generate high volume with no security signal. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.