Agent filtering - Hilt Documentation

The collector captures kernel-level telemetry (file access, process execution, network flows) via eBPF. On a busy host, raw syscall volume can exceed 100k events/second, the vast majority of which is OS housekeeping noise with low governance value. The filtering system reduces this to a manageable stream of security-relevant events through a two-tier architecture.

Architecture overview

Tier	Where it runs	Purpose
Kernel-space	Inside eBPF programs on syscall tracepoints	Drops obvious noise before it ever leaves the ring buffer, at zero userspace cost
Userspace	In the collector process after ring buffer read	Applies richer logic (string matching, CIDR math) that isn’t feasible in BPF

Events that survive both tiers are serialized and either written to disk or shipped to the ingest endpoint.

What gets filtered

The filtering pipeline removes events that have no governance or security value:

Category	Examples	Why filtered
Self-events	The collector’s own I/O	Prevents feedback loops
OS housekeeping	Locale loads, shared library opens, procfs reads	High volume, zero security signal
Infrastructure noise	Kubernetes node-agent file reads, systemd transient units	High volume from well-understood system components
Loopback traffic	localhost and link-local network connections	Internal-only, not governance-relevant (configurable)
Repeat reads	The same process re-reading the same file within 60s (interpreter imports, config polls, log tailers)	The first read ships with full fidelity; only the duplicate edge is deduplicated (see below)

Repeat-read coalescing

On container-dense hosts most shipped volume is the same process re-reading the same file. The collector deduplicates these: a successful, read-only open of a path already reported by the same process instance within the coalescing window (default 60 seconds) is not re-shipped. The first open of every distinct (process, file) pair always ships with its original timestamp — coalescing removes repeats of an already-reported access, never an access edge itself. Never coalesced: write opens, failed opens (access attempts), sensitive-path reads (credentials, tokens, secrets — including customer-defined sensitive paths and detection-critical files such as /etc/shadow and /etc/sudoers), anything under /home, /root, or removable/network-share mounts, and network/exec/process events (which are never subject to any volume reduction). Three additional fidelity guards: after a process writes any file, its subsequent reads ship per-row again (so read-copy-read sequences keep full evidence); a file replaced at the same path is treated as new (inode identity is part of the deduplication key); and an event dropped under queue backpressure never arms the window, so retries remain independent. A keepalive re-emits one coalesced event if the host has been otherwise silent, so quiet hosts never appear offline. On Kubernetes workers this typically removes 30–60% of shipped rows with no loss of distinct access edges.

What is never filtered

Certain event categories bypass all filtering regardless of source:

All process executions: the collector always records what was executed, even from filtered processes
All write operations: file writes to any path are always security-relevant
Container lifecycle events: required for maintaining container identity mapping
Sensitive path access: reads of credential files, tokens, secrets, and security-relevant configuration are preserved even from infrastructure processes

Filtering behavior guarantees

Guarantee	Detail
Write-through	Any file open with write flags always passes; writes are never filtered by path
Exec-through	Process execution events are never filtered by process name
Defense in depth	Kernel and userspace filters are layered; userspace re-checks as a backstop
No content inspection	Filters operate on metadata (path prefix, process name, CIDR), never on file or packet contents

Opt-in content inspection (DAG_CONTENT_INSPECT) is a separate capture path after a public DATA_FLOW; it does not change how events are filtered. See Data handling.

Metrics and observability

The collector tracks filter hit rates and reports them periodically:

Metric	Description
`events_total`	Total events received from the ring buffer
`filtered_self`	Dropped: event came from the collector itself
`filtered_comm`	Dropped: process name matched infrastructure filter
`filtered_path`	Dropped: path matched noise category
`filtered_net`	Dropped: destination matched excluded CIDR
`passed`	Events that survived all filters (before repeat-read coalescing)
`ringbuf_drops`	Events the kernel couldn’t submit (ring buffer full)
`dag_agent_open_coalesced_total`	Repeat reads deduplicated by the coalescing window
`dag_agent_open_coalesce_window_seconds`	Live coalescing window (0 = disabled)

On shutdown, a summary is printed:

--- Event Stats ---
Total: 482031, Filtered: 459218 (95.3%), Passed: 22813

Configuration

Variable	Default	Description
`DAG_EXCLUDE_NET_CIDRS`	(none)	Comma-separated CIDRs to exclude from network flow events
`DAG_DISABLE_DEFAULT_LOCALHOST_FILTER`	`0`	Set to `1` to keep loopback and link-local traffic
`DAG_OPEN_COALESCE_SECONDS`	`60`	Repeat-read coalescing window; `0` disables (max 600). Also the config key `open_coalesce_seconds`, reloadable via SIGHUP. The environment variable always wins over the config key, including across reloads
`DAG_OPEN_KEEPALIVE_SECONDS`	`25`	Re-emit one coalesced event after this much output silence; `0` disables (max 59)
`DAG_CONTENT_INSPECT`	off	Opt-in copy of the source file on public `DATA_FLOW` (100 MiB cap). Never started under `DAG_HOST_PROFILE=latency_critical`
`DAG_SAMPLE_RATE`	(none)	Optional additional 1-in-N sampling of routine non-sensitive file opens (e.g. `FILE_OPEN=8`, max 16). Unlike coalescing, sampling is lossy — distinct paths can be missed — so prefer coalescing first

Coalescing is disabled automatically under DAG_STRICT_SENSITIVE_CAPTURE=1 (strict sensitive capture) and when a configured local sensitive-path file is missing or invalid (fail-closed), the same gates that govern sampling. The built-in noise filters are compiled into the collector and are not user-configurable. They target well-known system paths and infrastructure processes that generate high volume with no security signal.