Troubleshooting - Hilt Documentation

BTF / BPF load failures at install time

Kernels 5.8+: If /sys/kernel/btf/vmlinux is missing, the kernel was built without CONFIG_DEBUG_INFO_BTF=y. Upgrade to a kernel with native BTF, or use a distribution that ships it (see System requirements). Kernels 4.18–5.7: Native BTF is not required. Ensure you extracted the full release tarball so the btf/ directory is present alongside the binary. If install still fails, verify the bundled BTF matches your running kernel.

DaemonSet CrashLoopBackOff with BPF load failure

If the DaemonSet enters CrashLoopBackOff with a BPF program load failure, the issue is typically related to missing kernel mounts or insufficient privileges. Ensure that /sys/kernel/btf and /sys/kernel/debug are mounted inside the container and that the pod is running with privileged: true. Verify with:

kubectl describe pod -l app=dag-agent

And by checking filesystem access inside the container.

Agent in “Ship mode” but no events in the dashboard

If the agent logs indicate “Ship mode” but events do not appear in the dashboard:

Verify DNS resolution from the node using getent hosts. If the domain does not resolve, the Private Route 53 association may be missing.
Confirm connectivity. If the request hangs, check that the PrivateLink security group allows traffic from the node’s security group.
Ensure the configured URL is correct: it must include the /ingest path (for example https://<name>-ingest.hilt.ai/ingest). The agent POSTs telemetry to DAG_SHIP_URL verbatim — a bare domain with no path sends events to the wrong place and typically returns 404/405.

High ring buffer drop counts

If high ring buffer drop counts are observed on busy hosts, increase the configured ringbuf_size (for example to 8M or 16M) and restart the agent. Drops can be monitored via verbose mode (-v) or through the agent’s logs, and typically indicate sustained event throughput exceeding the current buffer capacity.

Logs remain after uninstall

If /var/log/dag-agent/events.jsonl remains after uninstall, this is intentional. The uninstall process preserves logs to allow for post-removal analysis or handoff. If a clean host state is required, remove the directory manually.