Deploy on Kata Containers - Hilt Documentation

Kata Containers (and similar micro-VM runtimes) run each pod inside a lightweight virtual machine with its own guest kernel. The host-level eBPF collector deployed as a DaemonSet cannot observe syscalls across that VM boundary — the same constraint as ECS Fargate. So the same answer applies: the in-workload collector runs inside the guest, wrapped around your application.

How it works

The collector binary is the same dag-fargate-agent used on ECS Fargate; it detects at startup that it is running in a Kubernetes pod rather than a Fargate task. It runs as PID 1 inside your application container, installs a seccomp-BPF filter that traps only the ~20 syscalls Hilt cares about (execve, open, unlink, rename, chmod, connect, etc.), then fork+execs your original application. Every descendant process inherits the filter; everything else runs at native speed, uninstrumented. All of this happens inside the pod’s guest VM — nothing needs to cross the hypervisor boundary. You do not rebuild your application image. An initContainer (the same installer image used for Fargate) copies the agent binary into a pod-scoped emptyDir volume before your app starts; the container’s command is rewritten to launch the agent first, which then launches your original command.

This mechanism is for pods whose runtimeClassName puts them behind a guest kernel (Kata, and by extension other VM-isolated runtimes). Ordinary runc pods on the same cluster should keep using the eBPF DaemonSet — it has strictly more coverage. The two coexist: the DaemonSet simply cannot see Kata pods, and the in-workload agent fills exactly that hole.

Prerequisites

A Kubernetes cluster with a Kata runtime class (e.g. runtimeClassName: kata), x86_64 or arm64
A reachable Hilt ingest endpoint (DAG_SHIP_URL, provided during onboarding)
The installer image URI Hilt gives you at onboarding (the same dag-fargate-installer image used for ECS Fargate). The nodes must be able to pull it.

What gets injected into your pod spec

Change	Detail
Shared volume	A pod-scoped `emptyDir` (`dag-volume`) mounted at `/dag`
initContainer	`dag-fargate-installer` — copies the agent binary into the shared volume, then exits (Kubernetes-native equivalent of the ECS `dependsOn: COMPLETE` gate)
App container command	Rewritten to `command: ["/dag/dag-fargate-agent", "--"]` with your original entrypoint/command in `args`
App container volume mount	`/dag`, read-only
App container capability	`SYS_PTRACE` added — the only capability this mechanism needs
App container environment	`DAG_SHIP_URL`, `DAG_FARGATE_MODE`, `DAG_CLUSTER_ID`, plus downward-API identity fields (below)

If something fails to attach at runtime, the agent fails open: your application always starts and runs normally, with or without telemetry.

Installation

There are two ways to get the collector into Kata pods. Both produce the same pod spec.

	Automatic (recommended)	Manual
How	Deploy the `dag-kata-injector` admission webhook once, next to the eBPF DaemonSet	Patch each workload’s pod template
New pods (scale-up, rollouts, Jobs)	Covered automatically, like the DaemonSet covers runc pods	Covered only for templates you patched
Workload manifests	Unchanged	Edited

Automatic injection (admission webhook)

The dag-kata-injector webhook watches pod creation. Every pod whose runtimeClassName is kata or starts with kata- (kata-qemu, kata-clh, kata-fc, …) is created with the collector already wrapped around its entrypoint. Ordinary runc pods are never sent to the webhook. Pods that already exist are not changed; they pick up the collector the next time they are recreated (a rollout, a reschedule, or a scale-up).

Apply the manifest

Download kata-injector.yaml for your collector version (it ships next to daemonset.yaml), fill in the placeholders, and apply it:

sed -e "s#__REGISTRY__#<REGISTRY>#" \
    -e "s#__TAG__#<TAG>#" \
    -e "s#__SHIP_URL__#https://<name>-ingest.hilt.ai/ingest#" \
    -e "s#__CLUSTER_ID__#<your-cluster-name>#" \
    -e "s#__INSTALLER_IMAGE__#<INSTALLER_IMAGE_URI>#" \
    kata-injector.yaml | kubectl apply -f -

Use the same DAG_CLUSTER_ID as your DaemonSet so Kata pods and node-level telemetry group under one cluster. No cert-manager is needed: the webhook creates its own TLS certificate on first start.

Roll a Kata workload

kubectl rollout restart deployment/<your-kata-deployment>
kubectl get pod <new-pod> -o jsonpath='{.metadata.annotations.dag\.hilt\.ai/inject-status}'

The annotation reads injected mode=full containers=…. If a pod was not injected, it says why.

Safe by default. The webhook never blocks a pod. If it is unavailable, or anything about a pod cannot be handled, the pod is created exactly as submitted. A pod created while the webhook was unavailable runs without the collector until it is next recreated; the webhook runs two replicas to keep that window closed. Other defaults:

Pod Security. In namespaces that enforce the Pod Security baseline or restricted level, SYS_PTRACE is not allowed, so pods there are injected in metadata mode (no capability). If your API server enforces baseline cluster-wide through its admission configuration, set DAG_INJECT_DEFAULT_MODE=metadata on the webhook.
Image entrypoints. Kubernetes command replaces an image’s ENTRYPOINT instead of prepending to it. For containers that don’t set command, the webhook reads the original ENTRYPOINT/CMD from the image registry (public images by default). For private registries, enable DAG_INJECT_PULL_SECRETS=true (and the optional secret-read rule in the manifest), set command explicitly, or annotate the pod:
metadata: annotations: dag.hilt.ai/image-entrypoint.<container>: '["/docker-entrypoint.sh"]' dag.hilt.ai/image-cmd.<container>: '["nginx", "-g", "daemon off;"]'
A container whose entrypoint cannot be determined is left unwrapped rather than started with a guessed command. The pod gets a dag.hilt.ai/inject-skipped annotation and kubectl shows a warning.

Control	Effect
Label a namespace or pod `dag.hilt.ai/inject: disabled`	Never inject
Annotate a pod `dag.hilt.ai/inject: "false"`	Never inject
Annotate a pod `dag.hilt.ai/inject: "true"`	Inject even without a Kata runtime class (for example another VM-isolated runtime)
Annotate a pod `dag.hilt.ai/mode: metadata`	Poll-only mode, no `SYS_PTRACE`
Annotate a pod `dag.hilt.ai/inject-containers: app,worker`	Wrap only the named containers (default: all)
`DAG_INJECT_RUNTIME_CLASSES` on the webhook	Runtime classes to inject (default `kata,kata-*`)

Manual injection (pod template patch)

Use this if you cannot run an admission webhook in your cluster.

Add the volume and initContainer

In your pod template (Deployment, StatefulSet, Job — anything that produces a pod spec):

spec:
  runtimeClassName: kata
  volumes:
    - name: dag-volume
      emptyDir: {}
  initContainers:
    - name: dag-fargate-installer
      image: <INSTALLER_IMAGE_URI>
      volumeMounts:
        - name: dag-volume
          mountPath: /dag

Wrap your container's entrypoint

Rewrite the app container’s command so the agent starts first. If your image relies on its default ENTRYPOINT/CMD, spell them out explicitly — Kubernetes command/args replace, they never prepend.

  containers:
    - name: app
      command: ["/dag/dag-fargate-agent", "--"]
      args: ["<your-original-entrypoint>", "<arg1>", "<arg2>"]
      securityContext:
        capabilities:
          add: ["SYS_PTRACE"]
      volumeMounts:
        - name: dag-volume
          mountPath: /dag
          readOnly: true

Add environment and identity

There is no metadata endpoint inside the guest VM, so the pod spec hands the agent its own coordinates via the downward API:

      env:
        - name: DAG_SHIP_URL
          value: "https://<name>-ingest.hilt.ai/ingest"
        - name: DAG_CLUSTER_ID
          value: "<your-cluster-name>"
        - name: DAG_RUNTIME_CLASS
          value: "kata"
        - name: DAG_POD_NAME
          valueFrom:
            fieldRef: { fieldPath: metadata.name }
        - name: DAG_POD_NAMESPACE
          valueFrom:
            fieldRef: { fieldPath: metadata.namespace }
        - name: DAG_POD_UID
          valueFrom:
            fieldRef: { fieldPath: metadata.uid }
        - name: DAG_NODE_NAME
          valueFrom:
            fieldRef: { fieldPath: spec.nodeName }
        - name: DAG_SERVICE_ACCOUNT
          valueFrom:
            fieldRef: { fieldPath: spec.serviceAccountName }

The agent never contacts the Kubernetes API server — identity is entirely env-driven, so no serviceaccount token or RBAC is needed.

Roll out and verify

Apply the change and check the wrapped container’s logs: the line kata: identity loaded — host_id=<pod-uid> host_type=kata … appears on stderr before your application’s own output.

A complete, runnable reference pod is maintained at deploy/kata/pod-reference.yaml.

Configuration

Variable	Description	Default
`DAG_SHIP_URL`	Hilt ingest endpoint URL	(required)
`DAG_FARGATE_MODE`	`full` or `metadata` — same modes as Fargate	`full`
`DAG_CLUSTER_ID`	Cluster grouping (Kubernetes exposes no cluster name to pods — set this)	(empty)
`DAG_HOST_TYPE`	Host type label; override for other VM-isolated runtimes	`kata`
`DAG_TAGS`	Extra host tags, `key=value,key=value`	(empty)
`DAG_FLUSH_INTERVAL_MS` / `DAG_BATCH_THRESHOLD` / `DAG_MAX_BATCH`	Shipper tuning, same as Fargate	—

Coverage

Matches the ECS Fargate collector — precise exec/file/network-lifecycle events via seccomp+ptrace in full mode, and the metadata kill-switch mode — with one Kata-specific difference: network byte counters are not available. The sampled sock_diag byte counters need INET_DIAG in the kernel serving the pod, and Kata’s default guest kernels are built without it, so on Kata the agent reports network connection lifecycle (who connected where, when) without byte volumes. If your Kata deployment ships a custom guest kernel with CONFIG_INET_DIAG, byte sampling lights up automatically. Data-flow lineage, cgroup lifecycle events, and latency probes are eBPF-collector features and are not available inside VM-isolated pods. Visibility is scoped to the wrapped container’s process subtree — one agent per wrapped container.

Identity mapping

Field	Value on Kata	Dashboard behavior
Host ID	Pod UID	Each pod shows as its own host entity
Host type	`kata`	Distinguished from Kubernetes nodes, VMs, and Fargate tasks in the fleet view
Boot ID	The guest kernel’s real boot ID (per pod-VM lifetime)	Restart detection works as on VMs
Cluster grouping	`DAG_CLUSTER_ID`	Pods group under their cluster
Tags	`pod_name`, `namespace`, `pod_uid`, `node_name`, `service_account`, `runtime_class`	Visible on the host card

Kata pods are as ephemeral as Fargate tasks. Expect turnover in the fleet view; the namespace and pod_name tags (and your own DAG_TAGS) give you stable grouping across pod restarts.

Verification

After deploying, confirm on the Collectors page that:

The pod appears with host_type = kata
It’s grouped under the correct DAG_CLUSTER_ID
The pod_name, namespace, and runtime_class tags are present
Exec and network events are arriving

Upgrade

Update the installer image tag (DAG_INJECT_INSTALLER_IMAGE on the webhook, or the initContainer image in a patched template) and roll the workload. The agent binary is copied fresh from the initContainer on every pod start, so a new pod always picks up the latest agent.

Uninstall

Automatic: kubectl delete -f kata-injector.yaml (and the dag-kata-injector-tls Secret), then roll your Kata workloads. New pods are created without the collector. Manual: Remove the injected pieces (initContainer, emptyDir volume, command rewrite, SYS_PTRACE capability, and DAG_* environment variables) from the pod template and roll the workload.