What is Tetragon (and eBPF)?

Executive Summary

• What: Tetragon is a security observability tool using eBPF. It is already running experimentally in Aurora; we will implement it in The Zone to meet corporate eBPF security standards.
• Why: eBPF-based security solutions are mandatory. Tetragon provides real-time, kernel‑level visibility with minimal overhead.
• Risk: Low – eBPF verifier guarantees safety; Tetragon is already validated in Aurora.
• Cost: Minimal – <1% CPU overhead, ~100-200 MiB memory per node (kernel space, no extra agents).

From research to production-ready deployment.

What is Tetragon?

eBPF-based Security Observability and Runtime Enforcement

• Kernel‑level visibility impossible with userspace tools
• Kubernetes native via Custom Resource Definitions (TracingPolicy)
• Filters and enforces policy directly in the kernel – minimal overhead
• <1% CPU, ~100-200 MiB memory per node

Security monitoring from the kernel up.

Learn more: tetragon.io

What is eBPF?

eBPF (Extended Berkeley Packet Filter): An in‑kernel virtual machine that safely runs user‑supplied programs.

• Runs inside the Linux kernel – no kernel module required
• Verifier guarantees safety before loading (proven mathematically)
• Attaches to system calls, function entry/exit, tracepoints, LSM hooks
• Safety guarantees: no infinite loops, memory safe, resource bounded

Provably safe code in the kernel – verified before loading.

Why Kernel Space Matters

The Principle of Proximity: Move computation to where data resides.

The Problem: Userspace security tools pay a hidden tax:

• Context switches (kernel↔user)
• Data copying
• Latency – by the time userspace sees the event, the moment has passed

Real-world example: NTSYNC (Wine 11)

• Thread sync handled in‑kernel → up to 778% FPS improvement in multi‑threaded games

Move computation to data, not data to computation.

How Tetragon Uses eBPF Hooks

What we monitor:

• Process starts: sys_enter_execve
• File reads: vfs_read
• TCP connections: tcp_connect
• DNS queries: udp_recvmsg (kprobe on DNS traffic, filtered by destination port 53)

In our Kubeflow cluster:

• Jupyter notebooks – detect shell escapes via sys_execve
• Training jobs – catch credential access via vfs_read on /etc/shadow, ~/.kube/config
• Inference endpoints – monitor external connections via tcp_connect

Event Flow & Notifications

Event Lifecycle:

Kernel event → eBPF capture → Kubernetes enrichment → Export → Alert

Notification Paths:

• Critical: PagerDuty → on-call
• High: Slack #security-alerts
• Medium: Email digest
• Low: Daily report

Zone Integration:

ELK stack, existing PagerDuty, Kubernetes dashboards

How We Interact with Tetragon

CLI (tetra)

Real‑time event streaming and debugging from your terminal.

# Stream all events in compact format
tetra getevents -o compact

# Filter by namespace and pod
tetra getevents --namespace default \
  --pods my-app

# Export to JSON for further processing
tetra getevents -o json | jq '.process.exec'

Installation:

go install github.com/cilium/tetragon/tetra@latest

How We Interact with Tetragon

Policies as Code

TracingPolicies are Kubernetes CRDs – defined in YAML, versioned in Git.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: monitor-curl
spec:
  kprobes:
  - call: sys_execve
    syscall: true
    args:
    - index: 0
      type: "string"
    selectors:
    - matchBinaries:
      - operator: "In"
        values:
        - "/usr/bin/curl"

Deployed via GitOps (ArgoCD) – automated sync, version control, rollback.

How We Interact with Tetragon

Export & Dashboards

Exporters:

Elasticsearch, Kafka, gRPC, stdout

Grafana Dashboards:

Build custom visualizations using Elasticsearch as data source.

• Track process executions across namespaces
• Alert on suspicious network connections
• Filter by pod labels, namespaces, or specific syscalls

Example query:

Count of shell executions per namespace in last hour.

How We Interact with Tetragon

gRPC API

Programmatic access for custom integrations (SIEM, automation, alerting).

import grpc
from tetragon import sensors_pb2_grpc, events_pb2

channel = grpc.insecure_channel("localhost:54321")
stub = sensors_pb2_grpc.FineGuidanceSensorsStub(channel)

for event in stub.GetEvents(events_pb2.GetEventsRequest()):
    if event.process_exec:
        if event.process_exec.process.binary == "/bin/bash":
            ns = event.process_exec.process.pod.namespace
            print(f"Shell detected in namespace {ns}")

Use cases:

• Feed events into a custom SIEM
• Trigger automated responses
• Enrich with external threat intelligence

What We Monitor: Key Use Cases

• Credential Access: /etc/shadow, ~/.kube/config – detect theft and lateral movement
• Shell Escapes: /bin/bash from notebooks – container breakout attempts
• Data Exfiltration: Connections to unknown external IPs – potential data theft
• Privilege Escalation: setuid, setgid, capset – prevent privilege escalation
• Sensitive Files: Secrets, tokens, certificates – protect service accounts

Why these matter:

Common MITRE ATT&CK techniques in containerised environments.

Policy Management: Avoiding False Positives

Start in Audit Mode:

Deploy without enforcement, baseline for 1‑2 weeks.

Be Specific:

Use exact paths (/usr/bin/python3) not patterns (*python*).

Namespace‑Specific:

Different baselines for Jupyter vs training vs inference.

Iterative Refinement:

Deploy → observe → refine → repeat.

Document:

Maintain a runbook of known false positives and their remediation.

Policy Management: Maintainable Policies

• Version Control: Store in Git, review via pull requests.
• CI/CD Pipeline: Validate YAML, deploy to DEV, test, promote.
• Documentation: Include purpose, false positives, response, ownership.
• Lifecycle: Review quarterly, remove unused, update for new threats.
• Testing: Simulate events against policies before deployment.

Deployment

Helm Installation

# 1. Add the Cilium Helm repository
helm repo add cilium https://helm.cilium.io
helm repo update

# 2. Create namespace and install Tetragon
helm install tetragon cilium/tetragon \
  --namespace tetragon \
  --create-namespace

# 3. Verify the DaemonSet is running
kubectl -n tetragon get pods

Expected output:

NAME               READY   STATUS    RESTARTS   AGE
tetragon-xxxxx     1/1     Running   0          1m
tetragon-operator  1/1     Running   0          1m

Deploying a TracingPolicy

# Example: detect credential access attempts
cat <<EOF | kubectl apply -f -
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: detect-credential-access
spec:
  kprobes:
  - call: sys_execve
    selectors:
    - matchBinaries:
        - operator: In
          values: [/bin/bash, /bin/sh]
      matchArgs:
        - index: 0
          operator: Contains
          values: [passwd, shadow]
EOF

Verify policy is active:

kubectl get tracingpolicies

Deployment

Viewing Events

Forward the gRPC port and stream events:

# Port‑forward to the Tetragon DaemonSet
kubectl port-forward -n tetragon ds/tetragon 54321:54321

# In another terminal, stream events
tetra getevents -o compact

Example event output:

 process /bin/bash cat /etc/shadow
   pod: default/test-pod, container: app

Export to Elasticsearch: Configure the Helm chart with exporters.elasticsearch.enabled=true.

Implementation Plan

Phase 1 – Validation in DEV

• Deploy Tetragon to Zone DEV AKS
• Apply baseline policies (process exec, file access, network)
• Export events to test Elasticsearch
• Measure overhead (<1% CPU, <200 MiB memory)

Implementation Plan

Phases 2–3 – Production & Automation

Phase 2: Production Readiness

• Define SLOs for event latency & throughput
• Create Terraform module in cloudnative-platform-charts
• Integrate with cluster provisioning scripts

Phase 3: Automation & Handover

• CI/CD pipeline for TracingPolicy updates
• Alerting on critical events (PagerDuty, Slack)
• Documentation and runbooks

Implementation Roadmap

Weeks 1–2 – Deploy & Baseline

• Deploy Tetragon to Zone DEV AKS
• Deploy baseline TracingPolicies (audit mode)
• Export events to test Elasticsearch
• Measure overhead

Implementation Roadmap

Weeks 3–6 – Tune, Integrate, Automate

Weeks 3–4: Tune & Integrate

• Refine policies based on observed events
• Define SLOs (latency <100ms, 99.9% availability)
• Create Terraform module for cloudnative-platform-charts
• Integrate with existing ELK/Splunk

Weeks 5–6: Automate & Alert

• CI/CD pipeline for TracingPolicy updates
• Implement alerting (PagerDuty, Slack, email)
• Document runbooks
• Share findings organization-wide

Conclusion

• What: Tetragon delivers kernel‑level security observability with minimal overhead.
• Why: eBPF is safe, efficient, and mandatory for modern security stacks.
• How: We’ll deploy to Zone DEV, validate, refine policies, and integrate with existing monitoring.
• Next Step: Port the Helm chart and deploy to Zone DEV this sprint.

References

Tetragon & eBPF:

1. Tetragon Documentation
2. eBPF.io – Introduction to eBPF
3. Cilium Tetragon GitHub

Technical Resources:

4. Starovoitov, A. (2014). "BPF: the universal in‑kernel virtual machine." LWN.net.
5. Rice, L. (2020). Learning eBPF. O'Reilly Media.
6. McCanne, S. & Jacobson, V. (1993). "The BSD Packet Filter." USENIX.

References

Linux Kernel:

7. Linux Kernel Source: kernel/bpf/verifier.c

Aurora Implementation:

8. Aurora Platform Charts