Kubernetes Network Policies vs Service Mesh: Choosing the Right Privacy Layer for AI Traffic

The Traffic Problem in AI Architectures

An AI-powered application is not a monolith. It is a network of services: an API gateway, an inference server, a vector database, a caching layer, a logging pipeline. Each of these services talks to others. Each connection is a potential path for an attacker who has compromised one component to move laterally to another.

Two Kubernetes-native tools address this problem at different layers of the stack: Network Policies and service meshes. Understanding what each one does — and what it cannot do — is essential for teams building AI infrastructure that handles sensitive data.

What Network Policies Do

A Kubernetes NetworkPolicy is a Layer 3/4 firewall rule written in YAML and enforced by the cluster's CNI (Container Network Interface) plugin. It controls which pods can send traffic to which other pods based on pod labels, namespace selectors, and port numbers.

# Allow the API gateway to reach the inference server on port 8080
# Block everything else — including cross-namespace traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: inference-ingress
  namespace: inference
spec:
  podSelector:
    matchLabels:
      app: model-server
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: api-gateway
          podSelector:
            matchLabels:
              app: gateway
      ports:
        - protocol: TCP
          port: 8080

This is powerful and importantly, it is zero-overhead at runtime. Network Policies are enforced by the kernel via iptables or eBPF rules — traffic that does not match the rules is dropped before it reaches the application. There is nothing extra running in your pod.

The default-deny baseline

The single most impactful Network Policy configuration is the default-deny:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: inference
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

With this in place, no traffic can enter or leave any pod in the namespace unless explicitly allowed by another policy. This is the starting point for any AI namespace that processes user data.

What Network Policies cannot do

Network Policies operate at IP and port level. They cannot:

• Verify the *identity* of the caller (only its IP and port)
• Encrypt traffic between pods
• Rate-limit requests at the network layer
• Provide observability into what traffic is flowing
• Enforce HTTP-level policies (e.g., "only GET requests to this path")

If a pod is compromised, an attacker can make requests that look exactly like legitimate traffic — same source IP, same port, same protocol. Network Policies cannot distinguish them.

What a Service Mesh Does

A service mesh (Istio, Linkerd, Cilium Service Mesh) adds a sidecar proxy to each pod. All traffic into and out of the pod routes through this proxy. The proxy enforces policies, encrypts traffic, and records telemetry — all without requiring changes to application code.

The key capability that Network Policies lack is mutual TLS (mTLS). With a service mesh, every service has a cryptographic identity (an X.509 certificate issued by the mesh's CA). When two services communicate, both sides verify each other's identity. A compromised pod cannot impersonate another service.

# Istio AuthorizationPolicy: only the gateway service can call the inference service
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: inference-authz
  namespace: inference
spec:
  selector:
    matchLabels:
      app: model-server
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/api-gateway/sa/gateway-sa"]
      to:
        - operation:
            methods: ["POST"]
            paths: ["/v1/completions", "/v1/chat/completions"]

This policy does what Network Policies cannot: it enforces at the HTTP method and path level and it verifies the cryptographic identity of the caller, not just its IP address.

Head-to-Head Comparison

| Capability | Network Policy | Service Mesh (Istio/Linkerd) |

|---|---|---|

| Layer | L3/L4 (IP/port) | L7 (HTTP, gRPC, paths) |

| Traffic encryption | No | mTLS between all services |

| Caller identity verification | No (IP only) | Yes (X.509 certificates) |

| HTTP-level policy | No | Yes |

| Observability / tracing | No | Yes (metrics, traces, logs) |

| Performance overhead | Near-zero | 2–10ms latency per hop |

| Operational complexity | Low | High |

| Works with any CNI | Requires CNI support | Yes |

| Default-deny posture | Yes | Yes |

The AI-Specific Case for Service Meshes

Inference pipelines have two properties that make service mesh identity verification particularly valuable.

Multiple inference endpoints, one model server. A single model server may expose endpoints for different use cases: a public completion API, an internal batch processing API, and an admin API. Network Policies can control which pods reach the server, but they cannot enforce that the public gateway only hits the public endpoint. An Istio AuthorizationPolicy can.

PII flows through multiple hops. A user's message may travel: API gateway → prompt sanitization service → inference server → response filter → logging service. Network Policies secure the edges of this pipeline. A service mesh secures every internal hop with encryption and identity verification. For applications subject to GDPR or HIPAA, the ability to demonstrate that all data in motion was encrypted — even within the cluster — is significant.

Observability into AI traffic. A service mesh automatically generates traces and metrics for every request. You can see inference latency, error rates by endpoint, and traffic volumes without instrumenting your application code. For AI workloads where latency and token throughput are critical metrics, this is operationally valuable beyond the security benefit.

The Case for Network Policies Alone

Service meshes are operationally complex. Istio adds sidecars to every pod, requires careful upgrade management, can introduce subtle TLS configuration bugs, and adds non-trivial latency that matters for low-latency inference. Linkerd is lighter, but still adds operational overhead that small teams may not want to manage.

Network Policies alone are the right choice when:

• You are running a single-tenant cluster with clear namespace isolation
• Inference latency is critical and sidecar overhead is unacceptable
• Your compliance requirements do not mandate encryption of in-cluster traffic
• Your team does not have experience operating a service mesh

The key is to pair Network Policies with rigorous defaults: default-deny in every namespace, explicit allowlists per service, and RBAC restrictions that prevent Pods from being created without the right SecurityContext.

A Practical Layered Approach

The most defensible AI infrastructure uses both tools at different layers:

1. Network Policies establish the coarse-grained perimeter. Default-deny all namespaces. Explicitly allowlist which namespaces and pod labels can communicate.

1. A service mesh provides fine-grained identity verification and mTLS within the allowlisted paths. Use Istio AuthorizationPolicy or Linkerd's authorization policies to enforce HTTP-level rules and verify caller identity.

1. Vault (not Kubernetes Secrets) manages the credentials and API keys that flow through these encrypted channels.

Each layer addresses a different threat model. Network Policies stop lateral movement at the IP level. The service mesh stops impersonation and adds encryption. Vault ensures the credentials themselves are never stored insecurely.

The Verdict

For teams just securing their first AI workloads, start with Network Policies. The default-deny pattern, namespace isolation, and explicit ingress/egress allowlists address the most common failure modes. This is low-effort, high-impact, and should be non-negotiable.

For teams operating AI infrastructure at scale — especially where PII flows through multiple internal services — a service mesh adds meaningful protections that Network Policies cannot provide: mTLS between all services, cryptographic identity verification, and L7 policy enforcement. The operational overhead is real, but so is the security value for regulated workloads.

---

Continue learning: Explore our Kubernetes Privacy Shields tutorial for hands-on Network Policy and Pod Security Standard implementation.

Test your knowledge: Take the Environments & Security quiz to check your understanding of Kubernetes security controls.