HashiCorp Vault vs Kubernetes Secrets: Which Should You Use for AI Workloads?

The Choice Every Platform Team Faces

When you move an AI application to Kubernetes, one of the first questions is: where do the secrets live? Your OpenAI API key, your database password, your Anthropic billing credential — these are high-value targets. The wrong answer is a .env file on a server. The right answer is less obvious.

Kubernetes ships with a built-in Secrets resource. HashiCorp Vault is a dedicated secrets management platform that can integrate with Kubernetes. They are not mutually exclusive — many teams use both. But understanding what each does well, where each falls short, and what the stakes are for AI workloads specifically will help you make a deliberate choice rather than defaulting to whatever is most convenient.

What Kubernetes Secrets Actually Are

A Kubernetes Secret is a first-class API object that stores up to 1MB of key-value data. Under the hood, it is a base64-encoded record in etcd — the cluster's key-value store.

apiVersion: v1
kind: Secret
metadata:
  name: openai-credentials
  namespace: inference
type: Opaque
stringData:
  api-key: "sk-your-key-here"

Pods consume secrets as environment variables or as files mounted into the container. The API is simple, the tooling support is universal, and there is zero infrastructure to provision. It is the default choice for good reason.

The problems with Kubernetes Secrets at scale

Base64 is not encryption. Anyone with read access to etcd can decode your secrets in milliseconds. By default, etcd stores Secret data in plain base64. Encryption at rest requires deliberately enabling EncryptionConfiguration — most clusters do not have it enabled out of the box.

RBAC is coarse. You can restrict which ServiceAccounts can read which Secrets, but you cannot restrict access to individual keys within a Secret. If a pod can read the Secret, it can read everything in it.

No native rotation. Kubernetes Secrets have no built-in concept of secret rotation. Rotating an API key means manually updating the Secret, which may or may not trigger a pod restart depending on how the Secret is consumed.

No audit trail. Kubernetes audit logging can record when a Secret was accessed, but it requires cluster-level configuration that many teams skip. There is no native "who read this secret and when" visibility.

No versioning. If you rotate a key and something breaks, there is no one-command rollback to the previous value.

What HashiCorp Vault Actually Is

Vault is a dedicated secrets management service. It stores secrets, generates secrets dynamically, enforces fine-grained access policies, rotates secrets on a schedule, and records every access in a tamper-evident audit log. It is a standalone service that your cluster communicates with.

# Vault stores and retrieves secrets
vault kv put secret/ai/openai api-key="sk-prod-key"
vault kv get -field=api-key secret/ai/openai

# Vault generates temporary database credentials on demand
vault read database/creds/inference-role
# → username: v-inference-a1b2c3 (expires in 1 hour)

# Vault records every access
vault audit enable file file_path=/var/log/vault/audit.log

What Vault does that Kubernetes Secrets cannot

True encryption. Vault encrypts all data before writing it anywhere. The encryption happens in Vault's core before data reaches any storage backend. This is not a configuration toggle — it is the default.

Dynamic secrets. Instead of storing a database password that never changes, Vault generates a unique username and password for each workload with a configured TTL. The credentials expire automatically. If they are compromised, they are already worthless by the time an attacker tries to use them.

KV v2 versioning. Every write to a Vault KV path creates a new version. You can see the full history, diff versions, and roll back to any previous version in one command. Zero-downtime key rotation becomes a routine operation.

Comprehensive audit log. Every read, write, and delete is recorded with the identity of the caller and a timestamp. Secret values are HMAC-hashed so they are never exposed in logs. This is exactly what GDPR, SOC 2, and HIPAA auditors want to see.

Kubernetes auth method. Pods authenticate to Vault using their Kubernetes ServiceAccount token — no credentials need to be injected. There is no "secret zero" problem.

Head-to-Head Comparison

| Capability | Kubernetes Secrets | HashiCorp Vault |

|---|---|---|

| Encryption at rest | Optional (requires config) | Always on |

| Dynamic credentials | No | Yes |

| Secret versioning / rollback | No | Yes (KV v2) |

| Audit log | Requires cluster audit setup | Built-in, per-secret |

| Fine-grained access control | Namespace + resource name | Per-path, per-key, per-capability |

| Secret rotation | Manual | Automated |

| Zero-infra setup | Yes | No (Vault is a service) |

| Works without Kubernetes | No | Yes |

| Learning curve | Low | Medium |

The AI Workload Dimension

AI applications raise the stakes in two specific ways that make this comparison more pointed.

LLM API keys are unusually valuable. A compromised OpenAI or Anthropic API key does not just leak data — it runs up a billing tab. A stolen key used for even a few hours of aggressive inference can generate thousands of dollars in charges. The faster you can detect a compromise and revoke the key, the smaller the damage. Vault's audit logging gives you detection. Vault's KV v2 versioning gives you instant rotation with one command.

AI workloads process PII. Models that handle user data — chat history, document content, health queries — process PII that may be subject to GDPR, HIPAA, or CCPA. Compliance frameworks require demonstrable evidence that only authorized services accessed the credentials protecting that data. "We used Kubernetes Secrets" is a harder story to tell to an auditor than "every secret access is recorded in Vault's audit log with service identity and timestamp."

AI-generated infrastructure may introduce misconfigurations. When AI coding tools generate Kubernetes YAML, they frequently omit encryption configuration, grant overly broad RBAC permissions, or reference Secrets in ways that violate least-privilege principles. Vault's per-path policies are harder to accidentally misconfigure than Kubernetes RBAC over Secrets.

The Case for Using Both

These tools are not competitors — they are complementary layers.

A common production pattern is to use Vault as the source of truth for all secrets and use Vault Agent (running as a sidecar or init container) to inject secrets into Kubernetes pods as environment variables or files. The pod sees a standard environment variable. It never knows or cares that Vault is involved. Kubernetes Secrets are used only for non-sensitive configuration.

# Pod annotated for Vault Agent injection
metadata:
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "inference-role"
    vault.hashicorp.com/agent-inject-secret-config: "secret/data/ai/openai"

This pattern gives you the operational simplicity of Kubernetes-native secret consumption combined with Vault's security guarantees, rotation, and audit logging.

When Kubernetes Secrets Alone Is Enough

Vault has operational overhead. You need to deploy it, secure it, monitor it, and back it up. For teams early in their Kubernetes journey, or for non-production environments, Kubernetes Secrets with encryption at rest enabled is a reasonable baseline.

The checklist for making Kubernetes Secrets acceptable:

• Encryption at rest is enabled via EncryptionConfiguration
• Cluster-level audit logging is enabled and secrets access events are captured
• RBAC policies follow least privilege with named resourceNames restrictions
• A documented rotation procedure exists and has been tested
• Secrets are never committed to Git in any form

The Verdict

For AI workloads in production, Vault is the better long-term investment. The features that matter most for AI — audit logging, dynamic credentials, zero-downtime rotation, and fine-grained access — are either unavailable or require significant manual configuration in Kubernetes Secrets alone.

For teams getting started, starting with Kubernetes Secrets and enabling encryption at rest is a pragmatic choice. Plan the migration to Vault before you hit production scale — retrofitting secrets management is harder than building it right the first time.

---

Continue learning: Explore our HashiCorp Vault tutorial and Kubernetes Privacy Shields tutorial for hands-on implementation details.

Test your knowledge: Take the Environments & Security quiz to check your understanding of secrets management best practices.