Vault

S4E On-Prem integrates with HashiCorp Vault for centralized secrets management. This integration eliminates the need to store sensitive values in Kubernetes Secrets, Helm values files, or environment variables in plain text.

Overview

HashiCorp Vault provides:

  • Centralized secret storage -- passwords, API keys, and certificates in one secure location.
  • Dynamic secrets -- auto-generated, short-lived database credentials.
  • Audit logging -- every secret access is logged for compliance.
  • Access policies -- fine-grained control over which services can access which secrets.
  • Encryption as a service -- transit encryption without managing keys.

Architecture

+-------------+       +------------+       +-----------+
| S4E Pods    | ----> | Vault      | ----> | Secret    |
| (sidecar or |       | Server     |       | Backend   |
|  init)      |       +------------+       | (Consul,  |
+-------------+              |             |  File,    |
                             |             |  Raft)    |
                      +------v-------+     +-----------+
                      | Audit Log    |
                      +--------------+

S4E services access Vault through one of two methods:

  1. Vault Agent Sidecar Injector -- a sidecar container fetches secrets and writes them to a shared volume.
  2. Init Container -- an init container fetches secrets before the application starts.

Prerequisites

  • HashiCorp Vault server (v1.12+) deployed and unsealed.
  • Vault Agent Injector installed in the Kubernetes cluster.
  • Kubernetes authentication method enabled in Vault.
  • Network connectivity from S4E pods to the Vault server.

Setting Up Vault

Step 1: Enable the KV Secrets Engine

vault secrets enable -path=s4e kv-v2

Step 2: Store S4E Secrets

vault kv put s4e/database \
  host=postgresql.s4e.svc.cluster.local \
  port=5432 \
  name=s4e_production \
  username=s4e_app \
  password="<strong-db-password>"

vault kv put s4e/redis \
  connection_string="redis://redis:6379" \
  password="<redis-password>"

vault kv put s4e/rabbitmq \
  host=rabbitmq.s4e.svc.cluster.local \
  port=5672 \
  username=s4e_mq \
  password="<rabbitmq-password>"

vault kv put s4e/app \
  secret_key="<application-secret>" \
  smtp_password="<smtp-password>"

Step 3: Configure Kubernetes Authentication

vault auth enable kubernetes

vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc:443" \
  token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Step 4: Create a Vault Policy

# s4e-policy.hcl
path "s4e/data/database" {
  capabilities = ["read"]
}

path "s4e/data/redis" {
  capabilities = ["read"]
}

path "s4e/data/rabbitmq" {
  capabilities = ["read"]
}

path "s4e/data/app" {
  capabilities = ["read"]
}

vault policy write s4e-read s4e-policy.hcl

Step 5: Create a Kubernetes Auth Role

vault write auth/kubernetes/role/s4e \
  bound_service_account_names=s4e-core,s4e-trigger,s4e-scan,s4e-crawler \
  bound_service_account_namespaces=s4e \
  policies=s4e-read \
  ttl=1h

Configuring S4E to Use Vault

Method 1: Vault Agent Sidecar Injector

Add annotations to S4E pod templates to inject secrets:

# In s4e-values.yaml
core:
  podAnnotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "s4e"
    vault.hashicorp.com/agent-inject-secret-db: "s4e/data/database"
    vault.hashicorp.com/agent-inject-template-db: |
      {{- with secret "s4e/data/database" -}}
      export DB_HOST={{ .Data.data.host }}
      export DB_PORT={{ .Data.data.port }}
      export DB_NAME={{ .Data.data.name }}
      export DB_USER={{ .Data.data.username }}
      export DB_PASS={{ .Data.data.password }}
      {{- end }}

The sidecar writes secrets to /vault/secrets/db, which the application sources on startup.

Method 2: External Secrets Operator

If you use the External Secrets Operator (ESO), configure an ExternalSecret to sync Vault secrets into Kubernetes Secrets:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: s4e-db-secrets
  namespace: s4e
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: s4e-db-credentials
    creationPolicy: Owner
  data:
    - secretKey: DB_HOST
      remoteRef:
        key: s4e/data/database
        property: host
    - secretKey: DB_PASS
      remoteRef:
        key: s4e/data/database
        property: password

External Secrets Operator

ESO is recommended when you want Vault secrets to appear as standard Kubernetes Secrets, allowing seamless integration with existing Helm charts and pod configurations.

Dynamic Database Credentials

Vault can generate short-lived PostgreSQL credentials automatically:

Enable the Database Secrets Engine

vault secrets enable database

vault write database/config/s4e-postgres \
  plugin_name=postgresql-database-plugin \
  allowed_roles="s4e-app" \
  connection_url="postgresql://{{username}}:{{password}}@postgresql.s4e.svc:5432/s4e_production?sslmode=disable" \
  username="vault_admin" \
  password="<vault-admin-password>"

vault write database/roles/s4e-app \
  db_name=s4e-postgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT ALL ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

Connection pool considerations

Dynamic credentials rotate periodically. Ensure your application handles credential rotation gracefully by reconnecting when authentication fails.

Secret Rotation

Automatic Rotation with Vault Agent

Vault Agent automatically renews and re-fetches secrets before they expire. No application changes are required when using the sidecar injector method.

Manual Rotation

To rotate a secret manually:

vault kv put s4e/database \
  host=postgresql.s4e.svc.cluster.local \
  port=5432 \
  name=s4e_production \
  username=s4e_app \
  password="<new-password>"

Then restart the affected pods to pick up the new values:

kubectl -n s4e rollout restart deployment/s4e-core
kubectl -n s4e rollout restart deployment/s4e-trigger

Troubleshooting

Issue Cause Solution
permission denied in Vault Agent logs Incorrect role or policy binding Verify bound_service_account_names and policy paths
Secrets file not created in pod Vault Agent Injector not installed Install the Vault Agent Injector Helm chart
403 Forbidden on secret access Service account not authorized Check the Kubernetes auth role configuration
Expired credentials TTL too short Increase default_ttl in the database role

Next Steps