Signing Mesh Tokens with Vault
Why the private signing key stays in Vault, how rotation avoids 403s, and the blast radius when a component is compromised
This blog is a part of the Zero-Trust Security on Istio Series, if you haven't read the previous article, please do, it will provide better context.
The previous post traced one request's identity flow from gateway to database. It left one piece unexplained: how Auth Service actually signs the identity token that carries a user through the mesh. That signing step is the system's root of trust. Steal the signing key and you can forge a token for any user and any role, and every downstream service will believe it.
A quick recap, so this post stands on its own. The system is an HR employee directory running on Istio. A gateway-facing service, Profile Aggregator, fans out to internal-only services like Employee Records (HR data) and Device Inventory (hardware assignments). Auth Service (auth-service) sits at the gateway: it authenticates the external user and mints a short-lived signed JWT, x-mesh-identity (the "mesh token"), that downstream service sidecars verify on every request. It signs that token with Vault, HashiCorp's secrets manager, using Vault's Transit engine, so the private key never touches application code. This post covers why that matters, how it works, and what happens when things go wrong.
The Problem: Secrets in Pods
The default Kubernetes pattern for secrets is to mount them into pods. Environment variables, files in /var/run/secrets, ConfigMaps with sensitive values. Assume any compromised pod leaks whatever secrets it holds. If your signing key is in the Auth Service environment, a container escape gives the attacker the key. They can mint tokens for any user, any role, indefinitely, from anywhere, until you detect the breach and rotate.
The blast radius of a compromised pod should be limited to what that pod can do right now, not what it can do forever with stolen material.
What Vault Solves, and What It Does Not
Token signing is only one use case. In a production cluster, Vault is the central control point for secret access, audit, and rotation across the system. The larger pattern is that applications should not carry durable secret material when the platform can issue short-lived authority from workload identity.
The production version removes static Vault tokens from application config. Auth Service runs under a dedicated Kubernetes service account with a projected, short-lived service-account token. Vault Kubernetes Auth maps that service account to the narrow signing policy. A Vault Agent sidecar authenticates, renews the resulting Vault token, and keeps the raw token inside the Agent boundary. Auth Service calls Vault through the Agent instead of reading a long-lived token from its own environment.
There is still a credential. The difference is that it is platform-issued workload identity, not a manually provisioned Vault token sitting in app config. The remaining credential is issued by Kubernetes, rotated by the platform, scoped by Vault policy, and auditable at Vault.
This POC uses the simpler version: Auth Service receives VAULT_TOKEN directly. That keeps the signing flow easy to see, but it is a demo shortcut, not the endpoint of the pattern.
Vault also does not make a compromised pod harmless. If the Auth Service pod is compromised, the attacker can use whatever runtime authority that pod has until you revoke or cut it off. What Vault changes is the value of that authority. Instead of stealing the RSA private key and signing offline forever, the attacker gets access to a narrow, revocable, auditable signing path. The private key stays inside Vault, every signing operation goes through a central API, and rotation is controlled in one place.
Vault Transit: Signing Without Holding Keys
Vault's Transit engine is a cryptography-as-a-service API. You send it bytes, it signs them with a key it manages, and returns the signature. The key material never leaves Vault's control.
In this architecture, Auth Service uses Transit for one specific operation: signing mesh identity tokens with RS256 (RSA-2048, PKCS#1 v1.5).
The flow:
Auth Service builds the JWT header and payload (claims assembled from the authentication result).
Auth Service encodes the signing input:
base64url(header).base64url(payload).Auth Service sends those bytes to Vault:
POST /v1/transit/sign/mesh-identity.Vault signs with the current RSA private key, returns
vault:v<version>:<base64_signature>.Auth Service strips the prefix, re-encodes as base64url, and assembles the final JWT.
def sign_payload(payload_bytes: bytes) -> str:
"""Signs a payload using Vault Transit and returns the raw signature"""
url = f"{settings.VAULT_URL}/v1/transit/sign/mesh-identity"
headers = {"X-Vault-Token": settings.VAULT_TOKEN}
encoded_payload = base64.b64encode(payload_bytes).decode("utf-8")
data = {
"input": encoded_payload,
"hash_algorithm": "sha2-256",
"signature_algorithm": "pkcs1v15",
}
response = _client.post(url, headers=headers, json=data)
response.raise_for_status()
result = response.json()
# Vault returns the signature as "vault:v<version>:<base64sig>"; strip the prefix for JWS.
sig = result["data"]["signature"]
parts = sig.split(":")
if len(parts) >= 3:
return parts[2]
return sig
Two Vault round-trips happen per token mint: one to fetch the current key version (for the kid header claim), one to sign. In this POC, Auth Service mints a fresh mesh token for every protected gateway request. That keeps the behavior simple and security-first: every request gets a new iat, exp, jti, and signing decision, and key rotation is observed immediately.
The tradeoff is that Vault sits in the synchronous request path. Under production load, you may choose to cache mesh tokens inside Auth Service until their 5-minute expiry, keyed by session, subject, roles, audience, and delegation context. The token still stays inside the platform boundary: it is not returned to the browser, stored in a user cookie, or exposed to frontend code. Caching removes most signing calls for repeated requests from the same user flow.
The cost is freshness: revocation, role changes, and key rotation take effect at token-expiry boundaries instead of per request.
Caching also constrains the token shape. Keep request-specific data such as x-request-id outside the token so the same mesh assertion can safely travel with multiple requests. That would require changing the current POC token, because this implementation includes request_id in the JWT payload.
The JWKS Endpoint: Public Key Distribution
Every service sidecar needs to verify mesh tokens. Auth Service exposes GET /auth/jwks, which fetches all active public key versions from Vault and formats them as a standard RFC 7517 JWKS response:
def get_jwks() -> dict:
"""Format Vault public keys as JWKS"""
keys = vault_client.get_public_keys()
jwks = {"keys": []}
for version, key_info in keys.items():
public_key = serialization.load_pem_public_key(key_info["public_key"].encode("utf-8"))
if isinstance(public_key, RSAPublicKey):
numbers = public_key.public_numbers()
jwks["keys"].append({
"kty": "RSA",
"kid": str(version),
"use": "sig",
"alg": "RS256",
"n": int_to_base64url(numbers.n),
"e": int_to_base64url(numbers.e),
})
return jwks
There's no version filtering in this code, and none is needed. get_public_keys() reads the keys map from vault read transit/keys/mesh-identity, and Vault only returns the working set there: versions from min_decryption_version through the latest. Anything below min_decryption_version is archived and dropped from that response, so it never reaches the JWKS output. The filtering happens inside Vault, not in application code. That makes min_decryption_version the single switch controlling which keys sidecars will accept, and it's why rotation and invalidation are two separate operations, covered next.
This cluster runs istiod with PILOT_JWT_ENABLE_REMOTE_JWKS enabled, so each Envoy sidecar fetches this endpoint directly over mTLS, rather than istiod fetching it once and pushing the keys. A sidecar caches the response and refreshes on its own schedule, on the order of 5 minutes. During rotation, there's a window where a sidecar still holds the old key set. More on that below.
Key Rotation
Rotation happens in two steps: create a new key version, then invalidate old versions.
#!/bin/bash
# Step 1: Rotate — creates a new key version, old versions remain valid
vault write -f transit/keys/mesh-identity/rotate
# Step 2: Invalidate — bumps min_decryption_version so old keys can't verify
vault write transit/keys/mesh-identity/config \
min_decryption_version=$NEW_LATEST
The --grace flag in the rotation script introduces a 300-second (5-minute) wait between these steps. Why 5 minutes? Because mesh tokens have a 5-minute TTL. After one full TTL passes, all tokens signed with the old key have expired. No valid old-key tokens remain in flight, so invalidating the old key causes no disruption.
if [ "$1" = "--grace" ]; then
echo "Grace period: waiting 300s for in-flight tokens to expire..."
sleep 300
fi
vault write transit/keys/mesh-identity/config \
min_decryption_version=$NEW_LATEST
Bumping min_decryption_version removes the old version from the JWKS endpoint, but sidecars don't pick that up instantly. Each one validates tokens against the JWKS it last fetched. A failed validation does not trigger an immediate refetch. The sidecar keeps using that cached key set until its next scheduled refresh, on the order of the 5-minute cache interval above.
That cache lag is exactly why the grace period exists. By waiting one full 5-minute token TTL before bumping the minimum, every token signed with the old key has already expired. After that, it no longer matters whether a given sidecar has refreshed its JWKS. If it still has the old public key cached, there are no valid old-key tokens left to accept. If it has refreshed, it only sees the new key. Skip the grace period and you risk rejecting still-valid old-key tokens the moment a sidecar refreshes mid-flight. Either way, bumping the minimum only archives a version; it stays recoverable by lowering min_decryption_version again, and is gone for good only once it drops below min_available_version.
The grace period covers the old key. The new key has a separate window. Auth Service sets kid to the latest version on every mint, so it starts issuing tokens signed with the new key the instant you rotate. A sidecar whose JWKS cache is still stale has no public key for that kid yet, and rejects those freshly-minted tokens with a 403. So rotation should also trigger a JWKS cache invalidation, forcing every sidecar to refetch immediately and pick up the new public key before any new-key token reaches it. That closes the window so clients never see a 403.
Vault Token Scoping
In this POC, Auth Service holds a Vault token with exactly two capabilities:
updateontransit/sign/mesh-identity(can sign)readontransit/keys/mesh-identity(can read public keys and current version)
It cannot rotate keys, cannot modify key configuration, cannot access any other Vault path. This is the minimum privilege needed for signing and JWKS serving.
The token has a 32-day renewable period. Rotation of the Vault token itself (revoking and reissuing) is a separate operational concern from key rotation. In the production model described earlier, this same policy would be attached to the Vault token issued through Kubernetes Auth and managed by Vault Agent.
Blast Radius: What Happens When Things Break
Auth Service compromised:
The attacker has the pod's signing authority: the direct Vault token in this POC, or access to the Agent-mediated signing path in the production model. They can sign arbitrary payloads under mesh-identity and read public keys. They can mint tokens for any user, any role, with any audience. That breaks the user-identity layer for any service that accepts x-mesh-identity.
It does not automatically bypass every mesh policy. Downstream AuthorizationPolicy rules still check the caller's mTLS workload identity, and the token still has to carry the expected aud (audience: which services may accept the token) and act (the delegation claim naming the authorized intermediary) claims for that route. A forged token from the wrong pod can still fail the source-principal check. The real danger is that the compromised Auth Service can mint tokens that look legitimate to the JWT layer, including tokens with audiences and delegation claims chosen by the attacker.
But the damage is bounded. The attacker cannot rotate keys or modify Vault configuration. You revoke the direct Vault token, or revoke the Agent-issued token and cut off the Kubernetes Auth path for that workload, and their signing ability stops. In-flight tokens they've already minted expire within 5 minutes. You rotate the key and bump min_decryption_version, which drops the compromised version from the JWKS endpoint so sidecars stop accepting anything signed with it once they refresh their cache.
Detection: Vault audit logs show every signing request with the token accessor. Anomalous patterns (sudden spike in signing, unusual audiences) are detectable if audit logging is enabled.
Vault compromised:
This is the worst case, but the severity depends on the compromise depth. If the attacker gains root token access or breaks into the Vault API, they can sign arbitrary tokens through Transit. There's still an audit trail (Vault audit log records each API call), and you recover by rotating the Transit key and revoking the compromised token.
If the attacker extracts the actual key material (storage backend compromise, memory dump, unsealed Vault snapshot), they can sign tokens offline, from anywhere, without making Vault API calls. No audit trail. Tokens they produce are indistinguishable from legitimate ones until you rotate to an entirely new key.
In this POC, Vault uses emptyDir storage (data lives only in the pod's ephemeral filesystem). In production, Vault would use encrypted backend storage with auto-unseal via a cloud KMS. The private key's residency model depends entirely on how Vault is deployed.
A downstream service compromised (Employee Records, Device Inventory, etc.):
The attacker can read projected headers for any request that arrives at that service. They see user IDs and roles for requests they handle. They cannot mint new mesh tokens (no Vault access) and cannot call other internal-only (Tier 2) services (AuthorizationPolicy blocks it).
For database access, the compromised service has its scoped DB role (Employee Records can access hr.* tables, Device Inventory can access it.*). A malicious process could call set_config('app.current_user_id', ...) with an arbitrary value to bypass row-level security (RLS) within its own schema. RLS stops application bugs, not a fully malicious process controlling its own DB session. The containment here is schema isolation (Employee Records can't query it.* tables) and AuthorizationPolicy (Employee Records can't reach Device Inventory's database path).
The Anti-Patterns
Local key pairs: Auth Service generates and holds an RSA key pair. Compromise means the attacker has the key forever (until you detect and rotate). No audit trail of signing operations. No centralized rotation procedure.
Shared HMAC secrets: All services share a symmetric key for signing/verification. Any compromised service can mint tokens. The blast radius of a single pod compromise becomes total. Rotation requires coordinated rollout to every service simultaneously.
Secrets in environment variables: The key material is readable by any process in the container (via /proc/self/environ), visible in crash dumps, and exposed through the container runtime debug surface. Even secretKeyRef values end up as plaintext in the process environment once the pod starts. This is the Kubernetes default for "sensitive config."
Each of these works fine at small scale with a trusted team. They become unacceptable when the question shifts from "can someone on our team see this secret" to "what happens when one pod in a 50-service mesh is compromised."
What's Next
The next post covers authorization in depth: how network-level, request-level, resource-level, and field-level authorization compose into a system where each layer catches what the one above it misses.
