Patterns, Anti-Patterns, and Operational Reality
How the mesh keeps identity, authorization, and data controls separate in practice
This blog is a part of the Zero-Trust Security on Istio Series, if you haven't read the previous article, please do, it will provide better context.
The previous posts covered what the architecture does and why each layer exists. This post covers what matters when the system is running: the patterns that compose cleanly, the mistakes that widen blast radius, and the operational work that falls out of the design.
Pattern: The Sidecar Contract
The simplest pattern in this architecture is the service contract. A protected application service reads two sidecar-projected headers (x-ms<N>-user, x-ms<N>-role) and does its domain logic. It doesn't validate JWTs. It doesn't call Keycloak. It doesn't manage TLS certificates. It doesn't need to know whether the user authenticated with a session cookie or a bearer token.
async def get_ms2_headers(
x_ms2_user: str | None = Header(None),
x_ms2_role: str | None = Header(None),
x_request_id: str | None = Header(None),
):
if not x_ms2_user or not x_ms2_role:
raise HTTPException(status_code=401, detail="Missing required legacy headers")
return {"user": x_ms2_user, "role": x_ms2_role, "request_id": x_request_id}
This has practical consequences. To add a protected service, you deploy it with a service account, add destination-side header stripping, configure RequestAuthentication, write an AuthorizationPolicy, and read the two projected headers. Gateway-facing services also need a route-to-audience entry in Auth Service. The application still avoids an auth library, OIDC issuer config, and a runtime dependency on Keycloak.
When the identity infrastructure changes (new signing algorithm, new identity provider, new token format), application code doesn't change. The mesh config updates, and services keep reading the same two headers. Resource checks still live in application code, but the identity input stays stable.
The service does not trust arbitrary incoming headers. Each destination sidecar has a Lua filter that strips existing x-ms<N>-user and x-ms<N>-role headers before JWT validation runs. After stripping, outputClaimToHeaders writes fresh values from the verified token. If a compromised internal service sends a request with a spoofed x-ms2-user header, the destination sidecar removes it and replaces it with the value from the signed JWT. The protected service only sees headers authored by its own sidecar.
The tradeoff is dependency on the mesh. If sidecar injection fails, the service has no independent way to verify identity. PeerAuthentication: STRICT and workload AuthorizationPolicy are what make that failure visible instead of silently accepting plaintext traffic. Security is centralized at the infrastructure layer, so the infrastructure has to be treated as part of the application contract.
Pattern: Namespace Segregation by Function
Four project namespaces, plus istio-system for the mesh control plane and ingress gateway:
zt-apps— application services (Profile Aggregator, Employee Records, Device Inventory, Holiday Calendar, Office Directory, Auth Service)zt-identity— Keycloak (OIDC provider)zt-security— Vault, Cerboszt-data— PostgreSQL
Namespaces are RBAC boundaries, network policy boundaries, and resource quota boundaries. An attacker with kubectl access scoped to zt-apps cannot read Secrets in zt-security, where the POC stores the Vault root token for tests. Auth Service's scoped Vault token does live in zt-apps, because it needs signing authority at runtime. That token can sign through Transit and read public keys. It cannot rotate keys, configure Vault, or access other secrets engines. Istio's SPIFFE identity includes the namespace, so AuthorizationPolicy can distinguish "any workload in zt-apps" from "Auth Service running as auth-service-sa."
With everything in default, accidental privilege broadening is easier. RBAC rules, Secrets, service accounts, and network policy exceptions all share the same namespace scope, so blast radius becomes harder to reason about during reviews.
Pattern: Signed Tokens Over Forwarded Credentials
Auth Service (auth-service) validates the user's session cookie or bearer token and then mints a short-lived, purpose-specific mesh token. Downstream services do not receive the original credentials. The session cookie and Keycloak JWT stop at the gateway/Auth Service boundary.
Why this matters:
If a downstream service is compromised, the original credential is absent. Depending on the compromise point, the attacker may see projected identity headers or the short-lived mesh assertion in transit, but they do not get the user's session cookie or Keycloak bearer token. The mesh assertion has a 5-minute TTL and still has to match the destination's
aud,act, and source-principal checks.The mesh token carries the platform claims this architecture uses: user, role, audience, delegation, groups, department, and request metadata. It does not carry session IDs or refresh tokens.
Each token is scoped to a specific set of destination services via the
audclaim. A token minted for the profile flow (Profile Aggregator, Employee Records, Device Inventory) is invalid at Holiday Calendar or Office Directory.
The mistake this avoids is forwarding the original bearer token or session cookie to every downstream service. That gives each service in the call chain access to authentication material that should have stopped at the boundary.
Pattern: Fail-Closed by Default
Protected paths are designed so that missing configuration or failed infrastructure denies access instead of opening it.
Protected workloads have scoped
ALLOWAuthorizationPolicyresources. In Istio, traffic is allowed by default when no policy targets a workload. Once anALLOWpolicy applies, unmatched requests are denied.Auth Service down? ExtAuthz cannot approve protected routes, so the gateway returns 401, 403, or 503 depending on where the failure surfaces.
Vault unreachable? Auth Service cannot mint a new mesh token. The JWKS endpoint returns 500 if it cannot read public keys from Vault, while sidecars may keep using cached keys until refresh.
Cerbos unreachable? Employee Records and Device Inventory fail closed and return 403. Holiday Calendar and Office Directory allow read-only public-data actions during Cerbos outages and deny writes. That is a product decision, not a universal security property.
Sidecar not injected?
PeerAuthentication: STRICTmeans mesh peers reject plaintext connections to that pod. Traffic from outside the mesh still needs KubernetesNetworkPolicyor another boundary if the cluster allows it.
This is verified by a negative-path test that scales the Auth Service deployment to zero replicas and confirms that protected requests do not return 200:
# Scale auth to 0, confirm requests fail-closed
kubectl scale deployment auth-service -n zt-apps --replicas=0
sleep 10
response=$(curl -s -o /dev/null -w "%{http_code}" https://app.localtest.me/api/profile/1)
assert_not_equal "$response" "200"
kubectl scale deployment auth-service -n zt-apps --replicas=1
For protected routes, a failure that returns application data is a security bug.
Anti-Pattern: mTLS-Only Security
mTLS verifies that the calling pod is who it claims to be (via its SPIFFE certificate) and encrypts the connection. That's necessary but not sufficient.
If Holiday Calendar is compromised and your only security layer is mTLS, the attacker has a valid mesh certificate. They can make encrypted, authenticated connections to any other service in the mesh. mTLS tells Employee Records "this connection is from Holiday Calendar," but without AuthorizationPolicy, Employee Records accepts it. The encrypted tunnel between a compromised pod and a sensitive service is still a compromised path.
mTLS is the foundation. It's the transport layer identity that AuthorizationPolicy references. But treating it as the entire security model means any single pod compromise gives the attacker lateral movement across the entire mesh.
Anti-Pattern: Security in Application Code
One common alternative is implementing identity verification and coarse authorization in every service. Each service validates JWTs, wires its own policy checks, and owns the failure behavior for missing or malformed credentials.
The problems compound at scale:
One service has a bug in JWT validation. Maybe it doesn't check
exp. Maybe it accepts tokens from the wrong issuer. That service is now a bypass.The signing algorithm needs to change. You roll it out to 47 services. Three of them are on the old version for a week because the team is busy. During that week, your security posture is inconsistent.
A new developer adds an endpoint and forgets the auth middleware. The code review misses it. That endpoint is now unprotected, and the mesh has no way to catch it because it doesn't know the endpoint should be protected.
With mesh-based security, identity verification and coarse enforcement (mTLS, AuthorizationPolicy, header projection) are expressed in mesh config and enforced by the sidecar. The application developer does not need to remember those checks on every endpoint because they run before application code. Resource-level authorization (Cerbos calls) and response shaping (field masking) still live in application code, but the identity those decisions rely on comes from the sidecar.
Anti-Pattern: Flat Trust Zones
The cluster-internal trust assumption is simple: anything inside the cluster can talk to anything else. The perimeter becomes the main security boundary.
A single compromised pod then becomes a useful foothold. Container escapes, supply-chain issues, and vulnerable dependencies all put the attacker inside the trusted zone. Without service-to-service policy, the network gives them too many next hops.
The tiered architecture limits those next hops. Tier 1 can reach the specific Tier 2 services it needs. Approved services can reach PostgreSQL. Reverse and lateral paths are not part of the allow-list. Istio AuthorizationPolicy enforces that at each destination workload.
Operational Reality: Debugging Auth Failures
Auth failures are easiest to debug by locating the first boundary that rejected the request. The same HTTP status can come from different layers, so start with where the request stopped.
401, 403, or 503 from the gateway: ExtAuthz denied the request, Auth Service was unavailable, Vault signing failed, or the route was not in the coarse policy map. Check the ingress gateway response code first, then Auth Service logs.
401 from the destination service: The sidecar validated the JWT but the service didn't find projected headers. This usually means RequestAuthentication is misconfigured (wrong fromHeaders name, wrong outputClaimToHeaders claim path) or the JWT is missing the expected claims.
403 before the destination service logs anything: The mTLS identity or JWT claims did not match AuthorizationPolicy. Check istioctl proxy-config listener <pod> and the policy when conditions. A common cause is a token whose aud claim does not include the destination service name.
Cerbos denial: The user passed the route and network checks, but the resource-level policy denied the action. Check Cerbos with the exact principal, resource, action, and resource attributes.
Empty result set (RLS): The query returns zero rows. Check what app.current_user_id was set to on the transaction and compare with the RLS policy conditions.
The rule of thumb: start from the gateway and move inward. If the request never reaches the service, investigate ExtAuthz, routing, JWT validation, and AuthorizationPolicy. If it reaches the service, investigate projected headers, Cerbos inputs, and RLS transaction context.
Operational Reality: Sidecar Overhead
Every pod in this mesh runs an Envoy sidecar. That adds CPU and memory per pod, latency per request path, and one more place to inspect when behavior does not match the application logs.
For this POC on a Kind cluster, the overhead is negligible. In production, it is a real cost. Istio ambient mesh (ztunnel + waypoint proxies) addresses part of that cost by moving L4 processing to a node-level daemon and making L7 processing opt-in.
The tradeoff is where L7 policy runs. Sidecar-per-pod gives each workload local L7 enforcement. Ambient mesh moves L4 security and policy to ztunnel; L7 features such as path and method matching, CUSTOM authorization, JWT claim handling, and header projection require waypoint proxies. This architecture depends on L7 behavior at the gateway and protected service hops, so an ambient version would need waypoints for those workloads. The resource savings come from workloads that only need L4 mTLS, peer identity, and connection-level authorization.
Operational Reality: Testing Security Properties
The tests that matter most verify that incorrect behavior is denied. The main negative-path script checks these cases:
Unauthenticated access to protected routes returns 401/403.
External requests with spoofed internal headers are rejected.
Direct access to Tier 2 services from the gateway is impossible.
A non-Profile Aggregator pod calling Employee Records in-cluster is rejected.
Tokens with wrong audience claims are rejected.
Expired tokens are rejected.
Profile Aggregator cannot connect directly to PostgreSQL.
Auth Service outage fails closed for protected routes.
Each test encodes a specific assumption about the security model. If any test returns application data where denial is expected, a security property has been violated. The repository also has a broader 12-category attack-test harness for spoofing, JWT abuse, token replay, path normalization, SSRF, RLS leakage, ExtAuthz fail-open behavior, Cerbos spoofing, mTLS, and Istio config audit. For this POC, I treat these as local release-gate scripts; production needs them wired into CI before mesh config changes can be treated as guarded.
The rotation test verifies the mint-and-rotate path: mint a token, record its kid, rotate the Vault Transit key, mint another token, and confirm the new kid changed. That proves Auth Service fetches the current key version at mint time. It does not, by itself, prove every sidecar refreshed JWKS without a temporary rejection window.
What I'd Do Differently
Vault storage: emptyDir means Vault's state is lost on pod restart. The bootstrap job re-initializes everything, which works for a POC but is unacceptable for production. Persistent storage with auto-unseal is the minimum for real use.
JWKS caching: Auth Service fetches public keys from Vault on every JWKS request. Under load, this is unnecessary. A short-lived cache with forced invalidation on rotation would reduce Vault load. The cache duration should be tied to the token TTL and the rotation process, because stale JWKS is part of the key-rotation failure mode.
Token TTL tuning: 5 minutes is conservative. For internal-only calls from Profile Aggregator to Employee Records, a 60-second TTL would be sufficient and would reduce the replay window. The tradeoff is more frequent token minting for slower operations.
Ambient mesh: I would evaluate ambient mesh service by service. Workloads that only need mTLS and peer authorization are good candidates for ztunnel without sidecars. Workloads that depend on JWT validation, header projection, path/method policy, or ExtAuthz need waypoints. In this architecture, most protected paths depend on L7 behavior, so ambient mesh would reduce overhead only where the workload boundary can stay at L4.
Wrapping Up
Five layers. Eight request checkpoints. The config is complex, but the idea is simple: every hop verifies independently, each layer covers a different failure mode, and application code stays focused on domain logic.
The patterns that make it work are all forms of separation: security enforcement from application code, identity creation from identity verification, coarse authorization from fine-grained authorization, network boundaries from data boundaries. Each separation limits how far one mistake can propagate.
The anti-patterns all collapse those separations: treating one layer as sufficient, spreading identity verification across services, or trusting cluster networking as the security boundary.
