# The Identity Flow: From Gateway to Database * * * This blog is a part of the Zero-Trust Security on Istio Series, if you haven't read the previous article, please do, it will provide better context. %[https://blogs.kaustavsarkar.dev/series/zero-trust-istio] The previous post laid out the five security layers as a static architecture. This post follows a single request through them, down to the question that matters in practice: what does a service actually trust, and why is trusting it safe? The system is an HR employee directory, built to be attacked. Five microservices sit behind the mesh, numbered `ms1` through `ms5`: Profile Aggregator (`ms1`) is the gateway-facing orchestrator; Employee Records (`ms2`) holds PII and salary data and Device Inventory (`ms3`) holds hardware assignments, both internal-only; Holiday Calendar (`ms4`) and Office Directory (`ms5`) serve holiday and office-location data, most of it public. Auth Service mints a signed identity token, `x-mesh-identity`, that carries the user through the mesh, and Vault signs it. This post traces one concrete request: an employee, Alice, requests her own profile with `GET /api/profile/{id}`. That request reaches Profile Aggregator, which fans out to Employee Records and Device Inventory. I'll follow the `ms1` to `ms2` hop, Profile Aggregator calling Employee Records, so every `ms2-*` resource below is that running example, not a special case. Employee Records reads the user's identity from a plain HTTP header, eg `x-ms2-user`. The header is named per service (`ms2` reads `x-ms2-user`, `ms3` reads `x-ms3-user`) so one service can never consume another's identity headers. Employee Records does no JWT parsing and makes no callback to an auth service. It reads that header and treats the value as the authenticated user. That should look fragile, because headers are trivially spoofable. Any external client can send `x-ms2-user: alice`, and so can any compromised pod inside the cluster. It works because of everything that happens before a service reads that header. A single request moves through eight steps from the gateway to the database, each adding one guarantee: 1. **Header stripping** at the gateway. 2. **Authentication** of the caller's credentials. 3. **Route-level authorization** against a coarse policy map. 4. **Token minting**, signed via Vault. 5. **Token validation and header projection** at the destination sidecar. 6. **AuthorizationPolicy**, binding peer identity to token claims. 7. **The service contract**, the two headers the application actually reads. 8. **Database identity context** for row-level enforcement. Six of those steps run in the mesh before the application sees the request. By the time Employee Records reads `x-ms2-user`, its value has been stripped of anything the caller sent and re-derived from the signed token the destination sidecar verified. The last two steps are the service's own contract and the database scoping it then sets up. ![The identity flow: eight checkpoints from the Istio gateway to the PostgreSQL row-level security query](https://cdn.hashnode.com/uploads/covers/6a0c37b1fd930acd9e15653e/36558b98-f8f8-4da1-8ef3-327a1f3eaa47.png align="center") ## The Trust Boundary The Istio Ingress Gateway is where external input is converted into internal identity. Everything before it is untrusted. An external client can send any header it wants, including headers that look exactly like internal identity assertions. The gateway's job is to strip those, authenticate the caller, and produce a cryptographically signed token that downstream services can verify independently. After the gateway, a service trusts only the headers its own destination sidecar wrote. The sidecar does the verification itself: it strips any inbound copy of the projected headers (like `x-ms2-user`), validates the JWT signature, checks the issuer, and verifies expiry, then writes the claims into fresh headers. The application reads those headers directly and runs no verification of its own, because the sidecar already ran it. That trust is narrowly scoped. It covers only the headers the sidecar itself wrote after validating the token. Anything that merely arrived on the request, from an external client or an internal pod, is ignored. So if a compromised internal pod sends a spoofed `x-ms2-user` header, the destination sidecar strips it before the JWT filter runs and re-projects the value from the verified token. The spoofed header never reaches the application. This model has a clear contract: the gateway is responsible for identity creation, sidecars are responsible for identity verification, and services are responsible for business logic. If any of those three fail in isolation, the other two still limit the blast radius. ## Step 1: Header Stripping Before authentication, before routing, before anything else, a Lua EnvoyFilter removes every internal trust header from the incoming request: ```yaml apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: name: gateway-prestrip namespace: istio-system spec: priority: -100 workloadSelector: labels: istio: ingressgateway configPatches: - applyTo: HTTP_FILTER match: context: GATEWAY listener: filterChain: filter: name: "envoy.filters.network.http_connection_manager" subFilter: name: "envoy.filters.http.ext_authz" patch: operation: INSERT_BEFORE value: name: envoy.filters.http.lua.prestrip typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua inlineCode: | function envoy_on_request(request_handle) local headers_to_remove = {} for key, value in pairs(request_handle:headers()) do local lower_key = string.lower(key) if string.match(lower_key, "^x%-platform%-") or string.match(lower_key, "^x%-role%-") or lower_key == "x-mesh-identity" or string.match(lower_key, "^x%-ms%d%-user") or string.match(lower_key, "^x%-ms%d%-role") then table.insert(headers_to_remove, lower_key) end end for _, key in ipairs(headers_to_remove) do request_handle:headers():remove(key) end end ``` The filter runs at priority `-100` with `INSERT_BEFORE` the ExtAuthz filter. The order is load-bearing: if ExtAuthz ran first and injected `x-mesh-identity` based on a valid session, but a spoofed `x-ms2-user` header also survived from the client, the downstream service would read the attacker-controlled value. The two-pass approach (collect keys, then remove) is required because mutating headers during iteration is undefined behavior in Envoy's Lua API. What gets stripped: `x-platform-*`, `x-role-*`, `x-mesh-identity`, and all `x-ms-user` / `x-ms-role` headers. Case-insensitive matching ensures no variant slips through. ## Step 2: Authentication (ExtAuthz) After stripping, the request hits the ExtAuthz filter. This calls Auth Service (`auth-service`), which resolves the caller's identity from one of two sources: **Bearer token:** Extracted from the `Authorization` header, validated against Keycloak's JWKS endpoint. Roles come from `realm_access.roles` in the Keycloak JWT. **Session cookie:** An opaque `zt_session` cookie, looked up in PostgreSQL. The session record holds the user's roles from the original OIDC flow. Both paths produce the same output: a subject identifier, username, email, roles, groups, and department. The subject is deterministic across both auth paths. It's computed as `uuid5(NAMESPACE_URL, "istio-security://users/")`, which means the same user always gets the same `sub` regardless of whether they authenticated via cookie or bearer token. This is important for RLS (which filters by `sub`) and for audit trails. ## Step 3: Route-Level Authorization Before minting a token, Auth Service checks a coarse policy map. This is the first authorization gate: ```python # Coarse route/role policy map (ext_authz.py) if path.startswith("/api/profile/"): allowed_roles = {"employee", "manager", "hr_admin", "it_admin"} decision["audience"] = ["ms1-profile-aggregator", "ms2-employee-details", "ms3-hardware-assets"] elif path.startswith("/api/holidays") and method == "GET": allowed_roles = {"employee", "manager", "hr_admin", "it_admin", "public_data_admin", "security_auditor"} decision["audience"] = ["ms4-holiday-calendar"] elif path.startswith("/api/holidays") and method in ("POST", "PUT", "DELETE", "PATCH"): allowed_roles = {"public_data_admin", "hr_admin"} decision["audience"] = ["ms4-holiday-calendar"] else: decision["reason"] = "Route not defined in coarse policy map" return decision user_roles = set(decision["roles"]) if not allowed_roles.intersection(user_roles): decision["reason"] = "Unauthorized role for route" return decision ``` If the user's role doesn't match the route, the request is denied with 401. No token is minted. The `audience` field determines which services this token will be valid for. A token for the profile endpoint is valid at Profile Aggregator (`ms1-profile-aggregator`), Employee Records (`ms2-employee-details`), and Device Inventory (`ms3-hardware-assets`). A token for the holidays endpoint is valid only at Holiday Calendar (`ms4-holiday-calendar`). This prevents a valid token from one flow being replayed against a different service. Any path not in the policy map is denied by default. `GET /api/offices` is deliberately absent, because those reads are public: the gateway's `AuthorizationPolicy` exempts them from ExtAuthz, so an unauthenticated read reaches Office Directory (`ms5-office-locations`) without a token and is treated as `anonymous/public`. Writes to `/api/offices` do go through the map above and require `public_data_admin`. ## Step 4: Token Minting If authentication and route authorization both pass, Auth Service mints a signed mesh token. The JWT is built manually (no library) and signed via Vault Transit: ```python header = {"alg": "RS256", "typ": "JWT", "kid": str(vault_key_version)} payload = { "iss": "auth-service", "sub": subject_uuid, "preferred_username": username, "roles": ["employee", "manager"], "roles_csv": "employee,manager", "aud": ["ms1-profile-aggregator", "ms2-employee-details", "ms3-hardware-assets"], "act": {"sub": "ms1-profile-aggregator"}, "exp": now + 300, # 5 minutes "jti": str(uuid4()), "request_id": request_id } ``` The signing input (`header_b64.payload_b64`) is sent to Vault Transit's `/v1/transit/sign/mesh-identity` endpoint. Vault signs it with the RSA-2048 private key and returns the signature. Auth Service never holds the private key. Key details: * `roles_csv` exists because Istio's `outputClaimToHeaders` can only project string claims, not arrays. * `act` (the delegation claim from RFC 8693) is set to `ms1-profile-aggregator` when Profile Aggregator is in the audience. Auth Service sets this at mint time because it already knows (from the route-policy map) that the profile flow will pass through Profile Aggregator to reach Employee Records and Device Inventory. The `act` claim asserts that Profile Aggregator is the authorized intermediary for this request. When Profile Aggregator forwards the token downstream, the receiving AuthorizationPolicy verifies both the mTLS identity (is this actually Profile Aggregator?) and the `act` claim (was Profile Aggregator designated as the intermediary?). If a different service somehow obtained this token, the mTLS check would fail. * `kid` is fetched live from Vault on every mint. When keys rotate, new tokens immediately reference the new version. * TTL is 5 minutes. Short enough to limit replay windows, long enough that a fan-out from Profile Aggregator to Employee Records and Device Inventory completes comfortably. * The JWT is assembled manually rather than via a library like PyJWT. This is because the signing step goes through Vault Transit's API (not a local key), and standard JWT libraries assume they hold the private key. Building the token by hand keeps the signing interface clean: encode header and payload, send bytes to Vault, attach the returned signature. ## Step 5: Token Validation and Header Projection The signed token travels as the `x-mesh-identity` header. The destination sidecar handles it in two stages before the application sees the request. First, a Lua EnvoyFilter strips any inbound `x-ms2-user` / `x-ms2-role` headers. It runs at `priority: 10` with `INSERT_BEFORE` the JWT authentication filter, so spoofed values are removed before any claim is projected: ```yaml apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: name: header-projection-ms2 namespace: zt-apps spec: priority: 10 workloadSelector: labels: app: ms2-employee-details configPatches: - applyTo: HTTP_FILTER match: context: SIDECAR_INBOUND listener: filterChain: filter: name: "envoy.filters.network.http_connection_manager" subFilter: name: "envoy.filters.http.jwt_authn" patch: operation: INSERT_BEFORE value: name: envoy.filters.http.lua.projection typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua inlineCode: | function envoy_on_request(request_handle) request_handle:headers():remove("x-ms2-user") request_handle:headers():remove("x-ms2-role") end ``` This is the destination-side counterpart to the gateway pre-strip described in *The Trust Boundary*: the same strip-then-project pattern, now enforced at the workload. Any inbound copy of these headers is removed before the JWT filter runs. Then the `RequestAuthentication` resource validates the token and projects the verified claims: ```yaml apiVersion: security.istio.io/v1beta1 kind: RequestAuthentication metadata: name: mesh-identity-ms2 namespace: zt-apps spec: selector: matchLabels: app: ms2-employee-details jwtRules: - issuer: "auth-service" jwksUri: "http://auth-service.zt-apps.svc.cluster.local:8000/auth/jwks" forwardOriginalToken: false fromHeaders: - name: "x-mesh-identity" outputClaimToHeaders: - header: "x-ms2-user" claim: "sub" - header: "x-ms2-role" claim: "roles_csv" ``` The sidecar fetches public keys from the Auth Service JWKS endpoint, validates the JWT signature, checks `iss` and `exp`, and then projects `sub` into `x-ms2-user` and `roles_csv` into `x-ms2-role`. Because the strip filter already cleared any inbound copies, these headers can only carry claims from a validated token. These are the only headers the application reads. One Istio nuance is easy to miss: `RequestAuthentication` validates a JWT when one is present. It does not, by itself, mean "a JWT is required." The requirement comes from the `AuthorizationPolicy` in the next step. That policy checks claims that only exist after successful validation, so a missing or invalid token has no matching rule and is denied. Profile Aggregator has `forwardOriginalToken: true` because it needs to propagate `x-mesh-identity` to Employee Records and Device Inventory. All other services have it set to `false` since they're terminal and don't make downstream calls that need identity propagation. ## Step 6: AuthorizationPolicy After token validation, Istio's `AuthorizationPolicy` enforces token presence, token scope, and peer identity: ```yaml apiVersion: security.istio.io/v1beta1 kind: AuthorizationPolicy metadata: name: allow-ms1-to-ms2 namespace: zt-apps spec: selector: matchLabels: app: ms2-employee-details action: ALLOW rules: - from: - source: principals: - cluster.local/ns/zt-apps/sa/ms1-profile-aggregator-sa to: - operation: ports: ["8000"] when: - key: request.auth.claims[aud] values: ["ms2-employee-details"] - key: request.auth.claims[act][sub] values: ["ms1-profile-aggregator"] ``` Three things must be true simultaneously: the request must come from `ms1-profile-aggregator-sa` over mTLS, the token must be valid for `ms2-employee-details`, and the token must name `ms1-profile-aggregator` as the authorized intermediary. The peer identity comes from the client certificate. The `aud` and `act` values come from the signed JWT. These are independent cryptographic assertions. Compromising one doesn't give you the other. The `aud` claim is a list because a single profile request legitimately reaches more than one service. Istio's claim matching checks whether the configured value is present in that claim, so a token minted for Profile Aggregator, Employee Records, and Device Inventory can match `ms2-employee-details` here. A token minted only for Holiday Calendar would not match. This is confused-deputy prevention: stopping a privileged intermediary from being tricked into using its authority on behalf of an attacker. If Profile Aggregator's service account is somehow used by a different process, the process would still need a validly-signed token that authorizes Profile Aggregator to call Employee Records for this user flow. If an attacker replays a legitimate token from a different pod, the mTLS identity check fails. Both bindings must match. The deny behavior depends on having an `AuthorizationPolicy` that targets the workload. Istio allows traffic by default when no policy applies. In this architecture, every protected workload has at least one scoped `ALLOW` policy, so unmatched traffic is denied. A request with no token, a bad signature, the wrong audience, or the wrong caller identity has no matching rule and never reaches the application. Fine-grained authorization (can this user see *this specific record*, and which fields?) also runs at this point, via Cerbos. That's the resource and field-level layer, covered in a later post. This post stays on the identity flow. ## Step 7: The Service Contract After all of this, what does the service actually see? Two headers: ```python async def get_ms2_headers( x_ms2_user: str | None = Header(None), x_ms2_role: str | None = Header(None), x_request_id: str | None = Header(None), ): if not x_ms2_user or not x_ms2_role: raise HTTPException(status_code=401, detail="Missing required legacy headers") return {"user": x_ms2_user, "role": x_ms2_role, "request_id": x_request_id} ``` The service reads `x-ms2-user` and `x-ms2-role`. It returns 401 if they're missing. That's the entire identity contract. The 401 message calls them "legacy" headers, but the name only refers to this plain-header interface: the service reads simple headers as if it had no awareness of the mesh, and the mesh does the real verification upstream. The service does no JWT parsing, OIDC configuration, or Vault calls of its own. It trusts that if these headers are present, the sidecar has already done the verification. Profile Aggregator forwards `x-mesh-identity` and `x-request-id` to downstream services. It explicitly does not forward `x-ms2-user` or `x-ms2-role` since the receiving sidecar projects those independently from the JWT. Profile Aggregator cannot influence what identity Employee Records sees. ## Step 8: Database Identity Context The final step. Before executing any query, the service sets the transaction context: ```python async def set_rls_context(session: AsyncSession, user_id: str, roles: str, request_id: str): await session.execute( text("SELECT set_config('app.current_user_id', :user_id, true)"), {"user_id": user_id}, ) await session.execute( text("SELECT set_config('app.current_roles', :roles, true)"), {"roles": roles}, ) if request_id: await session.execute( text("SELECT set_config('app.request_id', :request_id, true)"), {"request_id": request_id}, ) ``` PostgreSQL RLS policies read `app.current_user_id` and `app.current_roles` to filter rows; `app.request_id` is set for audit correlation. The `true` third argument makes each setting transaction-local, so it doesn't leak across pooled connections. At this point, the database sees the same user identity that the sidecar projected from the verified token. In normal operation, the service is a pass-through for identity. It reads the sidecar-projected header and sets it on the transaction. A fully compromised service process could call `set_config` with an arbitrary value. RLS doesn't protect against a malicious process controlling its own DB session. But the layers above constrain this: the compromised service only has its own scoped DB role (Employee Records can only access `hr.*` tables, Device Inventory only `it.*`), and AuthorizationPolicy prevents it from reaching other services' databases. The blast radius of a compromised Employee Records service is limited to the HR data it already has access to, scoped by RLS to the identities it actually receives requests for. ## Failure Behavior The secure path matters, but so does the failure path. A missing or invalid mesh token fails at `AuthorizationPolicy` because the required claims are absent. If Auth Service is down, ExtAuthz cannot approve the request and no mesh token is minted. If Vault is unreachable, Auth Service cannot sign a token. If the sidecar cannot refresh JWKS, it continues using its cached keys until the cache expires; after that, validation fails closed for tokens it cannot verify. The one operational caveat is key rotation. Istio sidecars cache JWKS, so there is a short window where a sidecar may still verify tokens with a cached old public key. The next post covers that rotation window and why the token TTL is only 5 minutes. ## The Complete Chain Gateway strips external headers -> Auth Service validates credentials -> Auth Service checks route-level policy -> Auth Service mints a signed token -> sidecar validates the token and projects claims -> AuthorizationPolicy checks peer identity and token claims -> service reads projected headers -> database enforces row-level visibility. Eight steps. By the time Employee Records reads `x-ms2-user`, that value has passed through header stripping, credential validation, route authorization, token signing, sidecar validation, and peer binding. The service receives a narrow identity contract and passes that identity into PostgreSQL for row-level enforcement. ## What's Next The next post covers the signing infrastructure: why Vault Transit instead of local keys, how key rotation works without downtime, and what the blast radius looks like when different components are compromised. ## Code %[https://github.com/Kaustav-Sarkar/Istio-Defense-In-Depth]