The Identity Flow: From Gateway to Database

This blog is a part of the Zero-Trust Security on Istio Series, if you haven't read the previous article, please do, it will provide better context.

https://blogs.kaustavsarkar.dev/series/zero-trust-istio

The previous post laid out the five security layers as a static architecture. This post follows a single request through them, down to the question that matters in practice: what does a service actually trust, and why is trusting it safe?

The system is an HR employee directory, built to be attacked. Five microservices sit behind the mesh, numbered ms1 through ms5: Profile Aggregator (ms1) is the gateway-facing orchestrator; Employee Records (ms2) holds PII and salary data and Device Inventory (ms3) holds hardware assignments, both internal-only; Holiday Calendar (ms4) and Office Directory (ms5) serve holiday and office-location data, most of it public. Auth Service mints a signed identity token, x-mesh-identity, that carries the user through the mesh, and Vault signs it. This post traces one concrete request: an employee, Alice, requests her own profile with GET /api/profile/{id}. That request reaches Profile Aggregator, which fans out to Employee Records and Device Inventory. I'll follow the ms1 to ms2 hop, Profile Aggregator calling Employee Records, so every ms2-* resource below is that running example, not a special case.

Employee Records reads the user's identity from a plain HTTP header, eg x-ms2-user. The header is named per service (ms2 reads x-ms2-user, ms3 reads x-ms3-user) so one service can never consume another's identity headers. Employee Records does no JWT parsing and makes no callback to an auth service. It reads that header and treats the value as the authenticated user. That should look fragile, because headers are trivially spoofable. Any external client can send x-ms2-user: alice, and so can any compromised pod inside the cluster.

It works because of everything that happens before a service reads that header. A single request moves through eight steps from the gateway to the database, each adding one guarantee:

1. Header stripping at the gateway.

2. Authentication of the caller's credentials.

3. Route-level authorization against a coarse policy map.

4. Token minting, signed via Vault.

5. Token validation and header projection at the destination sidecar.

6. AuthorizationPolicy, binding peer identity to token claims.

7. The service contract, the two headers the application actually reads.

8. Database identity context for row-level enforcement.

Six of those steps run in the mesh before the application sees the request. By the time Employee Records reads x-ms2-user, its value has been stripped of anything the caller sent and re-derived from the signed token the destination sidecar verified. The last two steps are the service's own contract and the database scoping it then sets up.

The Trust Boundary

The Istio Ingress Gateway is where external input is converted into internal identity. Everything before it is untrusted. An external client can send any header it wants, including headers that look exactly like internal identity assertions. The gateway's job is to strip those, authenticate the caller, and produce a cryptographically signed token that downstream services can verify independently.

After the gateway, a service trusts only the headers its own destination sidecar wrote. The sidecar does the verification itself: it strips any inbound copy of the projected headers (like x-ms2-user), validates the JWT signature, checks the issuer, and verifies expiry, then writes the claims into fresh headers. The application reads those headers directly and runs no verification of its own, because the sidecar already ran it.

That trust is narrowly scoped. It covers only the headers the sidecar itself wrote after validating the token. Anything that merely arrived on the request, from an external client or an internal pod, is ignored. So if a compromised internal pod sends a spoofed x-ms2-user header, the destination sidecar strips it before the JWT filter runs and re-projects the value from the verified token. The spoofed header never reaches the application.

This model has a clear contract: the gateway is responsible for identity creation, sidecars are responsible for identity verification, and services are responsible for business logic. If any of those three fail in isolation, the other two still limit the blast radius.

Step 1: Header Stripping

Before authentication, before routing, before anything else, a Lua EnvoyFilter removes every internal trust header from the incoming request:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: gateway-prestrip
  namespace: istio-system
spec:
  priority: -100
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
              subFilter:
                name: "envoy.filters.http.ext_authz"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.lua.prestrip
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
            inlineCode: |
              function envoy_on_request(request_handle)
                local headers_to_remove = {}
                for key, value in pairs(request_handle:headers()) do
                  local lower_key = string.lower(key)
                  if string.match(lower_key, "^x%-platform%-") or
                     string.match(lower_key, "^x%-role%-") or
                     lower_key == "x-mesh-identity" or
                     string.match(lower_key, "^x%-ms%d%-user") or
                     string.match(lower_key, "^x%-ms%d%-role") then
                    table.insert(headers_to_remove, lower_key)
                  end
                end
                for _, key in ipairs(headers_to_remove) do
                  request_handle:headers():remove(key)
                end
              end

The filter runs at priority -100 with INSERT_BEFORE the ExtAuthz filter. The order is load-bearing: if ExtAuthz ran first and injected x-mesh-identity based on a valid session, but a spoofed x-ms2-user header also survived from the client, the downstream service would read the attacker-controlled value.

The two-pass approach (collect keys, then remove) is required because mutating headers during iteration is undefined behavior in Envoy's Lua API.

What gets stripped: x-platform-*, x-role-*, x-mesh-identity, and all x-ms<N>-user / x-ms<N>-role headers. Case-insensitive matching ensures no variant slips through.

Step 2: Authentication (ExtAuthz)

After stripping, the request hits the ExtAuthz filter. This calls Auth Service (auth-service), which resolves the caller's identity from one of two sources:

Bearer token: Extracted from the Authorization header, validated against Keycloak's JWKS endpoint. Roles come from realm_access.roles in the Keycloak JWT.

Session cookie: An opaque zt_session cookie, looked up in PostgreSQL. The session record holds the user's roles from the original OIDC flow.

Both paths produce the same output: a subject identifier, username, email, roles, groups, and department. The subject is deterministic across both auth paths. It's computed as uuid5(NAMESPACE_URL, "istio-security://users/<username>"), which means the same user always gets the same sub regardless of whether they authenticated via cookie or bearer token. This is important for RLS (which filters by sub) and for audit trails.

Step 3: Route-Level Authorization

Before minting a token, Auth Service checks a coarse policy map. This is the first authorization gate:

# Coarse route/role policy map (ext_authz.py)
if path.startswith("/api/profile/"):
    allowed_roles = {"employee", "manager", "hr_admin", "it_admin"}
    decision["audience"] = ["ms1-profile-aggregator", "ms2-employee-details", "ms3-hardware-assets"]
elif path.startswith("/api/holidays") and method == "GET":
    allowed_roles = {"employee", "manager", "hr_admin", "it_admin", "public_data_admin", "security_auditor"}
    decision["audience"] = ["ms4-holiday-calendar"]
elif path.startswith("/api/holidays") and method in ("POST", "PUT", "DELETE", "PATCH"):
    allowed_roles = {"public_data_admin", "hr_admin"}
    decision["audience"] = ["ms4-holiday-calendar"]
else:
    decision["reason"] = "Route not defined in coarse policy map"
    return decision

user_roles = set(decision["roles"])
if not allowed_roles.intersection(user_roles):
    decision["reason"] = "Unauthorized role for route"
    return decision

If the user's role doesn't match the route, the request is denied with 401. No token is minted. The audience field determines which services this token will be valid for. A token for the profile endpoint is valid at Profile Aggregator (ms1-profile-aggregator), Employee Records (ms2-employee-details), and Device Inventory (ms3-hardware-assets). A token for the holidays endpoint is valid only at Holiday Calendar (ms4-holiday-calendar). This prevents a valid token from one flow being replayed against a different service.

Any path not in the policy map is denied by default. GET /api/offices is deliberately absent, because those reads are public: the gateway's AuthorizationPolicy exempts them from ExtAuthz, so an unauthenticated read reaches Office Directory (ms5-office-locations) without a token and is treated as anonymous/public. Writes to /api/offices do go through the map above and require public_data_admin.

Step 4: Token Minting

If authentication and route authorization both pass, Auth Service mints a signed mesh token. The JWT is built manually (no library) and signed via Vault Transit:

header = {"alg": "RS256", "typ": "JWT", "kid": str(vault_key_version)}

payload = {
    "iss": "auth-service",
    "sub": subject_uuid,
    "preferred_username": username,
    "roles": ["employee", "manager"],
    "roles_csv": "employee,manager",
    "aud": ["ms1-profile-aggregator", "ms2-employee-details", "ms3-hardware-assets"],
    "act": {"sub": "ms1-profile-aggregator"},
    "exp": now + 300,  # 5 minutes
    "jti": str(uuid4()),
    "request_id": request_id
}

The signing input (header_b64.payload_b64) is sent to Vault Transit's /v1/transit/sign/mesh-identity endpoint. Vault signs it with the RSA-2048 private key and returns the signature. Auth Service never holds the private key.

Key details:

• roles_csv exists because Istio's outputClaimToHeaders can only project string claims, not arrays.

• act (the delegation claim from RFC 8693) is set to ms1-profile-aggregator when Profile Aggregator is in the audience. Auth Service sets this at mint time because it already knows (from the route-policy map) that the profile flow will pass through Profile Aggregator to reach Employee Records and Device Inventory. The act claim asserts that Profile Aggregator is the authorized intermediary for this request. When Profile Aggregator forwards the token downstream, the receiving AuthorizationPolicy verifies both the mTLS identity (is this actually Profile Aggregator?) and the act claim (was Profile Aggregator designated as the intermediary?). If a different service somehow obtained this token, the mTLS check would fail.

• kid is fetched live from Vault on every mint. When keys rotate, new tokens immediately reference the new version.

• TTL is 5 minutes. Short enough to limit replay windows, long enough that a fan-out from Profile Aggregator to Employee Records and Device Inventory completes comfortably.

• The JWT is assembled manually rather than via a library like PyJWT. This is because the signing step goes through Vault Transit's API (not a local key), and standard JWT libraries assume they hold the private key. Building the token by hand keeps the signing interface clean: encode header and payload, send bytes to Vault, attach the returned signature.

Step 5: Token Validation and Header Projection

The signed token travels as the x-mesh-identity header. The destination sidecar handles it in two stages before the application sees the request.

First, a Lua EnvoyFilter strips any inbound x-ms2-user / x-ms2-role headers. It runs at priority: 10 with INSERT_BEFORE the JWT authentication filter, so spoofed values are removed before any claim is projected:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: header-projection-ms2
  namespace: zt-apps
spec:
  priority: 10
  workloadSelector:
    labels:
      app: ms2-employee-details
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_INBOUND
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
              subFilter:
                name: "envoy.filters.http.jwt_authn"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.lua.projection
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
            inlineCode: |
              function envoy_on_request(request_handle)
                request_handle:headers():remove("x-ms2-user")
                request_handle:headers():remove("x-ms2-role")
              end

This is the destination-side counterpart to the gateway pre-strip described in The Trust Boundary: the same strip-then-project pattern, now enforced at the workload. Any inbound copy of these headers is removed before the JWT filter runs.

Then the RequestAuthentication resource validates the token and projects the verified claims:

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: mesh-identity-ms2
  namespace: zt-apps
spec:
  selector:
    matchLabels:
      app: ms2-employee-details
  jwtRules:
    - issuer: "auth-service"
      jwksUri: "http://auth-service.zt-apps.svc.cluster.local:8000/auth/jwks"
      forwardOriginalToken: false
      fromHeaders:
        - name: "x-mesh-identity"
      outputClaimToHeaders:
        - header: "x-ms2-user"
          claim: "sub"
        - header: "x-ms2-role"
          claim: "roles_csv"

The sidecar fetches public keys from the Auth Service JWKS endpoint, validates the JWT signature, checks iss and exp, and then projects sub into x-ms2-user and roles_csv into x-ms2-role. Because the strip filter already cleared any inbound copies, these headers can only carry claims from a validated token. These are the only headers the application reads.

One Istio nuance is easy to miss: RequestAuthentication validates a JWT when one is present. It does not, by itself, mean "a JWT is required." The requirement comes from the AuthorizationPolicy in the next step. That policy checks claims that only exist after successful validation, so a missing or invalid token has no matching rule and is denied.

Profile Aggregator has forwardOriginalToken: true because it needs to propagate x-mesh-identity to Employee Records and Device Inventory. All other services have it set to false since they're terminal and don't make downstream calls that need identity propagation.

Step 6: AuthorizationPolicy

After token validation, Istio's AuthorizationPolicy enforces token presence, token scope, and peer identity:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-ms1-to-ms2
  namespace: zt-apps
spec:
  selector:
    matchLabels:
      app: ms2-employee-details
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - cluster.local/ns/zt-apps/sa/ms1-profile-aggregator-sa
      to:
        - operation:
            ports: ["8000"]
      when:
        - key: request.auth.claims[aud]
          values: ["ms2-employee-details"]
        - key: request.auth.claims[act][sub]
          values: ["ms1-profile-aggregator"]

Three things must be true simultaneously: the request must come from ms1-profile-aggregator-sa over mTLS, the token must be valid for ms2-employee-details, and the token must name ms1-profile-aggregator as the authorized intermediary. The peer identity comes from the client certificate. The aud and act values come from the signed JWT. These are independent cryptographic assertions. Compromising one doesn't give you the other.

The aud claim is a list because a single profile request legitimately reaches more than one service. Istio's claim matching checks whether the configured value is present in that claim, so a token minted for Profile Aggregator, Employee Records, and Device Inventory can match ms2-employee-details here. A token minted only for Holiday Calendar would not match.

This is confused-deputy prevention: stopping a privileged intermediary from being tricked into using its authority on behalf of an attacker. If Profile Aggregator's service account is somehow used by a different process, the process would still need a validly-signed token that authorizes Profile Aggregator to call Employee Records for this user flow. If an attacker replays a legitimate token from a different pod, the mTLS identity check fails. Both bindings must match.

The deny behavior depends on having an AuthorizationPolicy that targets the workload. Istio allows traffic by default when no policy applies. In this architecture, every protected workload has at least one scoped ALLOW policy, so unmatched traffic is denied. A request with no token, a bad signature, the wrong audience, or the wrong caller identity has no matching rule and never reaches the application.

Fine-grained authorization (can this user see this specific record, and which fields?) also runs at this point, via Cerbos. That's the resource and field-level layer, covered in a later post. This post stays on the identity flow.

Step 7: The Service Contract

After all of this, what does the service actually see? Two headers:

async def get_ms2_headers(
    x_ms2_user: str | None = Header(None),
    x_ms2_role: str | None = Header(None),
    x_request_id: str | None = Header(None),
):
    if not x_ms2_user or not x_ms2_role:
        raise HTTPException(status_code=401, detail="Missing required legacy headers")
    return {"user": x_ms2_user, "role": x_ms2_role, "request_id": x_request_id}

The service reads x-ms2-user and x-ms2-role. It returns 401 if they're missing. That's the entire identity contract. The 401 message calls them "legacy" headers, but the name only refers to this plain-header interface: the service reads simple headers as if it had no awareness of the mesh, and the mesh does the real verification upstream. The service does no JWT parsing, OIDC configuration, or Vault calls of its own. It trusts that if these headers are present, the sidecar has already done the verification.

Profile Aggregator forwards x-mesh-identity and x-request-id to downstream services. It explicitly does not forward x-ms2-user or x-ms2-role since the receiving sidecar projects those independently from the JWT. Profile Aggregator cannot influence what identity Employee Records sees.

Step 8: Database Identity Context

The final step. Before executing any query, the service sets the transaction context:

async def set_rls_context(session: AsyncSession, user_id: str, roles: str, request_id: str):
    await session.execute(
        text("SELECT set_config('app.current_user_id', :user_id, true)"),
        {"user_id": user_id},
    )
    await session.execute(
        text("SELECT set_config('app.current_roles', :roles, true)"),
        {"roles": roles},
    )
    if request_id:
        await session.execute(
            text("SELECT set_config('app.request_id', :request_id, true)"),
            {"request_id": request_id},
        )

PostgreSQL RLS policies read app.current_user_id and app.current_roles to filter rows; app.request_id is set for audit correlation. The true third argument makes each setting transaction-local, so it doesn't leak across pooled connections.

At this point, the database sees the same user identity that the sidecar projected from the verified token. In normal operation, the service is a pass-through for identity. It reads the sidecar-projected header and sets it on the transaction.

A fully compromised service process could call set_config with an arbitrary value. RLS doesn't protect against a malicious process controlling its own DB session. But the layers above constrain this: the compromised service only has its own scoped DB role (Employee Records can only access hr.* tables, Device Inventory only it.*), and AuthorizationPolicy prevents it from reaching other services' databases. The blast radius of a compromised Employee Records service is limited to the HR data it already has access to, scoped by RLS to the identities it actually receives requests for.

Failure Behavior

The secure path matters, but so does the failure path. A missing or invalid mesh token fails at AuthorizationPolicy because the required claims are absent. If Auth Service is down, ExtAuthz cannot approve the request and no mesh token is minted. If Vault is unreachable, Auth Service cannot sign a token. If the sidecar cannot refresh JWKS, it continues using its cached keys until the cache expires; after that, validation fails closed for tokens it cannot verify.

The one operational caveat is key rotation. Istio sidecars cache JWKS, so there is a short window where a sidecar may still verify tokens with a cached old public key. The next post covers that rotation window and why the token TTL is only 5 minutes.

The Complete Chain

Gateway strips external headers -> Auth Service validates credentials -> Auth Service checks route-level policy -> Auth Service mints a signed token -> sidecar validates the token and projects claims -> AuthorizationPolicy checks peer identity and token claims -> service reads projected headers -> database enforces row-level visibility.

Eight steps. By the time Employee Records reads x-ms2-user, that value has passed through header stripping, credential validation, route authorization, token signing, sidecar validation, and peer binding. The service receives a narrow identity contract and passes that identity into PostgreSQL for row-level enforcement.

What's Next

The next post covers the signing infrastructure: why Vault Transit instead of local keys, how key rotation works without downtime, and what the blast radius looks like when different components are compromised.

Code

https://github.com/Kaustav-Sarkar/Istio-Defense-In-Depth