2Apply Identity (KMS CMKs + tier roles)

1. Why this step exists

Block 1 (Bootstrap) created the substrate. Block 2 (Identity) creates the trust zones — the cryptographic and IAM-level boundaries that every downstream workload organizes itself around. Without identity in place, the network module would have nowhere to attach workload role ARNs, the data module would have no CMKs to encrypt with, and the agent modules would have no tier roles to assume.

Why identity is the second block, not the tenth

Every downstream module depends on identity's outputs:

• Network module uses audit_kms_key_arn to encrypt VPC flow logs.
• Data module uses agent_kms_key_arn, lockbox_kms_key_arn, and audit_kms_key_arn to encrypt the buckets and tables it creates.
• Lambda-tools module uses agent_tier_role_arns to know which role each Lambda function assumes.
• Gateway module uses agent_tier_role_arns for inbound authorization.

Building identity early means downstream modules can reference real ARNs from day one, with no placeholder values to retrofit later.

2. What this step creates

Ten resources, organized into two groups.

Group A: KMS CMKs (six resources)

Group B: IAM tier role templates (four resources)

Two tiers, each with a role + an inline policy. The roles are templates — they cannot be assumed yet (no trust relationship to any service principal). Downstream modules (lambda-tools, runtime) reference these role ARNs when attaching workloads.

Both roles have the bootstrap permission boundary (ai-sandbox-permission-boundary) attached. Both are assumable only by the AgentCore service principal (and any other principals the gateway module wires in later); not by humans, not by other AWS accounts.

3. The three-zone doctrine (why three CMKs not one)

Splitting key material into three CMKs (instead of using one account-wide CMK) is the single most important structural decision in this module. The cost is $2/month extra ($3 total vs $1 with a single CMK). The benefit is that the boundary between zones becomes cryptographic, not just policy-based.

The blast radius story

How the lockbox key policy enforces exclusion

The lockbox CMK's key policy explicitly enumerates the lockbox service principal (and only that). It does not include agent tier role ARNs anywhere. KMS rejects any decrypt call from an agent role with AccessDenied — before the IAM layer is even consulted. This is the property called "structural PII-blindness of agents" in the project doctrine.

4. Tier roles (sor-reader vs internal)

The agent_tiers variable defines a map of tier name to capability flags. Each tier becomes an IAM role + inline policy via for_each. New tiers can be added later by extending the map; existing tier policies can be tightened by changing flag values.

sor-reader — the default outward-facing tier

Agents that need to read from PayCargo's systems of record (Salesforce, Monday, GitHub, GitBook) and reason on Bedrock. They can read PII tokens (the opaque references in the lockbox table) but cannot detokenize them — can_call_lockbox_reveal: false. This is the default tier for any net-new agent that touches SoR data.

internal — the AWS-service-plane tier

Agents that operate purely on AWS services (Bedrock for reasoning, S3 for storage, etc.) and have no business with external SoRs or PII. The strictest of the two: no SoR access, no lockbox access, no PII reveal. Used for utility agents (summarizers, classifiers, internal coordinators) that should not touch customer data at all.

Adding a third tier later

Future tiers (e.g., sor-writer with can_write_sors: true, or pii-revealer with can_call_lockbox_reveal: true) are added by extending the agent_tiers map in environments/ai-sandbox/main.tf or via tfvars override. The role templates are designed for this expansion.

5. Cost & permanence

Cost

Per-request KMS charges ($0.03 per 10,000 requests) are negligible at POC volumes. At sustained production volumes (~1M agent decrypt requests/day), this rises to ~$3 additional per day; still a small fraction of the overall cost picture.

Permanence

Permanent. KMS CMKs cannot be deleted instantaneously — they enter a 7-30 day pending-deletion state, and during that window any data encrypted with them becomes irretrievably lost. Tier role ARNs are referenced by every downstream module; deleting them would break workloads. Treat all 10 identity resources as foundational for the life of the account.

6. How to apply

Preconditions

• Block 1 (Bootstrap) complete and verified.
• Composition wired to remote state (backend.tf populated, terraform init -migrate-state succeeded).
• substrate/environments/ai-sandbox/terraform.tfvars exists with permission_boundary_arn set to the bootstrap output.
• SSO session active (AWS_PROFILE=ai-sandbox, aws sts get-caller-identity returns account 959228203854).

Commands

cd substrate/environments/ai-sandbox

# Plan ONLY the identity module (not network, not anything else).
# -target is the right tool here because we are doing block-by-block apply
# rather than one big apply across all modules.
terraform plan -target=module.identity -out=identity.tfplan

# READ the plan output. Expected:
#   Plan: 10 to add, 0 to change, 0 to destroy.
# Every resource line should have a + create marker.

terraform apply identity.tfplan

Apply takes ~30-45 seconds — KMS key creation is the slow step (~15s each), and they are created in parallel.

7. Verification

# 1. Three KMS keys exist with correct aliases
AWS_PROFILE=ai-sandbox aws kms list-aliases --query \
  "Aliases[?starts_with(AliasName, 'alias/ai-sandbox-')].{Alias:AliasName,KeyId:TargetKeyId}" \
  --output table

# Expected output: 4 rows (state from bootstrap + 3 from identity)
#   alias/ai-sandbox-tfstate    (bootstrap)
#   alias/ai-sandbox-agent      (identity)
#   alias/ai-sandbox-lockbox    (identity)
#   alias/ai-sandbox-audit      (identity)

# 2. Lockbox key policy DOES NOT include agent role ARNs
AWS_PROFILE=ai-sandbox aws kms get-key-policy \
  --key-id alias/ai-sandbox-lockbox \
  --policy-name default \
  | jq -r '.Policy' | jq '.Statement[].Principal'

# Expected: principals should be the account root and AWS services, NOT
# any role with "sor-reader" or "internal" in the name.

# 3. Two tier roles exist with permission boundary attached
AWS_PROFILE=ai-sandbox aws iam list-roles \
  --query "Roles[?starts_with(RoleName, 'ai-sandbox-agent-')].{Name:RoleName,Boundary:PermissionsBoundary.PermissionsBoundaryArn}" \
  --output table

# Expected: 2 rows, both with PermissionsBoundaryArn ending in
# /ai-sandbox-permission-boundary

# 4. Tier role inline policies reflect the per-tier capability flags
AWS_PROFILE=ai-sandbox aws iam get-role-policy \
  --role-name ai-sandbox-agent-sor-reader \
  --policy-name agent-tier-permissions \
  | jq '.PolicyDocument.Statement[] | {Sid,Effect,Action}' \
  | head -40

# Expected: Allow statements for Bedrock invoke, S3 scratch read/write,
# Secrets fetch, lockbox token READ (NOT reveal).

8. How to view the work in AWS Console

The most consequential thing this step deployed is the lockbox CMK key policy. Most of the value of reviewing in Console comes from sitting with that key policy and verifying that agent roles really aren't in it.

The key policy diff is worth a minute

Open the agent CMK key policy in one tab, lockbox CMK in another. The difference is the entire PII-blindness story. Agent CMK lists agent tier role principals; lockbox CMK does not. That's not a comment in the code, not a policy memo, not a Wiki page — it's a JSON document that AWS enforces at the cryptographic operation layer. Worth seeing it as text once so the mental model is concrete.

9. Anticipated gotchas

Warning: Resource targeting is in effect
You are creating a plan with the -target option...
The -target option is not for routine use...

9. Go / no-go gates

10. Doctrine implications

Cryptographic boundaries beat policy boundaries

The three-CMK design ensures the agent-vs-lockbox separation is enforced at the deepest possible layer (KMS), not at the IAM-policy layer (which is editable, audit-able-after-the-fact, but can be misconfigured). The doctrine is: when there is a choice between policy enforcement and cryptographic enforcement, pick cryptographic.

Tier roles are templates, not instances

Bootstrap created one role (the CI deploy role). Identity creates two roles, but they are templates — they have no service principal trust yet. Downstream modules (lambda-tools, runtime, gateway) instantiate the tier model by wiring specific workloads to these role ARNs. This separation lets identity exist independent of workloads, and lets new workload types reuse the existing tier definitions without re-deriving them.

Default-deny on the most sensitive zone

The lockbox CMK key policy starts from "deny everyone" and adds only the lockbox service principal. The agent CMK is more permissive (agent roles are allowed). The audit CMK is split (write-only for most callers, read for SecOps). The sensitivity gradient is reflected in the key policy gradient.

12. What actually happened during apply

Apply ran cleanly in a single pass — no fix-and-retry required. Three small things worth recording for future engineers.

The apply itself

Single terraform apply identity.tfplan from the composition directory created all 10 resources in ~30 seconds:

• 3 KMS keys created in parallel (~16s each, since AWS provisions them concurrently)
• 3 KMS aliases created immediately after their keys (instantaneous)
• 2 IAM tier roles + 2 inline policies created in parallel (~1s)

Gotcha A — the expected -target warning

As documented in § 8, Terraform printed:

Warning: Applied changes may be incomplete
The plan was created with the -target option in effect...

Ignored as designed. Block-by-block apply with explicit -target is the agreed workflow.

Verification tooling quirk 1 — kms get-key-policy rejects aliases

The verification command in § 7 said:

aws kms get-key-policy --key-id alias/ai-sandbox-lockbox --policy-name default

This errored with InvalidArnException: Key Aliases are not supported for this operation. Despite many other KMS CLI verbs accepting aliases freely, get-key-policy specifically requires the actual key ID.

Fix: use the key ID directly:

aws kms get-key-policy --key-id f855a443-2993-4109-8f7b-a5b438cd6e40 --policy-name default

Or look up the ID first via aws kms describe-key --key-id alias/ai-sandbox-lockbox --query 'KeyMetadata.KeyId' --output text and pipe.

Verification tooling quirk 2 — iam list-roles omits PermissionsBoundary

The verification command in § 7 used aws iam list-roles --query "...PermissionsBoundary..." to check that both tier roles had the boundary attached. The output showed both as None, which initially looked like a real bug (boundary missing).

In fact, list-roles returns a slimmed summary that doesn't include the PermissionsBoundary field, even when one is attached. The fix is to use aws iam get-role per role:

for role in ai-sandbox-agent-sor-reader ai-sandbox-agent-internal; do
  aws iam get-role --role-name $role | jq '.Role.PermissionsBoundary.PermissionsBoundaryArn'
done

Confirmed both roles have arn:aws:iam::959228203854:policy/ai-sandbox-permission-boundary attached.

11. What comes next

After identity apply succeeds and verification passes:

1. Mark Block 2 complete by updating Step.2.html's status banner from Pending to Completed (analogous to Step.0.html and Step.1.html closeouts).
2. Begin Step 3 — Network (Stages 1-6, alert mode). The big one: ~$425/month cost, 80+ resources, VPC + 9 subnets + NFW + DNS Firewall + flow logs. Lands in alert mode and stays there through Step 4. Step.3.html when ready.
3. Push the composition's backend.tf and terraform.tfvars.example updates to the repo so the canonical state of the composition reflects what was actually applied.

Network (Step 3) is the next destructive step and the first block where a mistake could affect traffic rather than just metadata. Stays in alert mode through the rest of the substrate build; enforcement flip moves to Step 5 (after data, observability, and the first agent are live in Step 4). The reasoning: the alert log needs real workload traffic to be meaningful, observability needs to be in place to monitor the flip, and the audit S3 bucket from the data block is a Stage 9 dependency anyway.

12. Forward reference — the second axis of CMKs (Step 4.5.a)

The three substrate keys (agent, lockbox, audit) protect platform machinery, indexed by which type of system writes the data. They are not the only CMKs the substrate provides. Once vibe apps started landing (Step 4.5 obs app), a second axis of keys was added: a data governance grid indexed by sensitivity tier × organizational boundary (Engineering, CEO office, future Sales/Ops/etc.). Each cell in the grid is a CMK; vibe apps consume cells by name.

The two axes are orthogonal:

• Substrate trust zones (this Step 2): what kind of platform system writes the data.
• Data governance cells (Step 4.5.a): who consumes the data and how sensitive it is.

A single resource lands on the right axis based on what it represents. Audit logs (CloudTrail data events, VPC flow logs, NFW logs) stay on the audit CMK forever. Vibe-app frontends, Lambda env vars, application log groups land on the data governance grid. See Step 4.5.a — Data Governance CMKs for the grid pattern, the tier policy templates, and the per-vibe-app onboarding flow.