PAYCARGO · AI-SANDBOX · PROCESS

4.5.aData Governance CMKs (substrate pattern for vibe apps)

A second axis of KMS keys, layered above the three substrate trust zones (agent / lockbox / audit). Every vibe app declares which governance cell its data lives in, and the substrate provides the right CMK by lookup. The obs app forces the issue; vibe apps 2 and 3 inherit the pattern.
Block: 4.5.a (immediate fix for 4.5; introduces a reusable substrate pattern) · Operator: Agentic build + human go/no-go on taxonomy · Cost: $2/month for the two CMKs Phase 1 needs (eng-low-risk + ceo-office-low-risk); $6/month total when the full Phase 1 grid is built · Permanence: Permanent — this is the data-classification substrate every vibe app from here forward inherits
In Progress

State: Pattern designed and approved. Two CMKs being built (eng-low-risk, ceo-office-low-risk). Obs app being rewired to consume them. Remaining four cells of the grid land next week with vibe apps 2 and 3.

Why this exists: Step 4.5 (obs vibe app) deployed cleanly but the frontend returned 403 AccessDenied from CloudFront because the audit CMK didn't grant CloudFront permission to decrypt. The naive fix is to broaden the audit CMK. The right fix is to recognize that application data has its own classification axis, distinct from substrate trust zones, and that the substrate should provide a grid of CMKs that vibe apps consume by tag.

Executive Summary

Add a second axis of KMS keys — a grid of sensitivity tier × organizational boundary — that lives above the three substrate trust zones and is consumed by every vibe app from here forward.

The substrate has three CMKs (agent / lockbox / audit) that protect platform machinery by the type of system writing the data. That layer doesn't tell you anything about application data — who consumes it, how sensitive it is, which department owns it. When the obs vibe app needed to encrypt frontend assets served to CTO and investors, those three keys had no natural home for it. We're adding a second axis: 3 sensitivity tiers (confidential 1:1, high risk, low risk) times N organizational boundaries (Engineering, CEO office, etc.). Each vibe app declares its governance cell and the substrate looks up the right CMK. Phase 1 builds two cells (the two the obs app needs). The remaining four come next week as vibe apps 2 and 3 demand them. The pattern is the deliverable, not the keys.

StatusIn Progress
Cost (now)$2/mo
Cost (full grid)$6/mo
RiskLow

If skipped: Each vibe app rolls its own KMS strategy. By app number five the substrate has nine ad-hoc keys with overlapping scopes and no single place to reason about which app encrypts what with which key. The governance question ("who can read this data?") is answered by reading code, not by reading a table.

1. Why this step exists (the audit CMK gotcha)

Step 4.5 deployed the observability vibe app. The frontend S3 bucket was encrypted with the substrate's audit CMK because the obs app emits things that look like audit data (platform telemetry shown to executives). CloudFront fetched /index.html via Origin Access Control, S3 tried to decrypt the object before returning it, and KMS rejected the call because the audit CMK key policy didn't list cloudfront.amazonaws.com as an allowed principal.

<Error>
  <Code>AccessDenied</Code>
  <Message>Access Denied</Message>
</Error>

The wrong fix

Add cloudfront.amazonaws.com to the audit CMK key policy with a CloudFront-distribution ARN scope. Zero cost, one new statement. But: the audit CMK protects CloudTrail data events, VPC flow logs, NFW logs, Bedrock invocation logs — the forensic trail you'd use during incident response. Broadening its policy to include "any CloudFront distribution in this account can decrypt" means an attacker who compromised CloudFront via a supply-chain bug suddenly has a path to decrypt your forensic data. The audit CMK should stay single-purpose.

The right fix

Recognize that the obs app's frontend bucket is not audit data — it's application data. Application data has its own classification axis distinct from the substrate trust zones. Add a second layer of CMKs that the substrate manages and exposes, indexed by sensitivity tier and organizational boundary. The obs app's frontend bucket consumes the cell that matches its audience and sensitivity. The audit CMK stays untouched.

The narrow framing is "fix the obs app's 403." The framing we're choosing is "build the data governance substrate that every future vibe app inherits." The cost differential is $2/mo and a single PR.

2. Two axes: substrate trust zones vs. data governance

The substrate keys and the data governance keys are orthogonal. They protect different things at different layers, and a single resource can be encrypted by either depending on what role it plays.

Layer Keys What they protect, indexed by
Substrate trust zones agent · lockbox · audit Platform machinery, indexed by what type of system writes the data. Agent runtime writes to agent. Lockbox service writes to lockbox. Logging services write to audit.
Data governance grid {org}-{confidential|high-risk|low-risk} Application data, indexed by who consumes it and how sensitive it is. The obs app's frontend assets are CEO-office-low-risk. The Sales monthly report app's CSV ingest is sales-high-risk (consumes PII). And so on.

A single resource can sit on either layer depending on what it represents. The obs app's CloudWatch log group holds application logs (eng operates and reads them, no PII) and lands in the data governance layer. The audit CMK still protects real audit data — CloudTrail data events, flow logs, NFW logs. They're different classes of log, not the same class.

3. The data governance grid

Three sensitivity tiers crossed with the organizational boundaries that own the data. Phase 1 has two organizational boundaries; future quarters add more.

  Engineering CEO office (Sales) (Ops)
Confidential (1:1) eng-confidential
future
ceo-office-confidential
future
High Risk eng-high-risk
future
ceo-office-high-risk
future
Low Risk eng-low-risk
built Phase 1
ceo-office-low-risk
built Phase 1

The grid is data-driven in Terraform via a frontend_governance_groups map variable. Adding a new cell — for instance, when Sales gets its first vibe app and needs sales-high-risk — is a one-line addition to the map. New CMK provisioned, policy templated to the tier, alias created, ARN exposed via the identity module's output map.

4. What each tier means operationally

Each tier has a policy template that the key inherits. The org dimension determines which IAM roles get kms:Decrypt grants. The tier dimension determines the shape of the policy: what service principals are allowed, what conditions are required, what the read-side audit story is.

Confidential (1:1)

High Risk

Low Risk

5. What we build in Phase 1

Only the two cells the obs app forces. The other four wait until a vibe app demands them.

CMK Alias Built when First consumer
eng-low-risk alias/ai-sandbox-eng-low-risk This PR (4.5.a) Obs app Lambda + log group + DDB cache partition
ceo-office-low-risk alias/ai-sandbox-ceo-office-low-risk This PR (4.5.a) Obs app frontend S3 bucket
eng-confidential alias/ai-sandbox-eng-confidential Future (when first demanded) TBD — likely individual dev secrets or per-engineer scratch storage
eng-high-risk alias/ai-sandbox-eng-high-risk Future TBD — likely debug exports from production systems
ceo-office-confidential alias/ai-sandbox-ceo-office-confidential Future TBD — M&A, exec comp, board materials
ceo-office-high-risk alias/ai-sandbox-ceo-office-high-risk Future TBD — escalation tracking, customer-specific exec follow-ups

The map variable lists all six entries with their descriptions. The two built now have their build_now = true flag set; the four pending have it set to false and are gated by a for_each filter. Adding a new cell to the live grid is flipping that flag and applying.

6. Obs app rewire

Four of the obs app's resources are KMS-encrypted. The audit CMK is wrong for three of them and the right key (CloudFront-allowed) didn't exist before this PR. We rewire as follows:

Resource Was Is Why
Frontend S3 bucket + objects audit CMK ceo-office-low-risk Audience is CEO / CTO / investors. CloudFront must decrypt. No PII.
Lambda function (env vars) audit CMK eng-low-risk Engineering owns. No PII in env.
Lambda CloudWatch log group audit CMK eng-low-risk Application logs (not platform audit logs). Engineering reads.
Obs-app cache partition (DDB) existing agent_state table CMK (unchanged) unchanged — the DDB table belongs to the platform substrate The DDB table is shared across vibe apps; per-vibe-app re-encryption is a substrate-design discussion for later.
CloudTrail / VPC flow logs / NFW logs audit CMK audit CMK (unchanged) These are audit data, not application data. They stay where they were.

Audit CMK ends up serving exactly what it was designed for: forensic logs. Application data lives on the data governance grid.

7. Cross-org access (the "higher audience" pattern)

The obs app's frontend is consumed by CEO office and by Engineering operators. The frontend bucket needs only one CMK. Which org's cell does it land in?

Pattern: assign to the higher-audience org as natural owner; cross-org reads via IAM grant.

The obs app's frontend bucket is encrypted with ceo-office-low-risk because the CEO office is the higher-stakes audience. Engineering doesn't need a second CMK; instead, Engineering's operator role gets an explicit kms:Decrypt grant on ceo-office-low-risk for the obs app's frontend bucket ARN. The grant is auditable in CloudTrail and revokable in one place.

This pattern reproduces cleanly. When the Sales monthly report app shares output with the CEO office, the data lands in ceo-office-{tier} with Sales gaining an IAM grant. When a future internal eng tool is shown to ops for triage, it lands in eng-{tier} with the ops role granted decrypt. Higher-audience is natural owner; lower-audience is grant.

Why not just create shared cells (eng-and-ceo-low-risk)? Because that path explodes combinatorially. The org dimension stays flat; cross-org sharing is always a grant on the owning org's key.

8. How vibe apps onboard to a cell

The flow for any new vibe app:

  1. Identify the app's audience (which org consumes the output). This is "who breaks if this data leaks".
  2. Identify the app's data sensitivity (confidential 1:1 / high risk / low risk). Use the definitions in § 4.
  3. Cross-reference the grid (§ 3). If the cell exists, consume it. If not, request the cell be added in a follow-up PR.
  4. In the app's Terraform module, accept a governance_cell variable. Look up module.identity.data_governance_kms_keys[var.governance_cell]. Use that ARN as the kms_key_id on every KMS-encrypted resource the app owns.
  5. If any other org needs read access, add an aws_kms_grant in the app module granting the cross-org role kms:Decrypt on the cell's CMK.

Adding a new cell is opening a small PR on the identity module: one entry added to the frontend_governance_groups map, build_now = true, apply, new ARN appears in the output map. Existing apps are unaffected.

9. Integration with the Vibe Code SDK

This pattern goes into MODULES.md as a first-class Vibe Code SDK feature. The SDK will expose:

The first three are small Terraform features. The fourth is a doc tool that runs in CI from terraform show output. None of them are needed for Phase 1 obs app; all of them become natural once vibe apps 2 and 3 land.

10. How to view in AWS Console

After the apply lands:

  1. KMS keys: AWS Console → KMS → Customer managed keys. Filter on alias ai-sandbox-. You'll see five keys now: agent, lockbox, audit (substrate), plus eng-low-risk and ceo-office-low-risk (governance). Each has its tier and org as tags.
  2. Frontend bucket encryption: S3 → ai-sandbox-obs-app-frontend-959228203854 → Properties → Default encryption. Should now reference the ceo-office-low-risk CMK by alias.
  3. Lambda env encryption: Lambda → ai-sandbox-obs-app-backend → Configuration → Environment variables → Encryption configuration. Should reference eng-low-risk.
  4. Verify the fix: Open https://d2kh3b2wl3msbg.cloudfront.net. If you still see AccessDenied, CloudFront is still serving the old cached error. Invalidate the distribution with aws cloudfront create-invalidation --distribution-id E17QRR36DCFNDW --paths "/*".

11. Anticipated gotchas

S3 bucket encryption changes don't re-encrypt existing objects

When we change the bucket-default encryption to a new CMK, existing objects stay encrypted with the old CMK. The four objects in the bucket (index.html, app.js, styles.css, config.json) need to be re-uploaded after the bucket policy changes. Terraform's aws_s3_object will see the kms_key_id changed and re-upload — but only because we have source_hash or content tracking. For config.json the timestamp() ensures content changes every apply. For the three frontend assets, source_hash = filemd5(...) means they only re-upload when the file changes. We force a re-upload by touching the files in this PR, or by running terraform apply -replace on each.

CloudFront caches the AccessDenied error

The initial 403 response is cacheable. After the apply lands, CloudFront may still serve the cached 403 for several minutes. Invalidate the distribution after apply (create-invalidation --paths "/*") to force a re-fetch. Future fix: lower the error response TTL on CloudFront for 4xx responses.

The grid variable is in the identity module, but cells are tagged with org/tier

Cells are read by their flat name (eng-low-risk) for module wiring simplicity. The richer information (org, tier, description) lives on the CMK as tags. A downstream consumer that wants to enforce "this app must use a tier ≤ low risk" reads the tag, not the flat name. Don't try to parse the flat name — treat it as an opaque cell identifier.

12. Quantum resistance

Because PayCargo holds payments-adjacent data with a long sensitivity tail, "is this quantum-safe" comes up in security reviews. This section gives the precise answer per layer so reviewers and operators have one place to read.

Symmetric encryption (the bulk of what we do)

KMS uses AES-256-GCM for envelope encryption of all data at rest: S3 objects, DynamoDB items, Secrets Manager values, CloudWatch Logs. The data encryption keys are themselves wrapped by the customer-managed master key inside the AWS HSM and never appear in plaintext outside the HSM boundary.

Under the strongest known quantum attack (Grover's algorithm) AES-256 has effective key strength of 128 bits. NIST and every published post-quantum threat assessment consider AES-256 quantum-resistant for long-term storage. We do not need to migrate the symmetric layer. This is the bulk of what our CMKs encrypt.

Key material at rest in the HSM

The CMK master key never leaves the AWS HSM. There is no exposed public key for a quantum adversary to factor. The HSM hardware itself is tamper-resistant. The key material is structurally inaccessible, quantum or otherwise.

Asymmetric crypto we depend on indirectly

Three places where classical asymmetric crypto is in play, all upstream of our CMKs:

LayerTodayPQ Plan
TLS to KMS, S3, DDB, Cognito API endpoints Classical ECDHE / RSA-2048 key exchange AWS is rolling out hybrid post-quantum TLS (ML-KEM, formerly Kyber) on KMS, S3, EC2, and IAM endpoints. We inherit this automatically; no code change in our stack.
Cognito JWT signing (access tokens) RS256 (RSA-2048) AWS-managed. Cognito's signing key migration is on the broader IAM/STS PQ roadmap. We inherit it.
CloudFront TLS termination ECDHE or RSA per certificate ACM is testing post-quantum hybrid certificates. We are on AWS-issued certs so this upgrade lands transparently.

The real threat model: "harvest now, decrypt later"

Today's adversaries do not have a cryptographically relevant quantum computer. Estimates for when (or whether) one arrives range from 10 years to never. The realistic concern is HNDL (harvest now, decrypt later): an adversary records our encrypted traffic today and waits 15-30 years for a quantum computer big enough to crack it.

HNDL is a problem for any classical TLS exchange done today. Symmetric data at rest is not affected (AES-256 is post-quantum safe). The mitigation for HNDL is hybrid PQ key exchange in TLS, which AWS is already deploying. We are on the safe side of every HNDL debate because:

What this means for an audit answer

When a reviewer asks "is this quantum-safe," the precise answer is:

The action item, if any, is to track AWS's hybrid-PQ rollout per service and confirm we have inherited it. There is no PayCargo-side migration required.

Non-technical readers should also look at Playground 1.5 (The Locks), which frames the same answer in the playground metaphor.

13. Doctrine

This step encodes three doctrines that already governed the rest of the substrate but didn't have a name in the obs-app layer:

14. What comes next