4.5Observability Vibe App

1. What this vibe app is, in 60 seconds

A web app at https://obs.ai-sandbox.paycargo.com (CloudFront-served). Three audience tabs: Operators, Exec, Audit. Each tab is a single-page view with 4-8 panels. Each panel pulls live data from CloudWatch / CloudTrail / GuardDuty / Bedrock invocation logs via a Lambda backend. A daily 09:00 ET job invokes Bedrock to generate a narrative paragraph ("Yesterday the platform processed 1,247 agent calls across 3 tiers; cost was $4.31; one NFW alert fired against an undocumented destination, investigated and confirmed as test traffic"). The paragraph appears at the top of the Exec tab.

It is itself an agent-assisted application. It reads from the same audit streams every other agent writes to. The same guardrail enforces the narrative generation. The same KMS keys encrypt its data. It is the substrate watching itself.

2. Where it lands — the 9 AWS components

Every box below is a discrete AWS resource Terraform deploys as part of the vibe app module. Each one is priced in § 4.

3. What happens when a user opens the app — call flow

End-to-end, from "the user clicks the bookmark" to "the user sees the Operators dashboard." Every arrow is a real AWS API call that produces a CloudTrail event.

Every call leaves a trail

• Step 1-2: CloudFront access logs (90 days retention)
• Step 3: S3 access logs (audit bucket)
• Step 4: Cognito events (sign-in success/fail, MFA challenges) in CloudTrail
• Step 5: Lambda invocation in CloudWatch + CloudTrail data event
• Step 6: Every AWS API call the Lambda makes appears in CloudTrail

So the "the user opened the dashboard" event itself is part of the audit trail the dashboard displays. Recursive observability in literal form.

4. Cost — every dollar tracked

Assumptions

• 5-10 internal users (PayCargo execs + operators + external developers)
• ~20 sessions/day, each loading ~5 dashboard panels with 60s auto-refresh for ~5 min
• ~1500 Lambda invocations/day (panel loads + refreshes)
• 1 daily narrative generation (Bedrock Haiku, ~3k input + 600 output tokens)
• ~50 MB total static asset transfer/month (assets cached aggressively at the edge)
• No public traffic — Cognito gates all access

Cost breakdown

Cost scaling — what happens with more users

5. Auth — dual path with explicit cutover

Today (Phase 1): manual enrollment + TOTP MFA

• Cognito User Pool with email/password sign-in
• Email domain restricted to @paycargo.com
• MFA required (mfa_configuration = "ON") via TOTP authenticator app — no SMS, no email codes (both phishable)
• Enrollment: an admin invites a user, user gets email with temporary password, first sign-in forces password change + TOTP setup
• Session: 1-hour access token, 1-day refresh token. Reauth required daily.

Tomorrow (Phase 2): SSO federation via Identity Center

• Configure Identity Center as an IdP attachment on the Cognito User Pool (SAML 2.0)
• Cognito stays in the path as a token-issuance facade so the JS bundle doesn't need to change
• The "Sign in" button federates to Identity Center, which handles auth + MFA centrally
• Existing manual users migrate by signing in with their @paycargo.com email through the IdP — Cognito attribute mapping links the two

The cutover gate — two feature flags

# cartridges/administrative/obs/variables.tf
variable "enable_manual_enrollment" {
  description = "Allow Cognito-native email+password+TOTP signup. Flip to false once SSO migration is complete."
  type        = bool
  default     = true
}

variable "enable_sso_federation" {
  description = "Wire Identity Center as a SAML IdP. Flip to true once Identity Center is configured."
  type        = bool
  default     = false
}

Cutover steps:

1. Identity Center configured (Phase 2 prerequisite work)
2. Flip enable_sso_federation = true, apply — users can now sign in either way
3. Communicate to all users: "Use the new SSO button from now on"
4. Watch the obs dashboard's own auth panel — when manual sign-ins drop to zero, proceed
5. Flip enable_manual_enrollment = false, apply — new manual signups blocked. Existing manual users can sign in but must migrate or be disabled.
6. Optionally: 30 days later, run a script to delete manual user accounts that haven't signed in via SSO. Document deletion in the audit log.

6. The three audience tabs

Operators tab (platform operators, operations, the external developer)

Question this tab answers: "Is the platform healthy right now?"

• Bedrock invocations per minute (real-time chart)
• NFW packets accepted vs dropped (alert mode shows dropped=0 today)
• Lambda errors + throttles (cross all functions)
• Recent NFW alert log entries (last 50, table view)
• State backend health (DDB lock table reads + S3 state object age)
• Refresh: every 60 seconds, auto

Exec tab (CTO, VP Eng, CEO, CISO, strategic leadership, investors)

Question this tab answers: "What is the platform doing, what is it costing, is it safe?"

• Today's narrative (Bedrock-generated paragraph, top of page)
• Agent activity, 30 days (line chart)
• Token usage = cost proxy (input + output, separate lines)
• Month-to-date AWS spend (Cost Explorer pull, broken down by service)
• GuardDuty findings, severity ≥ 4 (table)
• Last successful deploy (CI workflow link)
• Refresh: on tab open, manual refresh button

Audit tab (SecOps, compliance, external auditors)

Question this tab answers: "Who did what, when, on what data?"

• Recent role assumptions (CloudTrail AssumeRole events, last 100)
• Recent KMS Decrypt events (any agent decrypting lockbox data shows here, even though Phase 1 doesn't exercise this)
• Recent CloudTrail events on the audit bucket (who read what)
• GuardDuty findings, all severities
• Pre-built Logs Insights queries (clickable to open in CloudWatch console)
• Refresh: on demand

7. The vibe: Bedrock-augmented daily narrative

Once a day at 09:00 ET, the Lambda backend's generateNarrative handler fires. It:

1. Queries CloudWatch + Cost Explorer for the last 24h of platform data (invocations, cost, alert counts, error counts)
2. Formats a structured prompt: "You are the AI platform's daily narrator. Yesterday's data: {data}. Generate a 2-paragraph executive summary suitable for CTO + CEO + investors. Mention notable events. Be honest about anomalies. Don't editorialize."
3. Calls Bedrock InvokeModel with Claude Haiku 4.5 + the default platform guardrail
4. Receives ~400-800 token response
5. Writes to DDB cache with key session_id=obs-app, item_key=narrative-{date}
6. Exec tab reads the latest narrative on load

Example output:

The narrative is generated by an agent, against the same Bedrock + guardrail every other agent uses. It is itself an agent invocation that appears in the Bedrock invocation log group, that this very dashboard displays. The recursion is intentional and documented.

8. Deployment — CI through OIDC, same path as substrate

The vibe app deploys via the same CI workflow (Step 3a) that deploys the substrate. There is no separate pipeline.

The Terraform module

cartridges/administrative/obs/
├── versions.tf
├── variables.tf       (5 inputs: substrate references + feature flags)
├── frontend.tf        (S3 bucket, OAI, CloudFront, ACM cert, Route 53)
├── auth.tf            (Cognito User Pool, manual enrollment, SSO IdP attachment)
├── backend.tf         (Lambda function, Function URL, IAM role + policies)
├── eventbridge.tf     (daily schedule)
├── outputs.tf         (URL, Cognito Pool ID, Lambda ARN — for downstream apps)
├── lambda/
│   ├── package.json
│   ├── handler.js     (route handlers + cache logic)
│   └── narrative.js   (Bedrock invocation for daily summary)
└── frontend/
    ├── index.html
    ├── operators.html
    ├── exec.html
    ├── audit.html
    ├── app.js         (Cognito client + fetch wrapper + render)
    └── styles.css     (matches Sales mockup visual language)

Deploy flow

1. Developer commits change in a feature branch
2. PR opens → CI plan workflow runs → posts plan as PR comment
3. Reviewer approves PR → merge to main
4. Operator manually triggers terraform-apply.yml with confirm=APPLY
5. CI assumes ai-sandbox-github-actions-deploy via OIDC, applies the change
6. Static asset deploys: CI uploads new HTML/CSS/JS to the S3 bucket; CloudFront invalidation runs for /index.html and /*.html only (not /static/* which is content-hashed)
7. Lambda code deploys: CI packages lambda/ into a zip, updates the function
8. Done. New URL serves new content within ~60 seconds.

9. The app watches itself — recursive observability

Every component of the vibe app produces logs that the vibe app displays. The recursion is deliberate.

This is a useful property for two reasons. First, it proves the substrate is honest — there are no "magic" data sources that aren't real AWS resources. Second, it means the app is self-debugging: if the dashboard isn't updating, the very dashboard tells you why (Lambda errors, throttles, KMS denies).

10. Why serverless, not Fargate

You asked specifically about Fargate. The choice not to use it is deliberate and worth documenting.

The cold start trade-off is real. For an internal app that 10 people use occasionally, a 400ms cold start once per visit is invisible. For a customer-facing app with strict SLAs, the calculus is different and Fargate (or Lambda with provisioned concurrency, ~$5/month per concurrency unit) might win.

Documented in MODULES.md under "What we DON'T have, deliberately" so this choice doesn't get reconsidered every quarter.

11. How this proves the substrate doctrine

The observability vibe app is a working proof of every doctrine the substrate makes structural. A tech leader walking through it should see each property enforced in code, not in policy.

12. Documentation we have delivered (and where this fits)

The vibe app does not stand alone. It consumes substrate built in Steps 0-4, follows the doctrine in MODULES.md, and serves the audiences identified in the 60-day plan. Tech leaders reviewing the app should have the full doc trail available.

13. Click-through prototype

An interactive mock of the obs vibe app lives at Step.4.5.clickthrough.html. It demonstrates:

• The three audience tabs (Operators, Exec, Audit) rendered with realistic sample data
• A working sign-in modal showing the Cognito + TOTP flow (no real backend; the modal is illustrative)
• Hover-over annotations on every panel showing: the AWS API call that fills it, the cost it incurs per query, and which Step.N.html doc explains the underlying resource
• The daily narrative (sample text generated to match the format Bedrock would produce)
• A "view source" link on each panel showing the actual Lambda code that would run in production

The click-through is the artifact you can hand to strategic leadership or an investor in advance of a demo, so they can explore at their own pace. It is not a working app; it is a high-fidelity prototype for stakeholder review.

14. Forward reference — data governance CMK grid (Step 4.5.a)

The first deploy of this app hit a CloudFront 403 AccessDenied because the audit substrate CMK didn't grant CloudFront permission to decrypt the frontend S3 objects. The fix was not to broaden the audit CMK (which would dilute the most important key in the substrate). The fix was to recognize that application data has its own classification axis, distinct from substrate trust zones, and to add a grid of CMKs that the substrate provides and vibe apps consume by name.

The pattern (sensitivity tier × organizational boundary), the policy templates per tier, the cross-org access pattern ("higher audience as natural owner; lower audience via IAM grant"), and the per-vibe-app onboarding flow are all documented in Step 4.5.a — Data Governance CMKs. The obs app's frontend bucket now lands on ceo-office-low-risk; the Lambda + log group land on eng-low-risk. Every future vibe app inherits the pattern by declaring its governance cell.