The doctrine docs (in each Step.N.html) explain what was built and why. This doc captures the opposite: what was considered and deliberately not built, and why. Without this record, future engineers tend to add things "just in case" — bloating the substrate, increasing cost, and accidentally weakening the security model. Every entry below was a real architectural alternative that was passed over for a stated reason.
| What we did not build | Considered alternative | Why omitted |
|---|---|---|
| An IAM user for the operator | IAM user + console password + MFA + access keys | IAM users carry long-lived credentials. The doctrine is "no long-lived AWS credentials anywhere." Adding even one IAM user would have created the precedent and, post-bootstrap, the permission boundary would have blocked the next one anyway. Better to start as we mean to continue. |
| A custom SAML federation (Okta / Azure AD / etc.) | Wire PayCargo's external IdP directly to AWS via SAML 2.0 | AWS IAM Identity Center already handles this if PayCargo wires its IdP at the org level. Doing it per-account is more configuration and gives the same outcome. |
| Root account access for routine work | Operator logs in as the AWS account root user | Root cannot be policy-scoped, cannot have MFA enforced for programmatic use, and represents unbounded blast radius. Use only for break-glass after MFA hardware token is provisioned for it. |
| A second short-lived path for backup | Both Identity Center and an IAM user "just in case" | Backup paths become production paths. If Identity Center is down, AWS Console direct login via root is the break-glass path; we do not need a second routine path. |
| What we did not build | Considered alternative | Why omitted |
|---|---|---|
| Remote state for bootstrap itself | Migrate bootstrap to its own remote backend after first apply | Keeping bootstrap on local state preserves the chicken-and-egg solvability. A future operator rebuilding from scratch (lost account, audit recovery) does not need a working remote backend to run bootstrap. The mild operational quirkiness is worth permanent recoverability. |
| An AWS-managed KMS key for state encryption | aws/s3 AWS-managed key |
AWS-managed keys cannot be policy-restricted (they trust the entire account) and rotate on AWS's schedule. The $1/month CMK buys policy control, auditable key usage, and a key policy that future engineers can read. |
| Service Control Policies (SCPs) at the org level | Org-level SCP that mirrors the permission boundary | SCPs require management-account access to change and are coarser-grained. The account-level permission boundary lives in the same code as the rest of the substrate, gets versioned via Terraform, and applies to exactly the roles we control. SCPs become valuable when there are multiple AI accounts to govern together; today there is one. |
| A separate CI role for plan-only operations | Two roles: one for plan, one for apply | The permission boundary already caps what the deploy role can do, and the apply workflow's manual approval gate prevents accidental writes. A second role would be configuration overhead with limited additional protection. |
| CloudTrail in bootstrap | Provision an account-wide CloudTrail during bootstrap | CloudTrail lands in Step 4b (observability) alongside the audit S3 bucket and the audit CMK consumer. Doing it in bootstrap would create a CloudTrail without a properly-encrypted destination bucket. Delayed for the right reason. |
| What we did not build | Considered alternative | Why omitted |
|---|---|---|
| A single shared CMK for the entire AI-sandbox account | One CMK encrypts everything (agent + lockbox + audit data) | Saves $2/month, costs the cryptographic boundary. The lockbox key policy is the structural enforcement of PII-blindness; merging it with agent and audit eliminates that property. Worth the $2. |
| A "super-tier" agent role that can do everything | One agent role with maximally permissive policy, used everywhere | Tier roles exist specifically so different workloads get different capability surfaces. Collapsing to a single role means every agent has lockbox reveal permission, which is the opposite of the doctrine. |
| External KMS via CloudHSM | Customer-owned HSMs backing the CMKs | CloudHSM is meaningful for FIPS 140-2 Level 3 compliance and for organizations where AWS holding the master key is itself a threat model. Neither applies at POC; KMS-managed CMK material is the right cost/security balance. |
| A "service-account" tier role for AWS-service-to-AWS-service calls | Distinct tier for inter-service calls (e.g., Lambda calling Bedrock) | AWS services use their own service principals (e.g., lambda.amazonaws.com), not tier roles. The tier roles are for workload identities. A "service-account" tier would conflate the two. |
| KMS key rotation more frequent than annual | 90-day or 180-day rotation | Annual is AWS-recommended for CMKs holding long-lived data. More frequent rotation increases the surface for failed rotations and audit-log noise without measurable security benefit at this data volume. |
| What we did not build | Considered alternative | Why omitted |
|---|---|---|
| Three-AZ redundancy from day one | az_redundancy = "multi" with NFW + NAT + endpoints in all 3 AZs |
$1,217/month vs $425/month. The trade-off (AZ-a outage breaks egress until recovery) is acceptable for sandbox. Graduation ladder: 1 AZ → 2 AZ (internal app/SaaS replacement) → 3 AZ (production). See Step.3.html § 5. |
| AWS WAF | Web Application Firewall in front of any HTTPS endpoint | WAF protects customer-facing web traffic. Nothing in the AI-sandbox is customer-facing today (no API Gateway, no ALB serving customer requests). WAF would be ceremony with no traffic to protect. |
| CloudFront | CDN in front of agent endpoints | Same reason as WAF. No customer-facing content distribution today. |
| Site-to-Site VPN or Direct Connect to PayCargo HQ | Private connectivity from PayCargo's corporate network to this VPC | No workload currently lives in PayCargo's corporate network that needs to call into AI-sandbox. When/if such a workload appears (e.g., an on-prem orchestrator), Transit Gateway + VPN can be added without rebuilding the VPC. |
| Transit Gateway for multi-VPC peering | Hub-and-spoke topology with TGW | One VPC today. TGW becomes valuable when AI-research account or other PayCargo accounts need cross-VPC routing through a central inspection point. |
| A dedicated DNS zone (Route53 private hosted zone) | Custom DNS names for internal services | VPC endpoints use AWS service DNS automatically via private DNS. No internal services exist yet that would need a custom name. Adding a private hosted zone for nothing is configuration without a use case. |
| A jump host / bastion in the VPC | EC2 bastion in a public subnet for break-glass SSH access | SSM Session Manager (which AWS provides without a bastion) is the right break-glass mechanism. No long-running bastion to patch, no SSH keys to rotate. Bastions are an antipattern. |
| PrivateLink to PayCargo's internal services | VPC endpoint services exposing PayCargo internal APIs to AI-sandbox | No PayCargo internal services exist today that the AI-sandbox needs to call privately. The 4 SoRs are external SaaS; PayCargo's internal services (if relevant later) would be added via PrivateLink at that time, not pre-built. |
| VPC peering to AI-research account | Cross-account peering for shared workloads | AI-research is a different account by design (per project doctrine: separate accounts for separate risk profiles). Connecting them undoes that separation. If a future workload genuinely needs both, it should not be in either of those accounts. |
| What we did not build | Considered alternative | Why omitted |
|---|---|---|
| Automatic apply on merge (no approval gate) | Push to main → apply runs immediately | Removes the human-in-the-loop checkpoint that is the entire point of the gated apply path. The two-step "merge → approve" sequence is the structural enforcement of doctrine. |
| A separate non-production environment | Staging/dev environments with their own approval rules | There is one AWS account (AI-sandbox). Adding "production" and "staging" GitHub environments without backing AWS accounts would be cargo culting. If a research account comes online, it gets its own environment then. |
| GitHub Actions self-hosted runners in the VPC | Runners running on EC2 inside the AI-sandbox VPC | Self-hosted runners are useful when CI needs network access to internal resources. Our applies need access to AWS APIs (reachable from anywhere) and Terraform Cloud (none). The free GitHub-hosted runners are the right choice; self-hosted would be operational overhead with no benefit. |
| Atlantis or Terraform Cloud as the CI tool | Pull-request-based Terraform automation product | Atlantis is a great tool that would also work. GitHub Actions is sufficient, costs zero on the free tier, and doesn't introduce another vendor. The day Atlantis features become operationally necessary, the path to migrate is clear (the workflow files are simple; the substrate stays the same). |
| Long-lived AWS access keys for CI | Generate access keys for a CI-only IAM user, store as GitHub secrets | The entire point of OIDC. Secrets in GitHub are the canonical leak surface; we have none. |
| Hosting the AI-sandbox repo under the main PayCargo GitHub org | Move the repo from PayCargoDevOps to PayCargo, share the Enterprise plan, get Environments-with-required-reviewers gating for free |
Org separation is intentional blast-radius isolation, same doctrine as the AWS account separation. A compromise of PayCargo main org (broader user base, more business surface) cannot propagate to the AI infrastructure repos. The $4/user/month for PayCargoDevOps Team plan is a separate, smaller decision than anything that happens in PayCargo main; we stay independent on cost and access. CI gate currently uses workflow_dispatch + confirmation input as a free equivalent. |
| GitHub Environments with required-reviewer protection (now) | Pay for GitHub Team plan on PayCargoDevOps to unlock Environments on private repos | Equivalent gate today via workflow_dispatch + a confirmation guard job that aborts if the operator does not type APPLY. For a single-operator setup the gate is functionally identical (the human still has to take a deliberate, recorded action before AWS state changes). When PayCargoDevOps grows beyond one operator and we want the stricter "approver must differ from author" property, upgrade to Team plan and revert the workflow_dispatch commit; the Environments gate takes over. |
Step 4 has not been built yet; these are the omissions already committed-to in the design, captured here so they don't get re-debated during implementation.
| What we will not build | Considered alternative | Why omitted |
|---|---|---|
| Aurora / RDS | Relational DB for application state | Access pattern is key-value (token → PII) and per-agent state lookups. DynamoDB matches the pattern with on-demand pricing and no idle cost. Aurora would be standing $50+/month for a workload that does not need joins or transactions. |
| ElastiCache / Redis | In-memory cache layer for hot keys | No hot-path latency requirements today. DynamoDB single-digit-ms reads are sufficient. Adding a cache adds operational complexity (warm-up, eviction, consistency) for unmeasured benefit. |
| EKS or ECS for the workload runtime | Kubernetes/ECS instead of AgentCore + Lambda | AgentCore is the platform doctrine for agent orchestration. Lambda handles the tool-execution layer. EKS would be configuration overhead duplicating what AgentCore provides. |
| Step Functions | State-machine orchestration for multi-step agent workflows | AgentCore Runtime handles agent loops natively. Step Functions would be a second orchestration layer above AgentCore, which is confusing and adds cost. |
| SQS / SNS / EventBridge as primary integration | Event-driven architecture between agents and tools | AgentCore Gateway handles the inbound request pattern; Lambda tools handle outbound SoR calls. Pub/sub is a future-when-needed addition (e.g., async result delivery), not a substrate primitive. |
| A separate observability account | Send CloudTrail + GuardDuty + flow logs to a security-team-owned account | Currently no SecOps team or org-level audit account at PayCargo. When one exists, the audit S3 bucket can be configured to replicate to it without changing the rest of the substrate. |
| Multi-region failover | Active-active or active-passive across us-east-1 and us-west-2 | Bedrock model availability in us-east-1 anchors the region. Multi-region adds VPC peering, replicated state, replicated KMS, replicated CMKs — substantial operational complexity for protection against a region-wide AWS outage. Acceptable risk at POC. |
| What we did not build | Considered alternative | Why omitted |
|---|---|---|
| A configuration management tool (Ansible, Chef, Puppet) | Provision EC2 instances with config management | No long-running EC2 instances in the substrate. Lambda + managed services + AgentCore replace the “configure a server” problem. CM tools have no substrate to manage. |
| A monitoring stack (Datadog, New Relic, Prometheus) | Commercial APM in front of CloudWatch | CloudWatch + GuardDuty + flow logs cover what the substrate needs. Commercial APM becomes valuable when agent traffic volume justifies the per-host cost. |
| A CI/CD pipeline for Lambda code (Step 4+) | Separate Lambda-build pipeline distinct from Terraform pipeline | The Terraform CI pipeline (Step 3a) handles Lambda function code as part of the module that defines the function. Splitting the pipelines duplicates approval gates. When Lambda packages get large enough to justify a separate build, this can be revisited. |
| Cost allocation tags beyond what we deploy | Per-agent / per-team / per-business-unit cost tagging | One project, one team. Project-level tags suffice. If multiple teams or business units share the account later, tags can be backfilled. |
| An organization-wide Terraform state backend | Shared state across multiple PayCargo accounts | One AWS account in scope. Per-account state is the right granularity. |
| Service Mesh (App Mesh, Istio) | Per-service mTLS, traffic policy, observability | No service-to-service traffic at the substrate layer that would benefit from mesh primitives. Workloads call Bedrock and SoRs; no microservices to mesh. |
| A WAF, an SBOM scanner, a SAST tool, a SCA tool | Security tooling layer above the substrate | Substrate doctrine handles the infrastructure security model. Code-level scanning is a Step 4+ concern when there is meaningful code (Lambda functions, agent prompts, etc.) to scan. |
Most entries above are “not yet” rather than “never.” The doctrine is that the substrate should be the smallest thing that lets the platform run safely — not the largest thing that might someday be useful. Things land when there is a workload that genuinely needs them, not before.
Examples of the distinction:
When someone proposes adding X, search this doc for X first. If it appears, the reasoning for not building X is right there — either the reasoning still applies (don't add it) or the conditions have changed (revisit the trade-off explicitly, update this doc with the new reasoning, and proceed). Either way, the conversation starts informed.