1Apply Bootstrap

1. Why this step exists

Bootstrap is the first Terraform apply in the AI-sandbox account. It does the smallest possible set of work that has to exist before any other Terraform module can run safely. Three properties make a module "bootstrap":

1. It is run by a human — no CI exists yet to run it (CI requires bootstrap to have run, so it cannot bootstrap itself).
2. It uses local state — remote state requires S3 + DynamoDB + KMS to exist, and bootstrap is the thing that creates them.
3. It creates the substrate for everything else — the state backend that downstream modules write to, the CI identity that downstream modules are applied through, and the permission boundary that constrains every IAM role created from this point forward.

2. What this step creates

Six classes of resources, all named with the ai-sandbox-* prefix so they are unambiguous in CloudTrail and the AWS console.

All resources are tagged with Project=ai-sandbox, Owner=platform-team, ManagedBy=terraform, Module=bootstrap.

3. Why bootstrap is on local state (the chicken-and-egg)

Every other Terraform module in this project uses remote state — the S3 bucket and DynamoDB lock table that bootstrap creates. Bootstrap itself cannot, because those resources do not yet exist when bootstrap runs. So bootstrap is the one and only module that lives on local state, with its terraform.tfstate file written to the operator's laptop in substrate/bootstrap/.terraform/.

Implications of local state

• The local state file is sensitive. It contains the ARNs of the resources bootstrap created, including the state KMS key ID. It is gitignored (*.tfstate in .gitignore) and never committed.
• Loss of the local state file is recoverable, but inconvenient. If the laptop dies, the resources still exist in AWS; rebuilding the local state requires terraform import of each resource by name. Resource names are deterministic, so this is mechanical work, not detective work.
• Bootstrap is rarely re-applied. Once it succeeds, the only reason to re-apply is to change bootstrap's own resources (rotate the state KMS key, expand the OIDC trust to a new ref, tighten the permission boundary). Those changes happen rarely and deliberately.

4. What this step secures (the permission boundary in detail)

The permission boundary is the most consequential resource bootstrap creates. It is a managed IAM policy that every downstream IAM role attaches via the permissions_boundary argument. The boundary acts as a ceiling on what any role can do, regardless of how permissive its inline or attached policies are.

What the boundary blocks (defense in depth, even against your own future mistakes)

What the boundary deliberately does not block

• Operational actions on resources within the role's intended scope (read S3, invoke Bedrock, write DDB, etc.). The boundary is a ceiling, not the operative policy.
• Anything the SSO admin (your session) does. The boundary applies to roles created by Terraform, not to SSO sessions.
• Anything the bootstrap-created deploy role does. The deploy role has the boundary attached, but the boundary explicitly allows the actions needed to apply downstream modules (creating KMS keys, VPCs, IAM roles, etc.).

5. Cost & permanence

Cost

The KMS CMK is the only standing charge. It exists to give this account control over key rotation and key policy, rather than relying on AWS-managed keys whose lifecycle is opaque.

Permanence

Permanent. Bootstrap resources are intended to be permanent for the life of the AWS account. Tearing them down would invalidate every Terraform state file in S3 (breaking every downstream module), remove the only CI deploy path, and remove the permission boundary every role depends on. Removal would be operationally equivalent to deleting the account.

Modifying bootstrap resources is rare but supported (rotate state KMS key, add a new repo to the OIDC trust, tighten a clause in the permission boundary). These changes are made by re-applying bootstrap with the same local-state setup.

6. How to apply

Preconditions (must be true before apply)

• Step 0 complete (SSO authenticated as mtang@paycargo.com on AI-sandbox account 959228203854 with AdministratorAccess).
• Environment variable AWS_PROFILE=ai-sandbox exported in the current shell.
• SSO token not expired (run aws sts get-caller-identity; if it errors, re-run aws sso login --profile ai-sandbox).
• Working directory is substrate/bootstrap/ in this repo.
• File substrate/bootstrap/terraform.tfvars exists with github_org = "PayCargoDevOps" (case-sensitive override of the lowercase default).

Commands

cd substrate/bootstrap

# Confirm identity one more time
aws sts get-caller-identity   # Account must be 959228203854

# Initialize Terraform (downloads AWS provider, uses local state)
terraform init

# Plan the apply. Output saved to tfplan so the same plan is what gets applied.
terraform plan -out=tfplan

# READ THE PLAN OUTPUT CAREFULLY before applying.
# Expected: ~8 resources, every one with a "+ create" marker.
# Anything other than creates (~ update, - destroy) means something is off —
# do not apply, paste the plan output to the team for diagnosis.

# If plan is clean, apply.
terraform apply tfplan

# Capture outputs (needed for Block 2, which will wire the remote backend).
terraform output > /tmp/bootstrap-outputs.txt
cat /tmp/bootstrap-outputs.txt

Apply takes 3-4 minutes. Most of the time is S3 bucket creation and the eventual-consistency wait for the IAM role to become visible.

7. Verification

After terraform apply reports success, verify each created resource is functional. All six commands should succeed without error.

# 1. State bucket exists, versioning ON, encryption ON with our CMK
aws s3api get-bucket-versioning --bucket ai-sandbox-tfstate
aws s3api get-bucket-encryption --bucket ai-sandbox-tfstate

# 2. Lock table exists with point-in-time recovery
aws dynamodb describe-continuous-backups --table-name ai-sandbox-tfstate-lock \
  | jq '.ContinuousBackupsDescription.PointInTimeRecoveryDescription.PointInTimeRecoveryStatus'
# Expected output: "ENABLED"

# 3. OIDC provider exists
aws iam list-open-id-connect-providers
# Expected output includes: "...:oidc-provider/token.actions.githubusercontent.com"

# 4. Deploy role exists and trust policy is scoped to PayCargoDevOps/ai-sandbox ONLY
aws iam get-role --role-name ai-sandbox-github-actions-deploy \
  | jq -r '.Role.AssumeRolePolicyDocument.Statement[0].Condition.StringLike."token.actions.githubusercontent.com:sub"'
# Expected output (three lines):
#   repo:PayCargoDevOps/ai-sandbox:ref:refs/heads/main
#   repo:PayCargoDevOps/ai-sandbox:ref:refs/tags/*
#   repo:PayCargoDevOps/ai-sandbox:pull_request
# The casing MUST match "PayCargoDevOps" (not lowercase) or CI will fail later
# with AccessDenied on AssumeRoleWithWebIdentity.

# 5. Permission boundary exists
aws iam list-policies --scope Local \
  --query 'Policies[?PolicyName==`ai-sandbox-permission-boundary`].Arn' \
  --output text
# Expected output: a single ARN.

# 6. KMS CMK exists with the expected alias
aws kms list-aliases --query 'Aliases[?AliasName==`alias/ai-sandbox-tfstate`].TargetKeyId' \
  --output text
# Expected output: a single key ID.

8. How to view the work in AWS Console

Bootstrap creates six resource classes. Each is visible in the AWS Console at a specific place. Going through these in order is the best way to build a mental model of what's live.

Worth a side trip: the Resource map

None of these show up in a single visual yet (they're not network resources). The visual story of bootstrap is told via the relationships in each individual page above. Step 3's "Resource map" (VPC topology) is the first place where the visual comes into its own.

9. Anticipated gotchas

These are the failure modes most likely to surface during apply. Listed in order of frequency.

terraform import aws_iam_openid_connect_provider.github \
  arn:aws:iam::959228203854:oidc-provider/token.actions.githubusercontent.com
terraform apply tfplan

state_bucket_name = "ai-sandbox-tfstate-959228203854"

9. Go / no-go gates

10. Doctrine implications

Bootstrap is where the platform's doctrine becomes structural rather than aspirational. Three principles get enforced here in ways that future steps cannot undo.

The permission boundary is forever

From the moment bootstrap's permission boundary policy is created and attached to the first downstream role, every IAM role in the account inherits its constraints. The boundary is the single enforcement point for "no IAM users may be created in this account," "the OIDC provider cannot be deleted by an automation role," "logging cannot be disabled by an automation role." These properties become structural facts, not policy memos.

CI replaces laptop applies on day one

After bootstrap succeeds, the GitHub Actions deploy role exists. From this point forward, the operator's laptop should never run terraform apply against the AI-sandbox account except for re-applying bootstrap itself or in genuine break-glass situations. Every downstream module is applied by CI, which means every apply is a git commit that anyone can review, every apply has an audit trail in the GitHub workflow log, and no apply requires AWS credentials on a laptop.

The chicken-and-egg is permanently solvable

Keeping bootstrap on local state means a future operator (years from now, post-laptop-loss, post-audit-recovery, post-account-rebuild) can always reconstruct bootstrap from the Terraform code in the repo plus AWS admin credentials, with no dependency on prior state existing. This is the property that makes the platform's "everything in code" claim genuine rather than fragile.

12. What actually happened during apply

Apply executed in two passes due to one anticipated gotcha. Documented here so the next engineer learns from the actual run, not just the theoretical plan.

Pass 1 — partial success (7 of 13)

First terraform apply tfplan created the following resources in order:

• OIDC provider (1s)
• KMS CMK + alias (15s)
• Permission boundary IAM policy (0s)
• IAM deploy role + inline policy (1s)
• DynamoDB lock table (10s)

Then failed on aws_s3_bucket.tfstate with:

Error: creating S3 Bucket (ai-sandbox-tfstate):
  StatusCode: 409, BucketAlreadyExists

This is Gotcha B from Section 8 — S3 bucket names are global, and ai-sandbox-tfstate was already owned by another AWS account somewhere in the world.

Fix — suffix the bucket name with the account ID

Added one line to terraform.tfvars:

state_bucket_name = "ai-sandbox-tfstate-959228203854"

Account IDs are unique across all of AWS, so this guarantees the bucket name is also unique.

Pass 2 — clean completion (6 of 6 remaining)

Re-ran terraform plan; output was Plan: 6 to add, 0 to change, 0 to destroy — Terraform recognized the 7 resources from Pass 1 as already existing and only re-planned the bucket and its 5 configuration sub-resources. Apply finished in ~60 seconds; all 6 resources created cleanly.

11. What comes next

After bootstrap apply succeeds and all six verification commands return as expected:

1. Push to the repo. Push the Terraform code to PayCargoDevOps/ai-sandbox so it lives in the canonical place. CI is now wired (deploy role exists, OIDC trust scoped to this repo) but no workflow is in place yet to use it; Block 2 (Identity) will be applied locally for now, with CI workflows landing in a later block.
2. Migrate the environments composition to remote state. The output backend_config_block from bootstrap is the snippet that drops into substrate/environments/ai-sandbox/backend.tf. After that, terraform init -migrate-state in the composition directory wires it to remote state.
3. Proceed to Block 2: Identity. This deploys three trust-zone KMS CMKs (agent, lockbox, audit) and the tier IAM role templates (sor-reader, internal). Cost: $3/month. Risk: none — no compute, no network. See Step.2.html when generated.

If bootstrap fails or verification surfaces an issue, do not proceed to Block 2. Diagnose and either re-apply bootstrap or, in the worst case, destroy and re-apply (only the KMS CMK has a deletion delay; everything else is recreatable in minutes).