3aWire CI (GitHub Actions OIDC apply path)

1. Why this step exists (and why it has been deferred until now)

Bootstrap (Step 1) created two resources whose entire purpose is letting CI assume an AWS role: the GitHub OIDC provider, and the ai-sandbox-github-actions-deploy role with a trust policy scoped to PayCargoDevOps/ai-sandbox. Those resources are live and the trust policy is correctly cased. They just are not being used — because no GitHub Actions workflow file references them.

Why this step was deferred until after Step 3

• The substrate had to exist to be applied. Wiring CI before there is anything for CI to do is mostly ceremony.
• The operator's mental model had to be built block-by-block first. Doing the first 4 blocks under direct laptop control means the operator has seen exactly what each terraform apply does, which is the prerequisite for safely letting CI do them later.
• The doctrine of -target block-by-block was easier to enforce under human-driven applies. Once CI is in, the natural pattern is full-composition applies per PR.

Why deferring further would be a mistake

The cost of doing the next four blocks (Step 4a through 5g) by laptop apply is concrete: every one of those modules will have been "applied without going through the CI path that exists for exactly this purpose." The doctrine of "no long-lived AWS credentials anywhere" stays partially true (SSO is short-lived) but the auditable, peer-reviewable, blame-attributable apply trail lives only in terraform.tfstate, not in GitHub. After this step, every apply has a corresponding GitHub workflow run with a named human approver in the audit trail.

2. What this step creates

What this step does NOT create

• No new AWS resources. Everything CI needs already exists (deploy role, OIDC provider, permission boundary).
• No secrets in GitHub. OIDC means no static AWS keys are stored anywhere. The repo has zero AWS credentials.
• No automated rollback. CI can apply changes; rolling back requires a new PR that reverts. Intentional — rollback should be visible in git history.

3. The two workflows in detail

3.1 Plan workflow (read-only, runs on every PR)

# .github/workflows/terraform-plan.yml
name: Terraform Plan

on:
  pull_request:
    paths:
      - 'terraform/**'
      - '.github/workflows/terraform-*.yml'

permissions:
  id-token: write       # required to mint the OIDC token
  contents: read        # required to checkout the repo
  pull-requests: write  # required to post the plan as a PR comment

jobs:
  plan:
    name: Plan
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials via OIDC
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::959228203854:role/ai-sandbox-github-actions-deploy
          aws-region: us-east-1
          role-session-name: gha-plan-${{ github.run_id }}

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.9.8

      - name: Terraform init
        working-directory: substrate/environments/ai-sandbox
        run: terraform init

      - name: Terraform plan
        id: plan
        working-directory: substrate/environments/ai-sandbox
        run: terraform plan -no-color -out=tfplan 2>&1 | tee plan-output.txt

      - name: Post plan as PR comment
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const plan = fs.readFileSync('substrate/environments/ai-sandbox/plan-output.txt', 'utf8');
            const truncated = plan.length > 60000 ? plan.slice(0, 60000) + '\n\n... (truncated)' : plan;
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: '## Terraform Plan\n\n```\n' + truncated + '\n```'
            });

3.2 Apply workflow (write, gated by manual approval)

# .github/workflows/terraform-apply.yml
name: Terraform Apply

on:
  push:
    branches: [main]
    paths:
      - 'terraform/**'

permissions:
  id-token: write
  contents: read

jobs:
  apply:
    name: Apply
    runs-on: ubuntu-latest
    environment: production   # <-- This is the manual-approval gate
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials via OIDC
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::959228203854:role/ai-sandbox-github-actions-deploy
          aws-region: us-east-1
          role-session-name: gha-apply-${{ github.run_id }}

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.9.8

      - name: Terraform init
        working-directory: substrate/environments/ai-sandbox
        run: terraform init

      - name: Terraform plan
        working-directory: substrate/environments/ai-sandbox
        run: terraform plan -out=tfplan

      - name: Terraform apply
        working-directory: substrate/environments/ai-sandbox
        run: terraform apply tfplan

4. The agentic + human-in-the-loop sequence

This step is the first one where the apply itself is structured as an explicit dance between automated work and human approval. The sequence below is the exact order of operations, with the actor for each row.

How to read the color coding

• Agent — work done by Claude (writing files) or by GitHub Actions runners (executing workflows). Mechanical, deterministic, repeatable.
• Human — work done by the operator. Reviewing, approving, configuring repo settings. The decision-making and accountability moments.
• Handoff — the moment work transitions from one actor to the other. These are the points where the chain of custody for the change passes hands; each one is a checkpoint.

The doctrine: agents do the mechanical work, humans do the irreversible decisions, and every transition is an explicit checkpoint. This sequence is the first place that pattern is structurally enforced in CI rather than enforced socially in chat.

5. GitHub environment protection setup (the human-only step)

The environment: production reference in the apply workflow is meaningless until the environment is actually configured in the repo. This step is done in the GitHub UI; it cannot be done in code.

Exact clicks

1. Go to https://github.com/PayCargoDevOps/ai-sandbox/settings/environments
2. Click New environment.
3. Name: production. Click Configure environment.
4. Under Deployment protection rules, check Required reviewers.
5. Add yourself (and any other approvers). The order in which you add them does not matter; any one of them can approve.
6. Under Deployment branches and tags, choose Selected branches and tags. Add a rule: main. (This means only the main branch can deploy through this environment; PRs and feature branches cannot.)
7. Click Save protection rules.

What this enforces

• A push to a non-main branch cannot trigger an apply (branch policy).
• Even on main, the apply pauses until a designated reviewer approves (required reviewers).
• The reviewer cannot be the same person who pushed the change (GitHub auto-enforces this if "Prevent self-review" is checked; we keep it checked by default).
• The approver's identity and approval timestamp are recorded in the workflow run history and never editable.

6. How to test (the first CI run)

The first time CI runs, both workflows should fire and complete successfully without changing anything in AWS. That is the proof that the path works.

The test sequence

1. After the workflow files are committed and the production environment is configured, push to a feature branch and open a PR.
2. Observe the Terraform Plan workflow run start automatically on the PR.
3. Within 2-3 minutes, a comment appears on the PR with the terraform plan output. Expected content: No changes. Your infrastructure matches the configuration.
4. Merge the PR.
5. Observe the Terraform Apply workflow run start on the merge. It runs init + plan, then pauses with a "Waiting for review" badge.
6. Go to the Actions tab. Click the paused run. Click Review deployments. Approve.
7. The apply step runs. Expected output: Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
8. Done. CI is wired and operational.

Optional second test (after the first succeeds)

Push a real but harmless change to verify CI handles diffs. Recommended: add a new tag to one of the existing resources (e.g., git_commit_sha) so the plan shows 1 to change. The apply should successfully modify the resource. Visible end-to-end: code change → PR comment with plan → merge → approval → apply → resource updated.

7. How to view the work in GitHub + AWS Console

8. Anticipated gotchas

9. Go / no-go gates

10. Doctrine implications

The "no long-lived credentials" guarantee is now structural

Until this step, the doctrine was true in spirit (SSO admin tokens expire in 12 hours) but not in practice (any operator with SSO admin could apply at any time, and applies were happening by hand). After this step, the production apply path is GitHub OIDC, which produces ~10-minute tokens that expire automatically. SSO admin sessions still exist for break-glass and for bootstrap-level changes, but they are no longer on the routine apply path.

Every apply now has a named human approver in the audit trail

The CloudTrail AssumeRoleWithWebIdentity event records the GitHub Actions principal. The GitHub workflow run records the human approver. Cross-referencing the two gives you a permanent named-human-plus-machine-identity audit trail for every infrastructure change. No more "who applied this last Tuesday" mysteries.

Agents do mechanical work, humans do irreversible decisions

The handoff sequence in § 4 is the first place this pattern is enforced by infrastructure rather than by good intentions. CI cannot apply without human approval. The human cannot accidentally apply without going through the plan + approval flow. The two are locked together — not because anyone could be trusted to follow the convention, but because the convention is encoded in the deployment protection rules.

11. What comes next

After Step 3a is complete:

1. Mark Step 3a complete and update _process/index.html.
2. Push the Step 3 network-firewall.tf bug fix through the new CI path — the first real change to flow through GitHub Actions OIDC. Visible end-to-end test of the CI pipeline.
3. Begin Step 4a: Data — the audit S3 bucket + lockbox DDB table + per-zone secrets. First Step 4 sub-block. From here forward, every module change happens via the CI path documented in this doc.

For the rest of the build, the agentic + human-in-the-loop sequence in § 4 is the canonical pattern. Agent writes the Terraform; human reviews via PR; CI plans and posts; human approves via PR merge; CI applies through the production environment; human approves the apply; the change is live and audited. Every block from 5a onward will look exactly like that.