3Apply Network (Stages 1-6, alert mode)

1. Why this step exists

Bootstrap (Step 1) created the substrate. Identity (Step 2) created the trust zones at the cryptographic and IAM layers. Network (Step 3) creates the trust zones at the physical layer — the actual VPC, subnets, route tables, and firewall surfaces that workloads live in. After this step, every downstream resource (data, observability, lambda-tools, runtime) has somewhere to attach.

Why this block is the biggest by far

The network surface is where AWS gives you the most knobs — subnet topology, route tables, security groups, NACLs, VPC endpoints, NFW rule groups, DNS Firewall, flow logs, log destinations, encryption keys. Each is a small decision. The 82-resource count reflects how many distinct decisions are encoded in the design, not gratuitous complexity. The Network Module Reference document (_docs/Network Module Reference.html) walks the topology in detail.

2. What this step creates (82 resources, categorized)

All resources are tagged with Project=ai-sandbox, Owner=platform-team, ManagedBy=terraform, Module=network.

3. The three trust-zone topology

The doctrine: every subnet in the VPC belongs to exactly one of three trust zones, each with a distinct purpose and a distinct internet posture.

The egress packet chain

agent ENI → agent-sg (443 outbound only) → agent-private route table
          → NFW endpoint in egress subnet
          → (TLS SNI inspected against 4-SoR allowlist)
          → NAT Gateway in egress subnet
          → (source-NAT to NAT's EIP)
          → IGW
          → SoR API (Salesforce / Monday / GitHub / GitBook)

The AWS-API path (separate from the egress chain)

agent or lockbox ENI → VPC interface endpoint (in same subnet, private DNS)
                     → AWS service (Bedrock / KMS / Secrets / STS / Logs / ECR)

AWS API traffic never traverses NAT, never traverses IGW, never hits NFW. Cheaper, faster, more secure, and keeps the NFW allowlist focused on the 4 third-party SoRs.

4. What "alert mode" means — and why we defer enforcement

Both the Network Firewall and the Route53 DNS Firewall land in alert mode. This is the safe state.

NFW in alert mode (vs. drop mode in Step 5)

DNS Firewall in ALERT mode (vs. BLOCK mode in Step 5)

Same pattern: ALERT logs DNS queries that match the catch-all rule but resolves them anyway. BLOCK responds with NXDOMAIN.

Why we defer the flip to Step 5

1. Alert log is only meaningful when real workloads exist. Until Step 4 deploys lambda-tools, the agent runtime, and the first agent, the alert log mostly captures AWS-internal noise. The 4-SoR allowlist needs to be validated against actual agent traffic.
2. Observability has to be live to monitor the flip. CloudTrail, GuardDuty, and Bedrock invocation logging are part of the observability block (Step 4). Without them, flipping NFW to drop is flying blind.
3. The audit S3 bucket from the data block is a Stage 9 dependency. Network has a "flow logs to S3" stage that needs the data block's audit bucket; doing data first means network can reach final form in one pass.

5. Cost & permanence

Cost (single-AZ POC mode, locked in terraform.tfvars)

Two-AZ graduation (internal app / SaaS replacement tier)

When the platform graduates from POC to supporting internal apps (PayCargo's own SaaS-replacement workloads), the next step is two-AZ resilience — not full three-AZ production. Two AZs adds the cheapest meaningful failover (an AZ-a outage no longer breaks egress); three AZs adds marginal additional resilience at meaningful additional cost and is reserved for true production posture later.

Why 2 AZ is the right next tier (not 3)

• Failover, not redundancy. Two AZs gives "an AZ outage does not break egress," which is the single property POC is missing. Three AZs gives "two simultaneous AZ outages do not break egress," which is rarely the operational threat model that justifies the extra ~$395/month.
• Internal apps don't need three-nines. Internal SaaS replacement (the workloads PayCargo would put here) typically targets ~99.5% availability, which two-AZ comfortably supports. Customer-facing production where every minute of downtime has direct revenue impact is the workload that earns the third AZ.
• Graduation is reversible. The 2-AZ tier can be promoted to 3-AZ later by changing the same variable; there is no architectural rebuild required, just a Terraform apply.

Trade-off remaining at 2 AZ

A two-AZ deployment survives an AZ outage by routing traffic through the surviving AZ. The trade-off is that cross-AZ data charges ($0.01/GB) apply for workloads in AZ-c calling endpoints in AZ-a or AZ-b. At any reasonable sandbox or internal-app traffic volume, this is negligible (sub-$10/month). At sustained high-traffic production, it becomes a third-AZ argument.

Variable cost on top of standing

NFW data processing ($0.065/GB), NAT data processing ($0.045/GB), interface endpoint data ($0.01/GB), cross-AZ data ($0.01/GB). At POC traffic volumes (sub-1GB/day combined), variable cost is well under $20/month.

Permanence

Permanent. Tearing down the network breaks every downstream module that has any workload attached. NAT and EIP have a few-second creation time but a sticky deletion (release EIP, then delete NAT, then delete subnet routing). Don't terraform destroy network casually.

6. How to apply

Preconditions

• Step 2 (Identity) complete and verified.
• Composition on remote state (verified during Step 2).
• substrate/environments/ai-sandbox/terraform.tfvars has safety toggles in safe positions:
  • nfw_enforcement_mode = "alert"
  • dns_firewall_action = "ALERT"
  • flow_logs_destination = "cloudwatch"
  • az_redundancy = "single"
• SSO session active. Token lifetime is 12h; this apply takes ~15-20 minutes, well within the window, but re-authenticate if you're near the boundary.

Commands

cd substrate/environments/ai-sandbox

# Plan ONLY the network module (already done; re-plan if anything's changed)
terraform plan -target=module.network -out=network.tfplan

# READ the plan. Expected:
#   Plan: 82 to add, 0 to change, 0 to destroy.
# All resource lines should be + create. NFW + VPC endpoints dominate the count.

terraform apply network.tfplan

# Capture outputs for downstream blocks (data, observability, lambda-tools)
terraform output > /tmp/network-outputs.txt

Expected apply duration breakdown

7. Verification

# 1. VPC exists with the right CIDR
AWS_PROFILE=ai-sandbox aws ec2 describe-vpcs \
  --filters "Name=tag:Project,Values=ai-sandbox" \
  --query 'Vpcs[].{VpcId:VpcId,Cidr:CidrBlock,State:State}' --output table

# 2. 9 subnets exist (3 AZs x 3 tiers)
AWS_PROFILE=ai-sandbox aws ec2 describe-subnets \
  --filters "Name=tag:Project,Values=ai-sandbox" \
  --query 'length(Subnets)' --output text
# Expected: 9

# 3. NFW is live and the firewall policy is attached
AWS_PROFILE=ai-sandbox aws network-firewall describe-firewall \
  --firewall-name ai-sandbox-nfw \
  --query '{Status:FirewallStatus.Status, Policy:Firewall.FirewallPolicyArn}'

# 4. NFW is in ALERT mode (stateful default action)
AWS_PROFILE=ai-sandbox aws network-firewall describe-firewall-policy \
  --firewall-policy-name ai-sandbox-nfw-policy \
  --query 'FirewallPolicy.StatefulDefaultActions' --output text
# Expected: aws:alert_strict (or similar alert-mode default)

# 5. DNS Firewall rule group association is in ALERT mode
AWS_PROFILE=ai-sandbox aws route53resolver list-firewall-rules \
  --firewall-rule-group-id "$(aws route53resolver list-firewall-rule-groups \
    --query 'FirewallRuleGroups[?Name==\`ai-sandbox-dns-fw\`].Id' --output text)" \
  --query 'FirewallRules[].{Domain:FirewallDomainListId,Action:Action}' --output table
# Expected: catch-all rule has Action=ALERT (not BLOCK)

# 6. 13 VPC endpoints (2 gateway + 11 interface)
AWS_PROFILE=ai-sandbox aws ec2 describe-vpc-endpoints \
  --filters "Name=tag:Project,Values=ai-sandbox" \
  --query 'length(VpcEndpoints)' --output text
# Expected: 13

# 7. Flow logs are configured and writing to CloudWatch (audit CMK)
AWS_PROFILE=ai-sandbox aws ec2 describe-flow-logs \
  --filter "Name=tag:Project,Values=ai-sandbox" \
  --query 'FlowLogs[].{LogDest:LogDestination, FlowLogStatus:FlowLogStatus}' --output table
# Expected: FlowLogStatus=ACTIVE, LogDest contains "ai-sandbox" log group

8. How to view the work in AWS Console

This is the step where the visual story really helps. The VPC has nine subnets, three trust zones, a firewall, two NATs, and 14 endpoints — reading about it in text is one thing; seeing the topology in the AWS Console builds the mental model that makes the doctrine obvious. The "Resource map" view is especially worth a minute.

Recommended sequence for the first viewing (~15 minutes total)

1. Start with the Resource map — gives you the visual gestalt. Spend 3-5 minutes just looking at the topology. Click around. Note which subnets connect to which.
2. Open a lockbox route table — confirm no 0.0.0.0/0. This is the doctrine.
3. Open the NFW firewall policy — see aws:alert_strict. This is the safe-mode posture.
4. Open the NFW rule group — see the 4 SoRs as data. This is the allowlist.
5. Open the audit CMK key policy (Step 2) in a new tab and verify the flow log group's encryption references it. Cross-step doctrine connection visible.

9. Anticipated gotchas

aws_networkfirewall_firewall.main: Still creating... [11m20s elapsed]

9. Go / no-go gates

10. Doctrine implications

Structural enforcement at the routing layer

The lockbox subnets have no default route. This is the strongest enforcement possible: the internet is not unreachable because of a policy decision, it is unreachable because there is no route for the packet to take. Removing the route is a structural property of the subnet, not a state of a firewall. Even if NFW is misconfigured, even if SGs are wide open, even if IAM grants every permission — a packet from a lockbox workload destined for the internet has nowhere to go.

Two independent egress defense layers

NFW operates at L4 with stateful TLS SNI inspection. DNS Firewall operates at L7 with query-time hostname filtering. They are independent — a gap in one is covered by the other. Both are configured with the same 4-SoR allowlist, but they enforce it via different mechanisms. Defense in depth.

The egress chokepoint is identity-blind

NFW does not know which IAM role originated a packet — it sees TLS handshakes and TCP/IP headers only. That is intentional: the chokepoint enforces "this destination is allowed for any workload" rather than "this destination is allowed for this role." Role-level scoping happens at the application layer (IAM policies on tier roles), and the network layer enforces the floor that no role can bypass.

Alert-first deployment is a doctrine, not a quirk

Every traffic-affecting block in this build lands in observation mode first. Alert mode for NFW, ALERT for DNS Firewall, log-only for any future WAF. Enforcement is a separate, deliberate operator decision, gated on observed traffic patterns. The doctrine: never let an automation make the first enforcement decision.

11. What comes next

After network apply succeeds and verification passes:

1. Mark Step 3 complete in Step.3.html and update index.html.
2. Stay in alert mode. Do not flip nfw_enforcement_mode or dns_firewall_action yet. They flip in Step 5, after Step 4 deploys real workloads.
3. Begin Step 4 — the workload layer. Six sub-blocks: data, observability, guardrails, memory, gateway, lambda-tools, first agent. Each will have its own Step.5.x.html as we get there.

Network is now the substrate the rest of the platform attaches to. Every downstream module references at least one of network's outputs (vpc_id, agent_subnet_ids, lockbox_subnet_ids, agent_security_group_id, endpoint_security_group_id, etc.).