PayCargo · AI-Sandbox · Process

5NFW enforcement flip — alert → drop, DNS ALERT → BLOCK

The substrate's network egress posture transitions from observe-only to enforce. Allowlist that's been audited for 30 days under real obs-app traffic becomes the binding contract for every future vibe app's outbound calls.
Block: 5 (substrate hardening completion) · Risk: Medium — first network change that can affect live traffic · Reversible: Yes — single tfvar flip back to alert
Pending

State: NFW in alert mode since Step 3 deploy. DNS Firewall in ALERT mode. SoR allowlist captured 30+ days of real obs-app traffic. No surprise destinations in the alert log.

Why this exists: Alert mode proves the rules without enforcing them. After enough real traffic has flowed through alert without surprises, the safe move is to flip to drop. Until the flip, the substrate's egress-defense doctrine is aspirational; after, it's enforced.

1. What changes

Two tfvars in substrate/environments/ai-sandbox/terraform.tfvars:

# Before
nfw_enforcement_mode  = "alert"
dns_firewall_action   = "ALERT"

# After
nfw_enforcement_mode  = "drop"
dns_firewall_action   = "BLOCK"

Substrate modules (network) react to these toggles:

2. Why this is safe now

ValidationWhat it proves
30+ days of NFW alert logsThe allowlist of SoR FQDNs (Salesforce, Monday, GitHub, GitBook) matches actual obs-app outbound traffic. No unexpected destinations.
30+ days of DNS Firewall alert logsNo unexpected DNS lookups from agent-tier workloads. The DNS allowlist matches reality.
No application-level failures attributable to NFW/DNSApps that work in alert will work in drop because the allowlist is the same; only the response to non-matching traffic changes.
Rollback is a single PRIf drop mode breaks something, flip the tfvar back to alert in <5 minutes via the same CI workflow.

3. Pre-flip checklist

  1. Pull last 7 days of NFW alert logs from CloudWatch: aws logs filter-log-events --log-group-name /aws/network-firewall/ai-sandbox. Confirm zero unexpected destinations (only known SoRs).
  2. Pull last 7 days of DNS Firewall query logs: aws route53resolver list-resolver-query-log-config-associations + LogsInsights queries. Confirm zero unexpected DNS names.
  3. Verify the obs app's Lambda execution role egress patterns are stable (no new outbound destinations in the past week).
  4. Confirm the SES domain verification work is either complete OR scheduled separately — Step 5 is independent of that work.
  5. Confirm at least one operator is available to monitor for the first hour after flip.

4. The flip itself

Standard CI flow:

  1. Branch: step-5-nfw-enforcement-flip
  2. Edit substrate/environments/ai-sandbox/terraform.tfvars — flip the two tfvars
  3. Commit, push, open PR
  4. CI plan workflow runs (substrate-plan.yml). Expected diff: in-place update on aws_networkfirewall_firewall_policy + aws_route53_resolver_firewall_rule. Should be small, ~2-3 resources.
  5. Review plan output. Confirm no resources destroyed.
  6. Merge to main
  7. Dispatch substrate-apply.yml with confirm = APPLY
  8. Apply completes in ~2 minutes
  9. Watch CloudWatch for the next hour for any sudden uptick in dropped packets or NXDOMAIN responses

5. Rollback procedure (if needed)

If anything breaks unexpectedly after the flip:

  1. Same branch / new branch with the two tfvars reverted to alert / ALERT
  2. Plan + apply via the same CI workflow
  3. ~2 minutes from PR open to drop-mode disabled
  4. Investigate root cause in alert mode; identify the missing allowlist entry; add it; re-flip when ready

Doctrine: the flip is reversible

The substrate's "modeled completely" principle means every irreversible-sounding change is built to be reversible at the toggle level. NFW enforcement is one tfvar; reverting is one PR; the rollback path is the same as the apply path, just inverted. This is what makes Step 5 a safe move rather than a leap of faith.

6. What this completes

7. Coordination with other pending work

Step 5 is independent of:

Best timing: after the SES domain verification work lands (so any DNS hiccups during that work don't get blamed on Step 5).