PayCargo · AI-Sandbox · Step 6.m

6.mTiered inference: the model router substrate and the moat it sets up

Every Bedrock-invoking cartridge in the substrate goes through a model router that maps logical routes (scout-local, scout-cloud, writer-opus) to backend endpoints (Mac Mini, Bedrock, future). Routing is data, not code. The substrate decision lets PayCargo absorb every quality improvement in open-source inference at zero refactor cost, while keeping sensitive code structurally off third-party APIs.
Block: 6.m (substrate doctrine; companion to 6.k schedules and 6.j chatbot guardrail wiring) · State: Specced; ships behind code-agents Phase 1.5 · Module: substrate/modules/platform/model-router/
Doctrine
This doc is the substrate decision underneath the cartridges. We do not call Bedrock directly from any cartridge Lambda anymore. We call the model router with a logical route name. The router decides which backend serves that route. Operators reroute without code changes. As open-source models close the gap on frontier quality, the router lets us slide compute from cloud to local one route at a time, without touching cartridge code.

Contents

  1. Why this is substrate, not a cartridge concern
  2. The shape of the router
  3. Routing is data: the per-turn config
  4. Fallback chains and resilience
  5. The three-generation trajectory
  6. Why this is a moat
  7. What this commits the substrate to
  8. What's deferred

1. Why this is substrate, not a cartridge concern

Every Bedrock-invoking cartridge in the platform (chatbot, the obs Lambda's narrative path, the future code-agents, every agent ported from paycargo-ai) is one decision deep: which model to call. That decision lives today as a hard-coded modelId string in cartridge code, or as a manifest entry in the cartridge's DDB. When a cheaper or better model lands, every cartridge has to change.

That is the wrong place for the decision. Cartridges should call into substrate by logical role ("the scout role for my recon turn," "the writer role for my output turn"), not by vendor-specific model ID. The substrate maps roles to backends. When a new backend becomes available (an in-house model server, a new Bedrock model, a peered partner's API), the substrate gets one new entry in the route table and every cartridge benefits.

This is the same separation the substrate already enforces for KMS keys (cartridges declare data class, substrate selects key), for guardrails (cartridges declare risk tier, substrate selects guardrail), and for network egress (cartridges declare egress need, substrate selects allowlist entry). Tiered inference fits the same pattern: cartridges declare role, substrate selects backend.

2. The shape of the router

A small Terraform module plus a shared zero-npm client library. ~200 LOC total.

substrate/modules/platform/model-router/ main.tf SSM Parameter Store: /ai-sandbox/model-router/routes published JSON manifest, operator-editable via obs UI outputs.tf ssm_parameter_name (cartridges read at cold start) variables.tf routes input (default routes registered at apply) versions.tf cartridges/<cartridge>/lambda/model-router.js (copied into each Bedrock-invoking cartridge until we promote to a layer) invokeRoute(routeName, prompt, opts) => 1. Read the route table from SSM (cached 15min in module scope) 2. Try primary backend 3. On timeout/error, try fallback chain in order 4. Return result + metadata (which backend served, latency, tokens) 5. Emit *-complete log line with route + backend

The route table itself is a JSON manifest in SSM Parameter Store. Same pattern the schedules substrate uses (Step 6.k). Operator edits land within 15 minutes of cold start without code redeploy.

Default routes registered at apply: 

{
  "routes": {
    "scout-local": {
      "backend": "openai-compat",
      "url":     "http://10.42.0.10:11434/v1/chat/completions",
      "auth":    { "type": "bearer", "secret_arn": "...mac-mini-token" },
      "model":   "qwen2.5-coder:14b",
      "timeout_ms": 5000,
      "fallback": "scout-cloud"
    },
    "scout-cloud": {
      "backend": "bedrock",
      "modelId": "us.anthropic.claude-haiku-4-5-20251001-v1:0",
      "timeout_ms": 10000,
      "fallback": null
    },
    "writer-opus": {
      "backend": "bedrock",
      "modelId": "us.anthropic.claude-opus-4-6-<datestamp>-v1:0",
      "timeout_ms": 60000,
      "fallback": "writer-sonnet"
    },
    "writer-sonnet": {
      "backend": "bedrock",
      "modelId": "us.anthropic.claude-sonnet-4-5-20250929-v1:0",
      "timeout_ms": 30000,
      "fallback": null
    }
  }
}

3. Routing is data: the per-turn config

The (model, prompt) doctrine from Step 6.l becomes (route, prompt). An agent's turn list in DDB references logical routes: 

{
  "pk": "code-agents-config",
  "sk": "golden-spec",
  "turns": [
    { "name": "recon",  "route": "scout-local",  "prompt": "...", "max_tokens":  400 },
    { "name": "plan",   "route": "scout-local",  "prompt": "...", "max_tokens":  500 },
    { "name": "gather", "route": "scout-local",  "prompt": "...", "max_tokens":  300 },
    { "name": "write",  "route": "writer-opus",  "prompt": "...", "max_tokens": 4000 }
  ]
}

Two operators can serve the same agent from different backends just by editing the route table. A development environment can route writer-opus to Sonnet for cheaper testing. A production environment in a regulated tenant can route the same role to an on-prem endpoint. The cartridge code does not know.

The obs Cartridges tab's Trigger config section surfaces the per-turn route as a dropdown populated from the active SSM manifest. Adding a new route to the substrate makes it appear in every cartridge's dropdown automatically.

4. Fallback chains and resilience

Local inference has a different failure mode than cloud inference. The Mac Mini can be off, the Tailscale net can be flaky, the model server can be loading a different model. The router treats every route as having a primary backend and a fallback chain.

Try primary, timeout at the configured ms, move to the next fallback in the chain. Each fallback hop logs the failure (so the operator sees that the Mac Mini was unreachable). The agent does not fail unless every backend in the chain fails. The obs Cartridges tab gains a "Fallback rate" column per route so the operator can see when a backend is sick.

The default chain at GA: scout-local → scout-cloud → null and writer-opus → writer-sonnet → null. When the Mac Mini is healthy, scouts run free. When it is down, scouts fall over to cloud Haiku at $0.005-$0.03 per turn (acceptable burst cost during an outage). When Opus is throttled by Anthropic's regional capacity, the writer falls to Sonnet automatically.

5. The three-generation trajectory

This substrate makes a specific bet on where open-source inference is heading. The bet is concrete.

GenerationMac Mini / on-prem can doCloud is forCost per golden-spec invocation
Today (Phase 1.5) Scout, plan, gather. Navigation and cheap thinking. Qwen 2.5 Coder 14B-32B is good at "given a tree, pick the relevant files." Writer turn. Opus 4.6 or Sonnet 4.5. ~$0.48 (Opus writer) or ~$0.10 (Sonnet writer)
6-12 months (Phase 2) Most writer turns for routine work. 70B-class open models on a Mac Studio with 192GB unified memory will hit Sonnet 3.5-class quality on coding tasks. The next generation of Qwen Coder and DeepSeek Coder are converging fast. Genuinely novel architectural problems. Becomes the "I am uncertain, escalate" tier. ~$0.05-$0.10
1-2 years (Phase 3) Frontier open-source approaches Opus-class. Local handles nearly all PayCargo agent work. Verification and the rare genuinely hard problem. Near zero (electricity + occasional cloud escalation)

Two trends compound in favor of the substrate. Bedrock pricing is rising as Anthropic invests in better models. Open-source quality is rising while local inference cost falls toward electricity. The gap widens. The substrate captures every step of that widening without any cartridge code change.

6. Why this is a moat

The unit economics compound in our favor

Competitors who pile features on hosted-only Bedrock can match velocity for now. They cannot match unit economics once open-source crosses a quality threshold. Their cost structure is locked to Anthropic's pricing power; ours is locked to electricity. Five years out the gap is plausibly a 10x to 100x cost differential. A competitor at that point either sacrifices margin to match price or sacrifices price to match margin. Substrate compounds. Bolt-on features do not.

Sovereignty becomes structural, not promised

Scout turns processing customer code or PII never leave the perimeter once the Mac Mini handles them. As Phase 2 lands, the writer turn moves local too for routine work. By Phase 3 nearly nothing transits a third-party API. PayCargo's compliance posture becomes "by construction" rather than "by policy." That is a sellable position to financial regulators and to enterprise customers who do not want their architecture decisions sent to OpenAI or Anthropic.

New model classes plug in instantly

When a vendor ships a small-context-window specialist optimized for "given a file tree, return relevant paths," it is one entry in the router. When DeepSeek's next coder model hits 90% Sonnet quality at 1% the cost, it is one entry. When a regulated tenant requires a specific vetted model class for HR data, it is one entry scoped to that tenant. The substrate does not pick winners; it stays optionable.

Competitor lock-in becomes their cost, not ours

Vibe-coded competitors who write new BedrockRuntimeClient() directly in every Lambda inherit Anthropic's pricing power as a permanent dependency. Migration to local inference for them is a refactor across every cartridge. For us, it is a route table edit. The cost of switching matters more in five years than it does today, and asymmetric switching cost is one of the most durable forms of moat.

7. What this commits the substrate to

  1. No raw Bedrock clients in cartridge code. Every Bedrock-invoking Lambda uses the model router. Code review rejects PRs that import BedrockRuntimeClient directly outside the router itself.
  2. Every model invocation goes through a route name. Logical names, not vendor strings. The route table is the only place vendor IDs appear.
  3. Every route declares a fallback chain. No agent fails because the operator's preferred backend is down. Routes default to graceful degradation to cloud.
  4. Operators tune routes; engineers ship abstractions. Route table edits in the obs UI take 15 minutes to land. No deploy. Engineers reserve their time for new agents and new substrate, not model-ID find-and-replace PRs.
  5. The route table is auditable. Every *-complete log line carries the route name and the backend that actually served it. CloudTrail shows the SSM reads. Cost attribution per backend is visible in the obs Cartridges tab.

8. What's deferred