6.kScheduled-jobs substrate — Scheduler + DLQ + watchdog + UI

1. Why this exists

Three things broke silently with the old narrative wiring; the same three would break with every future scheduled job unless the substrate fixed the pattern:

1. No "did this fire?" check. EventBridge marks the rule "fired"; Lambda errored; only CloudWatch Logs knew. Nobody reads CW Logs daily.
2. No retry on failure. One bad fire = that day's run is gone.
3. No declared expectation. A watchdog can't say "this should fire daily at 09:00" because it's implicit in the Terraform, not in a queryable surface.

The fix is one substrate primitive (the schedules module) plus three thin layers (handler wrapper for run records, watchdog Lambda with SES alerts, Schedules tab in obs).

2. The substrate module

substrate/modules/platform/schedules/ takes a map of jobs and emits:

Plus one account-wide resource:

Job declaration shape (env composition)

module "platform_schedules" {
  source = "../../modules/platform/schedules"
  jobs = {
    "obs-app-narrative" = {
      cron               = "cron(0 9 * * ? *)"
      timezone           = "America/New_York"
      description        = "Daily exec narrative"
      app_id             = "obs-app"
      target_lambda_arn  = module.observability_app.lambda_function_arn
      target_lambda_name = module.observability_app.lambda_function_name
      sla_minutes        = 1440
      payload            = jsonencode({ rawPath = "/internal/narrative", ... })
    }
    "platform-watchdog" = {
      cron     = "rate(30 minutes)"
      sla_minutes = 60
      ...
    }
  }
}

3. The handler wrapper

Every scheduled Lambda imports wrapScheduledHandler from scheduled-handler.js (a ~100-line zero-npm helper kept alongside each cartridge's Lambda code today; promotes to a Lambda Layer when more than three cartridges need it). Wrap once per job:

import { wrapScheduledHandler } from './scheduled-handler.js';

const scheduledNarrative = wrapScheduledHandler('obs-app-narrative',
  async (event, context) => {
    const narrative = await generateNarrative({...});
    await longLivedWrite('narrative', narrative);
    return jsonResponse(200, { generated: true, length: narrative.text.length });
  }
);

// Then in the route dispatcher:
if (path === '/internal/narrative') return await scheduledNarrative(event, context);

What the wrapper writes to DDB (agent_session_id="job-run") every invocation:

{
  jobId:        "obs-app-narrative",
  firedAt:      "2026-06-04T13:00:24.390Z",
  finishedAt:   "2026-06-04T13:00:27.654Z",
  status:       "success",          // or "failure" with errorMsg, or "running" mid-flight
  errorMsg:     null,               // populated on failure (first 800 chars)
  durationMs:  3264,
  requestId:   "1a6a2176-...",
  expires_at:  <epoch + 30 days>
}

Re-throws on failure so the Scheduler's retry policy + DLQ + Lambda Errors metric all fire normally. The end record still lands; status is failure with the error message.

4. The watchdog

/internal/watchdog is itself a scheduled job (fires every 30 min). It reads the SSM manifest + the most recent job-run records per job, computes health, and SES-alerts when:

• No successful run within max(2 × slaMinutes, slaMinutes + 30min)
• Most recent run is failure
• DLQ depth > 0

Recipients come from the cartridge's configured recipientEmails (Cartridges tab → Budget & warnings); falls back to admin-group emails if none configured. Throttled to one alert per job per 6 hours; throttle state lives in the same DDB table under agent_session_id="alert-throttle".

The watchdog watches itself in the sense that its own run records appear in the manifest, so a downstream observer can detect "watchdog itself stopped firing." A CloudWatch alarm on the Lambda's Errors metric is the second-order backstop.

5. The operator surface (obs Schedules tab)

Admin-gated nav item next to Cartridges. Lists every job with: ID, app, cron + timezone, SLA, last fired, health (green / yellow / red / running), DLQ depth, description. Click any row → modal opens with the 7-day run history table (firedAt, finishedAt, status, duration, error) and a "▶ Run now" button that fires the target Lambda asynchronously via lambda:InvokeFunction (InvocationType=Event) and refreshes the table.

KPI strip across the top: total job count, healthy count, breached count, DLQ depth across all jobs. The breached and DLQ counters are how an operator knows at a glance whether to dig in.

6. Why the manifest goes through SSM, not a TF module output

Tried first: pass module.platform_schedules.jobs_manifest into module.observability_app as an env-var input so the obs Lambda's watchdog code knows what to watch. This is a circular module dependency:

obs_app             provides→  lambda_arn    consumed-by→  platform_schedules
platform_schedules  provides→  manifest      consumed-by→  obs_app

Terraform refuses to apply — two-cycle in the module DAG. Breaking the cycle requires one direction to be non-Terraform. The cleanest break: publish the manifest to SSM Parameter Store at a fixed path; target Lambdas read it at cold start (cached in module scope; 15-minute in-process TTL so a fresh apply lands by the next cold start).

IAM scopes for the target Lambda use the naming convention enforced by the schedules module (every resource is named ${project_name}-...), so the obs Lambda can scope sqs:GetQueueAttributes with a wildcard that matches every DLQ without taking the ARNs as TF inputs.

7. What this gives every future job for free

Cost is functionally zero at our scale (1440 watchdog Lambda invocations / month at $0.0000002 each).

8. Adding a scheduled job from a cartridge

1. Copy scheduled-handler.js into your cartridge's lambda/ directory (or import it once we promote to a Layer).
2. Wrap your handler function: const scheduledFn = wrapScheduledHandler('your-job-id', actualFn);
3. In your route dispatcher, send your scheduled path through scheduledFn(event, context).
4. Add a row in substrate/environments/ai-sandbox/main.tf's module "platform_schedules" { jobs = {...} } with the same job ID, your cron, app ID, target Lambda ARN/name, payload, and SLA.
5. Apply. The job appears in the Schedules tab within 15 min (SSM manifest read).

9. Doctrine