Native Subagent Observability

Final spec, v3, decisions locked · 2026‑08‑01. Ship Claude Code subagent/workflow and Codex collab-agent observability on the current orchestrator using only native provider emissions. Zero migrations, zero new server state. Designed to be deleted when orchestration‑v2 lands. Supersedes both prior drafts (this URL v1–v2 and the sol synthesis), the closed #4220 / #3650, and carries the open #4779 / #4662 / #4663 / #4664 stack's best ideas forward without depending on #2829.

• 1 · Outcome
• 2 · Drop points on main
• 3 · Principles
• 4 · Architecture
• 5 · Contracts and invariants
• 6 · Provider mapping
• 7 · Recovery, bounds, perf
• 8 · The UX
• 9 · Timeline behavior
• 10 · Reuse manifest
• 11 · Delivery: probe + five PRs
• 12 · Verification and acceptance
• 13 · The v2 handoff
• 14 · Risks, non-goals, deferred

1Outcome

Orchestrator v2 (#2829) is months from merge: 810 files, 18 conflicting (seven modify/delete against v1 files main keeps evolving), zero human approvals, a failing conventions gate, the correctness bot skipping ("241 blocking issues, diff too large"), and testers still filing regressions. Meanwhile every audit converged on one fact: the observability data already flows, and we drop it. Claude's task telemetry is parsed, persisted, and streamed today, then anonymized client-side; Codex's native collab-agent thread tree is deliberately suppressed. The gap is normalization and UI, not pipeline.

A user should answer five questions at a glance, without reading raw tool rows:

1. Who exists? Names, roles, models, direct spawns, workflow members, reusable Codex identities.
2. What is happening? Current tool or progress summary, workflow phase, elapsed time, recent activity.
3. What needs me? Waiting, approval, failure, interruption, and stale-transport states are unmistakable.
4. What did it cost? Provider-correct token totals and breakdowns, no pretend dollar figures.
5. Where do I inspect? Panel is the fleet view, timeline gives context, transcripts arrive on demand later.

2Drop points on main

The plumbing is ~70% built, the UI 0%. Claude's task_started/task_progress/task_notification already become persisted, streamed task.* activities with taskId, titles, lastToolName, and usage (ClaudeAdapter.ts:2678‑2736 → ProviderRuntimeIngestion.ts:489‑572). Where the rest dies, highest leverage first:

SDK options staying off: forwardSubagentText (nested live text multiplies stream volume; drill-in reads provider storage on demand) and agentProgressSummaries (recurring real model calls; opt-in later, lastToolName is the fallback activity line).

3Principles

1. Zero migrations, not zero storage. Agent lifecycle data is durably stored: widened task.* rows persist through the existing event log + activities projection, replay on restart, and stream to every client. What we refuse is new schema (any table or column takes migration slot 036, which #2829 claims, with a documented silent-skip hazard) and a second roster store v2 would have to reconcile. Additive fields inside activity payloads and runtime contracts are the sanctioned mechanism: minimal, migration-free, and living in files #2829 deletes wholesale. The activities table's existing kind column already makes task.* rows SQL-filterable; payload fields handle finer filtering client-side.
2. Provider truth, normalized once. Claude and Codex disagree about identity, completion, usage, and resumption. Preserve those semantics inside the adapters; emit one small shared lifecycle vocabulary.
3. The UI model sits above orchestration. Components consume AgentPanelModel only, derived through one source-precedence function. Legacy activities and future v2 projections each get a mapper; components never know which fed them.
4. No lying UI. Unsupported providers show no agent capability. Unsupported controls are absent, not disabled. Capability comes from data (canStop, probe-verified operations), never a hand-maintained support matrix. A child-scoped control must never fall back to interrupting the parent turn.
5. Distilled, not mirrored. Lifecycle rollups in the live stream; transcripts, scripts, and journals via on-demand RPCs from provider-owned storage. No transcript bodies in activities.
6. Never block ingestion. Observability failures log and continue; interrupts still propagate.
7. Perf rules stand. Status dots are static (no pulse, shimmer, spinner loops). Elapsed timers use the WorkingTimer DOM-write pattern, coarse cadence, frozen while disconnected. Token counts are plain counters, never bars or rings: burn has no denominator.
8. Remote is first-class. Live state crosses the existing typed activity subscription; hosted web, relay, and tunnel get the identical experience. No UI reads provider files directly.
9. Designed to be deleted. The fold is isolated legacy compatibility (client-runtime module + adapter normalizers), named as such, with a documented deletion trigger: v2 projection presence per thread, full deletion at v1 retirement.

4Architecture: fold the stream once

Claude SDK                     Codex app-server
task_* + workflow_progress        subAgentActivity + child thread stream
      │                               │
      ▼                               ▼
  ClaudeAdapter                   CodexSessionRuntime/Adapter
  carry linkage, handle           intercept child notifications (probe-verified),
  task_updated + tool_progress    synthesize task.* with timelineBypass
      │                               │
      ▼────────────────────────────▼
      self-describing task.* runtime events
      (existing family + task.updated; identity repeated on every row)
                    │
                    ▼
      EXISTING activity path: persisted, capped at 500/thread, streamed
      over thread-detail sync (web · desktop · hosted · mobile · remote)
      — no new table, row, reducer, or event-log write —
                    │
                    ▼
      packages/client-runtime: tolerant, memoized fold
      → exact v2-shaped RuntimeSubagent state
      → deriveAgentPanelModel({ nativeActivities, v2Projection })
         v2 wins when present; sources never merged
      → panel · strip · workflow card · timeline rows · mobile sheet

Why the fold won (decision record): the database already retains and streams task activities, so making rows self-describing gives reconnect, restart, and revert behavior for free; a server snapshot would re-create a small projection system (reducer, hydration, session-exit settlement, revert invalidation) exactly where we are avoiding one before v2, and every material change would append a full-roster payload to the append-only event log forever, in a codebase with replay-cost and OOM history. The snapshot's one real advantage, durability past the 500-row retention window, is mitigated by repeating identity on every row (live agents always reconstructible) and guarded by instrumentation (§7); if real threads measurably lose live rosters, the snapshot row is the documented escalation path, not the starting point.

5Contracts and invariants

Additive contract widening (packages/contracts/src/providerRuntime.ts)

type RuntimeTaskUsage = {
  totalTokens: number
  inputTokens?; cachedInputTokens?; outputTokens?; reasoningOutputTokens?
  toolUses?: number; durationMs?: number
}

// spread into task.started / task.progress / task.updated / task.completed
type TaskAgentLinkage = {
  title?; role?; model?; toolUseId?; parentAgentId?
  workflowName?; agentIndex?; phaseIndex?; phaseTitle?
  phases?: ReadonlyArray<{ index: number; title: string }>
  attempt?: number
  runHandles?: { runId?; scriptPath?; transcriptDir?; sessionUrl? }
  outputFile?: string
  agentPath?: string                 // Codex hierarchy (/root/marlow)
  timelineBypass?: boolean           // Codex-synthesized events skip timeline mapping
}

// Tool lifecycle payloads (tool.started/updated/completed/progress) gain
// optional attribution so clients can re-home subagent tool rows:
type ToolAgentAttribution = {
  agentId?: string                   // owning task/agent id when the tool ran inside a subagent
  parentToolUseId?: string           // the SDK's parent linkage, carried instead of dropped
}

// NEW event type task.updated (non-terminal patch):
{ taskId, status?, error?, endedAt?, isBackgrounded? } & TaskAgentLinkage

Identity and workflow linkage repeat on progress and terminal rows, not just start, so the fold recovers even when the start row aged out of the 500-activity window. Typed usage replaces Schema.Unknown. This is a provider-runtime contract (server-internal; activities reach clients as open-kind unknown payloads), not a new orchestration domain.

Runtime state (packages/client-runtime, derived values, not records)

type RuntimeSubagentStatus =
  | "pending" | "running" | "waiting" | "idle"
  | "completed" | "failed" | "cancelled" | "interrupted"

interface RuntimeSubagent {
  id: string                         // Claude task_id · Codex child thread id
  threadId: string
  driver: "claude" | "codex"
  parentAgentId: string | null
  kind: "subagent" | "workflow" | "workflow_agent"
  title: string
  role: { name: string; source: "provider" | "app_default" }
  model: string | null
  status: RuntimeSubagentStatus
  activationCount: number
  usage: RuntimeTaskUsage | null
  progress: string | null
  lastToolName: string | null
  approvalRequestId: string | null   // waiting-row deep-link (PR 5)
  result: string | null
  error: string | null
  canStop: boolean                   // verified provider capability (PR 5)
  workflow: { name?; phases; handles? } | null
  workflowMembership: { workflowId; agentIndex; phaseIndex: number | null; attempt } | null
  recentActivity: readonly { at: string; summary: string }[]   // ring, max 6
  startedAt: string; completedAt: string | null; updatedAt: string
}

interface RuntimeSubagentActivation {
  id; threadId; subagentId; ordinal
  status: Exclude<RuntimeSubagentStatus, "idle">   // deliberate asymmetry
  usage; startedAt; completedAt; updatedAt
}

Field names and transition semantics are #4779's, verbatim, so the v2 swap is aliases and a mapper, not remapping.

Invariants (each traces to a shipped bug or incident in the prior PRs)

• Identity is reusable; activations are one-shot. Codex resume/sendInput and Claude workflow retries reactivate one identity: increment ordinal, never create duplicates. Claude workflow members key on parentTaskId + agentIndex (stable slot), never per-attempt agent id. (#4662's surviving half; its run-ownership routing half disappears under a fold.)
• idle is real and nonterminal. A Codex child that completed a turn can receive another. Idle is counted deliberately in every count, phase predicate, and footer summary; waiting counts as active. (Three separate review findings.)
• Status conversion is total. Provider maps are exhaustive Records so a new raw state fails compilation instead of silently producing invalid state (idle → completed for settled-visual projection). The v2 lineage documents a partial mapping writing an event its own schema could not decode; the projection rebuild made the server permanently unbootable.
• Usage math is provider-specific: port SubagentObservability.ts verbatim. Codex is cumulative per thread (field-wise max-merge, idempotent under duplicate/late frames; never last, which shrinks on follow-ups). Claude is per-activation delta-accumulate. Swapping strategies double-counts or freezes the number. Partial usage never erases known fields.
• Reactivation clears result/error/completedAt; terminal timestamps are first-write.
• Order-robustness beats comparators. Completion can create an identity; a late start fills metadata but never reopens a terminal activation.
• Tolerant per-row decode: malformed rows skipped individually, never failing the fold.
• sessionUrl requires http/https at adapter and render (shipped XSS, high severity). Composite ids encoded structurally, never delimiter-joined (shipped collision).

6Provider mapping

Claude ships first PR 1

Codex is gated on a current wire probe PREP PR 3

Codex subagents are full child threads (auto-subscribed; parentThreadId, agentNickname, agentRole, source.subAgent.threadSpawn.{depth, agentPath}). The adapter synthesizes the same task.* activities with timelineBypass, so Codex requires zero client changes once Claude and the fold ship.

OpenCode participates automatically if its adapter ever emits the normalized lifecycle. ACP (Cursor/Grok): unsupported until a trustworthy native signal exists; never infer agents from tool names or transcript text. No app-owned subagents: this visualizes work the providers created.

7Recovery, bounds, and performance

Fold placement and cost

• One shared atom/selector in client-runtime, memoized by activity-list identity; never folded per-component. Decode only task-activity candidates, once per changed list.
• Adapters coalesce at the source: material transitions, terminal state, tool changes, phase discovery, meaningful usage steps (start at 5,000-token buckets, tune from traces); identical summary+tool repeats are not re-emitted. A development dispatch-rate counter ships with PR 1.
• Strings cap at 180 chars, recent-activity rings at 6 (dedupe consecutive identical summaries), derived roster at 100 identities preferring live → waiting → newest settled.
• React keys are stable agent/activation ids, never list position. Render-count test proves progress updates don't re-render the whole timeline.

Recovery semantics

• Reconnect/restart: the client reloads the persisted activities it already requests. No server hydration path exists or is needed.
• Session exit: nonterminal agents render as interrupted-by-session-end in the derived view; the presentation is not persisted as invented provider truth. Fast-follow: reconcile on resume via the SDK's typed background_tasks control request.
• Revert: state derives from retained activities, so reverting the owning turn removes its agents naturally. No invalidation code.
• Disconnect: keep last known status, add a transport-state label (live / reconnecting / stale), dim, freeze timers. Stale badge 30s past updatedAt. Transport is an axis separate from agent lifecycle.
• Retention guard (decided): instrument snapshot age and roster completeness against the 500-row window in tests and the dispatch counter. If real threads measurably lose live rosters, the escalation path is the documented server-upserted snapshot row, added then, not now.

8The UX: context plus depth

Direction locked: awareness in context, depth on demand. Three coordinated surfaces over one model. Inbox-zero rules: in-flight work is presence, not alarm; waiting is the only amber emphasis.

Composer live strip PR 2

Visible only while ≥1 agent is pending/running/waiting: total agents, waiting count, active phase, total tokens. Click opens the panel; never auto-opens.

Inline workflow run card PR 2 (decided: rich card)

One card per coordinator in the timeline, replacing its generic task and duplicate tool rows: phase counters, up to 8 member rows ordered by urgency (running + failed first, then most recent), overflow as "+N more → open Agents". Remote runs swap the body for a sanitized external session link. The card is a derived, capped view of the fold; the panel is the complete one. Perf guardrails: static visuals, dedicated roster context isolated from the virtualized list, and the render-count test as the acceptance gate.

Agents panel PR 2

First-class right-panel surface beside Plan/Diff/Files/Preview/Terminal. Tab auto-appears once the thread has agents (count badge when hidden), always reachable from the "+" menu with an explanatory empty state, plus a command-palette action ("Agents: Open", no default keybinding). Workflow members group by phase, then direct spawns. Rows: name, role, model, status, progress fallback (progress → lastToolName; settled rows lead error → result), elapsed, tokens, activation count. Expanded rows: the six-entry activity ring. Footer: running/waiting/settled counts + Σ tokens.

Mobile PR 4

Read-only first: a static active-agent count in the thread header opens a native sheet (phone) or inspector-pane tenant (tablet) over the exact shared derivation. Same grouping, status, result, usage vocabulary. Waiting and terminal signals survive any space-constrained condensation. No script/transcript viewer, no desktop metaphors, no filesystem affordances. Stop lands later with shared confirmation UX.

Status language

Elapsed measures the current activation, not first spawn, so a resumed agent's timer excludes idle gaps; frozen at completedAt and while disconnected.

9Timeline and work-log behavior: the quiet-timeline guarantee

Invariant: the main timeline carries the parent's narrative plus at most one row per agent or workflow run. Everything an agent does internally lives in the panel and card, not the chat. Today this is violated twice: Claude forwards subagent tool_use/tool_result blocks into the main stream (even with forwardSubagentText off) and the adapter ignores parent_tool_use_id when emitting item.* events, so subagent tool calls render as top-level rows; and task.progress ticks land as individual work-log entries. With many concurrent agents the chat becomes incoherent. The fix has three parts:

1. Attribute: the adapter stamps ToolAgentAttribution (§5) on every tool lifecycle event whose parent_tool_use_id resolves to a tracked agent, instead of dropping the linkage. Persisted in the payload, so it also filters correctly for filtering-aware clients replaying history, and gives v2 a clean signal to migrate.
2. Re-home: clients fold agent-attributed tool rows out of the main work log entirely; they surface as the owning agent's progress/lastToolName and recent-activity entries. Old clients keep seeing today's flat rows (no worse than the status quo).
3. Collapse: task.* progress ticks never render individually; they group into the one lifecycle row (direct agents) or the run card (workflows), exactly as below.

Routine result snippets stay out of the timeline; failure text renders inline because it changes the next user action. Full nested rendering stays out of scope: MessagesTimelineRow is a flat union, and #2829's own V2LifecycleRow is the pattern to converge on at merge time.

10Reuse manifest

11Delivery: probe + five PRs, first two are the release

12Verification and acceptance

Fold and contract tests (one per known failure mode)

• Progress creates an agent when its start row aged out · completion before start stays terminal · duplicate terminal events are idempotent · reactivation increments ordinal and clears result/error.
• Idle counted intentionally everywhere · waiting counts as active · phase labels say running only when a member runs · settled counts include success and error.
• Codex cumulative usage never shrinks or double-counts · Claude deltas accumulate once · partial usage preserves known breakdowns.
• Malformed rows skipped independently · strings cap at 180, rings at 6 · composite ids cannot collide · old activity payloads still decode · v2 projection fully replaces fold output (never merges).
• Widened fields survive ActivityPayloadProjection to the client (regression test).
• Quiet timeline: a fixture with N concurrent subagents produces exactly N lifecycle rows (+1 card per workflow) in the derived work log, zero agent-attributed tool rows; an unattributed tool row (no linkage resolvable) stays in the main log rather than disappearing.

Provider fixtures

Claude: direct lifecycle · local workflow with phases and retries · tool progress + output handle. Codex: child event before registration · follow-up on an idle child · approval and system error · v1 and current parent-transcript routing. Both: session exit and reconnect.

UI acceptance

• No continuously repainting animation anywhere; the panel handles 100 identities over a 500-row input without jank; progress updates don't re-render the timeline (render-count test).
• Expansion state follows stable ids; condensed mobile rows preserve waiting/failure/terminal; remote clients need no local filesystem and receive no transcript body in the live stream.

Integrated proof before each UI PR merges

• A real Claude subagent appears, progresses, settles, survives reload, retains its result.
• A real Claude workflow shows queued and running members by phase with no duplicate tool rows.
• A real Codex child shows nickname, activity, cumulative usage, idle state, and a second activation on reuse; nothing leaks into the parent chat.
• Hosted/remote web sees identical state through existing sync.

13The v2 handoff

1. When the v2 projection exists for a thread, deriveAgentPanelModel selects it and stops folding native activities for that thread. One precedence function; sources never merged (the duplicate-agents failure mode).
2. Runtime types swap to v2 aliases: field names, statuses, activation model, and usage semantics already match.
3. The panel, strip, card, mobile sheet, status language, and tests remain untouched: the product layer v2 would otherwise have to invent at merge time, already user-validated. #4663 becomes a diff against a shipped UI.
4. The fold and the widened v1 contracts are isolated legacy compatibility, deleted with the v1 adapters (#2829 removes those files wholesale). Provider normalizers stay adapter-local so v2 can cherry-pick the mapping logic. Inspection RPCs and verified controls remain useful regardless of orchestration source.
5. No migration slots consumed, no tables, no projection schema, no closed unions touched: nothing in #2829's DB surface (036–045) conflicts.

14Risks, non-goals, deferred

Non-goals: no new orchestration engine, graph, queue, or ownership model; no v1 subagent/activation tables; no ACP inference; no app-owned subagents; no server-side thread hiding; no cost-in-USD or fleet analytics; no history backfill (dropped fields exist only in rotating NDJSON logs).

Deferred: nested live text forwarding (forwardSubagentText); AI progress summaries by default (opt-in later, they cost model calls); mission-control tree + dossier zoom (post-PR 5, layout only, every field already present); #4664-style hiding with unreachable transcripts (revisit only after drill-in ships everywhere).