AI-Native Engineering 07: Parallelize Background Agents as a Bounded DAG

This is part 8 of 9 in the AI-Native Engineering series. Start with the course index or continue to the next part below.

Audience. You should know one programming language, Git, a shell, HTTP, and how to run tests. The examples use plain files, JSON, HTTP, Git, and shell so the ideas transfer across agent SDKs.

Shared capstone. Every part extends the same shop-agent: a support agent that searches products, inspects orders, and prepares—but never executes—a refund recommendation. The companion repository lives alongside the site source.

Parallelize Background Agents as a Bounded DAG

Capability. Split independent work across isolated agents, merge evidence, and keep cost and coordination under control.

Parallelism helps when subtasks are independent, broad, and expensive to explore. It hurts when every step depends on shared context, when agents duplicate work, or when the value cannot pay for the extra tokens. Anthropic’s multi-agent research system uses an orchestrator-worker pattern with explicit task boundaries and a final synthesis/citation pass.

Work graph.

flowchart LR
    L[Lead: plan + budget] -->|assigns| A[Agent A: code path]
    L -->|assigns| B[Agent B: tests + repro]
    L -->|assigns| C[Agent C: security threat model]
    A -->|returns evidence| S[Synthesizer: compare evidence]
    B -->|returns evidence| S
    C -->|returns evidence| S
    S -->|produces| R[One reviewable patch / handoff]
    A -.->|fails or times out| X[Retry within budget]
    X -->|returns evidence| S

Text equivalent: A lead assigns three bounded specialists. Each returns evidence to a synthesizer, which produces one reviewable patch or handoff. The specialists do not share a mutable worktree.

Module contract. Starting state: one safe worker and course/shop-agent/progress.md. Describe three independent read-heavy tasks and a synthesizer; run the full suite; prove duplicate completion is safe.

Each task packet contains objective, non-goals, allowed files/tools, source requirements, output schema, budget, and stop condition. Use worktrees or isolated sandboxes. Return compressed evidence, not a transcript.

Break it. Let three agents receive the vague instruction “investigate checkout.” Count duplicate searches, conflicting edits, tokens, and missed findings. Then rerun with three disjoint task packets and a synthesis step.

For work that outlives a request, use a background run with a durable ID and a webhook or poller. The handler should acknowledge quickly and let a worker consume completion events; it must be safe to receive a duplicate event.

Acceptance test. The orchestrator records a DAG, task IDs, budgets, retries, artifacts, and a final synthesis. No two agents edit the same worktree. A failed worker can be retried without rerunning completed work.

Use when / don’t use when. Use for independent research, test generation, security review, or separate modules. Do not parallelize tightly coupled edits or use more agents than the work can justify.

Continue the series

← Previous: Secure the Agent as an Untrusted Operator · Next: Build the Software Factory →

Return to the course index.