AI-Native Engineering 05: Review Agent Work with Intent and Evidence

This is part 6 of 9 in the AI-Native Engineering series. Start with the course index or continue to the next part below.

Audience. You should know one programming language, Git, a shell, HTTP, and how to run tests. The examples use plain files, JSON, HTTP, Git, and shell so the ideas transfer across agent SDKs.

Shared capstone. Every part extends the same shop-agent: a support agent that searches products, inspects orders, and prepares—but never executes—a refund recommendation. The companion repository lives alongside the site source.

Review Agent Work with Intent and Evidence

Capability. Review a change as a complete behavior, not as pretty text or a diff alone.

An agentic review reads the issue, diff, relevant code, dependencies, tests, and runtime evidence. OpenAI describes this as a supplement to human review, not a replacement. Anthropic’s eval guidance emphasizes grading the environment outcome and the full trace, not only the final message.

sequenceDiagram
    participant I as Issue
    participant V as Reviewer
    participant E as Environment
    participant H as Human owner
    I->>V: state intent and non-goals
    V->>E: run tests and failure cases
    E-->>V: return outcome and trace
    E-->>V: return failure or regression
    V->>E: add regression case
    V->>H: report evidence and severity
    H-->>V: request fix or approve merge

Text equivalent: The reviewer reads the issue, runs the change in the environment, receives tests and traces, and sends evidence-backed findings to a human owner. The owner requests a fix or approves the merge.

Module contract. Starting state: course/shop-agent/tests/ and labs/module-5.md. Write a review report; run the full unittest command; prove every escaped bug becomes a regression case.

Review protocol.

1. Restate the intended behavior and non-goals.
2. Map changed code to the task contract.
3. Run focused tests, then the full suite.
4. Exercise failure, retry, permission, and data-boundary cases.
5. Check observability, migration, rollback, cost, and security.
6. Report only actionable findings with severity, evidence, and a smallest fix.

Use narrow specialist passes (correctness, security, tests, operations) when a diff is risky. Prefer a few high-confidence findings to a flood of guesses; track finding precision and the percentage of comments engineers accept.

Task case format.

{
  "name": "order_lookup_denies_wrong_scope",
  "input": { "user": "support", "order_id": "o-123" },
  "must": ["deny without orders:read", "write an audit event"],
  "must_not": ["return order data", "leak the token"],
  "grader": "tests/evals/order_lookup.py"
}

Score behavior. Use deterministic checks for schema, permission, call count, latency, and tests. Use human or model-assisted review for meaning and usability, calibrated against a human rubric. Version the model, prompt, tools, data, and grader.

Break it. Give the reviewer a passing-looking message with no database change, a flaky test, and a missing authorization check. It must report the environment outcome, not trust the claim.

Acceptance test. A review report contains severity, file/line or trace evidence, reproduction, and a release decision. A regression case is added for every escaped bug.

Continue the series

← Previous: Make the Codebase Agent-Ready · Next: Secure the Agent as an Untrusted Operator →

Return to the course index.