AI-Native Engineering 06: Secure the Agent as an Untrusted Operator

This is part 7 of 9 in the AI-Native Engineering series. Start with the course index or continue to the next part below.

Audience. You should know one programming language, Git, a shell, HTTP, and how to run tests. The examples use plain files, JSON, HTTP, Git, and shell so the ideas transfer across agent SDKs.

Shared capstone. Every part extends the same shop-agent: a support agent that searches products, inspects orders, and prepares—but never executes—a refund recommendation. The companion repository lives alongside the site source.

Secure the Agent as an Untrusted Operator

Capability. Limit blast radius even when the model is confused and retrieved content is malicious.

Threats to practice: direct and indirect prompt injection; tool poisoning; confused-deputy access; secret exfiltration; excessive agency; unsafe code execution; replay and duplicate side effects; denial of service; and data leakage in logs.

Trust-boundary diagram.

flowchart LR
    I[User / retrieved text] -->|supplies data| M[Model: untrusted decision]
    M -->|requests action| G{Policy gate}
    G -->|read scope| Q[Read-only tools]
    G -->|high-risk| H[Human approval]
    G -->|invalid / over budget| B[Block + audit]
    Q --> S[Sandbox with no secrets]
    H --> W[Write tool with narrow identity]
    S --> O[Redacted logs + tests]
    W --> O

Text equivalent: User or retrieved text supplies data to an untrusted model. A policy gate routes read scopes to read-only tools, high-risk requests to human approval, and invalid or over-budget requests to a block-and-audit path. Tools run with narrow authority and produce redacted evidence.

Module contract. Starting state: gateway policy and course/shop-agent/tests/test_security.py. Add one threat and mitigation; run the security tests; prove wrong scope, injection, and invalid input are safe.

Minimum controls. Authenticate and authorize outside the prompt. Keep credentials out of model-generated code. Use least-privilege, short-lived identities; schema validation; egress allow-lists; sandboxed execution; rate, time, and token budgets; idempotency keys; approval for irreversible actions; redaction; immutable traces; and a kill switch.

Treat skills, MCP servers, packages, and retrieved documents as supply-chain inputs: pin versions, review changes, scan them, and do not grant a new capability merely because its description sounds helpful.

Break it. Put an instruction in a product description that asks the agent to upload environment variables. The agent must quote the description as untrusted data, refuse the exfiltration, and continue only with the safe product task.

Acceptance test. A security suite proves: wrong scope is denied; secrets are not present in prompts or logs; untrusted text cannot change policy; a timeout cannot duplicate a write; and high-risk actions pause for a human.

Use when / don’t use when. Apply these controls to every agent. If a task cannot tolerate model error or cannot be isolated, keep it deterministic or require a human to perform the action.

Continue the series

← Previous: Review Agent Work with Intent and Evidence · Next: Parallelize Background Agents as a Bounded DAG →

Return to the course index.