Course: Course 2B — Securing & Attacking Harnesses and LLMs
Module: B1 — Threat Model of Agentic Systems
Duration: 60–75 minutes
Environment: No special tooling. A text editor, a JSON linter, and Python 3.10+ (for the classify_untrusted_content() function). No GPU, no model API calls. This lab builds the surface map — the JSON threat model and the content classifier — that every subsequent module (B2–B12) references for scope and testing.
By the end of this lab you will have:
classify_untrusted_content() function that the harness uses to tag content crossing any trust boundary into the context window — the engineering realization of "every retrieved byte is untrusted until tagged."This lab is deliberately low-execution. The point is to build the map before any technique from B2 onward is pointed at a target. A red-team without this map tests one surface and misses six; a defender without it hardens one and leaves the rest open.
mkdir b1-threat-model-lab && cd b1-threat-model-lab
# No dependencies for the core lab. Python 3.10+ for the classifier function.
python3 --version # confirm 3.10+
No venv, no GPU, no model API. This is a modeling and code lab.
You will model the following sample agent — a realistic customer-support agent deployed by a fictional company, "Acme Support." Read it carefully; every component maps to a surface.
ACME SUPPORT AGENT — architecture summary
- Reasoning loop: ReAct-style (observe → reason → act → observe), up to 10 iterations.
- Tools:
(a) fetch_kb_article(id) — reads from Acme's knowledge base (internal API)
(b) fetch_url(url) — fetches a public web page (external, untrusted)
(c) query_customer_db(sql) — read-only SQL against the customer DB (PII)
(d) send_email(to, subject, body) — sends email from support@acme.example
(e) execute_python(code) — runs Python in a Docker container for data analysis
- Memory: a vector store of past ticket summaries, persisted across sessions, written by the agent.
- Provider: OpenAI gpt-4o-2026-05-01 (shared API, logs inputs, opt-out of training is set).
- Identity: a single service account "acme-agent-svc" with broad read access to the customer DB
and permission to send email from support@acme.example (no rate limit, no recipient allowlist).
- Sandbox: execute_python runs in a Docker container with no syscall allowlist, no network egress
filter, and a 60-second timeout. CPU/memory unbounded.
- Inter-agent: delegates to one peer agent "acme-billing-agent" via plain JSON messages over an
internal queue. No message authentication. The billing agent holds write access to the
billing DB.
Before building the JSON, on paper or in a scratch file, identify each component's surface. There are components that map to all seven surfaces. Confirm you can name all seven before proceeding to Phase 2.
Build agent_surface_map.json — the machine-checkable threat model. This is the artifact every subsequent module references. The schema requires all seven surfaces, each with its trust boundary, entry points, canonical attack, ASI risk, blast radius, and destination module.
{
"agent": {
"name": "Acme Support Agent",
"version": "1.0.0",
"model": "gpt-4o-2026-05-01",
"loop_type": "ReAct",
"max_iterations": 10
},
"surfaces": [
{
"id": "loop",
"name": "The loop (reasoning cycle)",
"trust_boundary": "between consuming untrusted context and selecting an action, every iteration",
"untrusted_entry_points": ["user input (direct injection)", "retrieved context (via memory/tool outputs re-entering the window)"],
"canonical_attack": "goal hijacking (ASI01) — injected content rewrites the objective mid-loop; multi-step variant spans tool calls",
"asi_risks": ["ASI01"],
"blast_radius": "session_wide",
"blast_radius_score": 2,
"defense_module": "B2",
"attacks_architecture": "CrabTrap DD-19",
"defense_preview": "instruction isolation, untrusted tagging, session-level intent tracking"
},
{
"id": "tools",
"name": "Tools (contracts and tool output)",
"trust_boundary": "(1) tool contract: does the tool do only what its schema claims? (2) tool output: return values flow into the context window",
"untrusted_entry_points": ["fetch_url return value (external web page)", "fetch_kb_article return (if KB is externally editable)", "query_customer_db result rows"],
"canonical_attack": "indirect injection via tool output (ASI01) + tool/skill abuse (ASI05) — e.g. read_file reads /etc/passwd, send_email exfiltrates",
"asi_risks": ["ASI05", "ASI01"],
"blast_radius": "tool_scope_wide",
"blast_radius_score": 4,
"defense_module": "B4",
"attacks_architecture": "CrabTrap egress filter DD-19",
"defense_preview": "least-privilege schemas, output sanitization + untrusted tagging, path/host allowlists, MCP identity verification"
},
{
"id": "memory",
"name": "Memory (long-term and context-window)",
"trust_boundary": "between the user's session and the persistent store; is retrieved content trusted as fact?",
"untrusted_entry_points": ["retrieved ticket summaries (possibly poisoned in prior sessions)", "agent-written memory from untrusted user input"],
"canonical_attack": "sleeper attack / memory poisoning (ASI04) — poison in session 1, activates in session 2; context contamination accumulates across turns",
"asi_risks": ["ASI04", "ASI06"],
"blast_radius": "cross_session_cross_user",
"blast_radius_score": 5,
"defense_module": "B3",
"attacks_architecture": "harness-managed writes (Course 1 Module 4.3)",
"defense_preview": "harness-managed writes, signed entries w/ provenance, retrieval-time tagging, TTL decay"
},
{
"id": "provider",
"name": "The provider (model API boundary)",
"trust_boundary": "between the deployer's harness and the provider's model — the model is a third party: opaque, shared, updated without deployer control",
"untrusted_entry_points": ["outbound context sent to the provider (may contain PII/credentials)", "provider logs / training pipeline"],
"canonical_attack": "prompt leakage (ASI02) — model emits system prompt (full tool map); provider logs hold customer context data",
"asi_risks": ["ASI02"],
"blast_radius": "cross_tenant_if_shared",
"blast_radius_score": 4,
"defense_module": "B2 + B11",
"attacks_architecture": "provider boundary",
"defense_preview": "treat as untrusted boundary, DLP on outbound context, pin model version, self-host for high-sensitivity"
},
{
"id": "identity",
"name": "Identity (non-human credentials)",
"trust_boundary": "between what the task needs and what the credential permits — the gap is excessive agency",
"untrusted_entry_points": ["credential exposure via tool-output disclosure or memory leak", "abuse by a hijacked agent"],
"canonical_attack": "excessive agency / credential abuse (ASI03/ASI10) — compromised agent uses over-scoped DB read + unscoped email to exfiltrate",
"asi_risks": ["ASI03", "ASI10"],
"blast_radius": "credential_scope_wide",
"blast_radius_score": 6,
"defense_module": "B5",
"attacks_architecture": "IronCurtain DD-20 credential quarantine",
"defense_preview": "least privilege, ephemeral/task-scoped creds, per-tool isolation, JIT elevation with human approval"
},
{
"id": "sandbox",
"name": "Sandbox (execution isolation)",
"trust_boundary": "the isolation primitive itself (Docker container) — if it holds, code is contained; if it fails, escape to host",
"untrusted_entry_points": ["execute_python code (agent-authored, possibly injection-driven)", "no syscall allowlist / no egress filter"],
"canonical_attack": "sandbox escape (ASI07) — container breakout reaches host, reads creds/memory/prompt, pivots; resource exhaustion (ASI09) — unbounded CPU/mem",
"asi_risks": ["ASI07", "ASI09"],
"blast_radius": "host_wide",
"blast_radius_score": 7,
"defense_module": "B7",
"attacks_architecture": "NemoClaw OpenShell DD-09 + IronCurtain V8 isolate DD-20",
"defense_preview": "hardened isolation, syscall allowlist (seccomp), CPU/mem/time/budget limits, egress filtering, kill switch"
},
{
"id": "inter_agent_edges",
"name": "Inter-agent edges (mesh communication, cascade)",
"trust_boundary": "between agent A's trust domain and agent B's; messages are unauthenticated plain JSON",
"untrusted_entry_points": ["plain-text messages from acme-billing-agent (forgeable)", "billing agent outputs consumed as fact (cascade)"],
"canonical_attack": "inter-agent trust escalation (forge supervisor/peer message) + cascading hallucination (ASI06) — one poisoned result propagates; blast radius mesh-wide",
"asi_risks": ["ASI06", "ASI08"],
"blast_radius": "mesh_wide",
"blast_radius_score": 7,
"defense_module": "B6",
"attacks_architecture": "ZeroClaw DD-16 HMAC receipts (open gap: ephemeral keys)",
"defense_preview": "HMAC-signed messages per edge, replay protection (nonces), per-agent identity isolation, output verification at every edge, cascade-depth limits"
}
],
"remediation_priority": [
{ "rank": 1, "surface": "sandbox", "reason": "host-wide blast radius; containment failure" },
{ "rank": 2, "surface": "identity", "reason": "credential-scope-wide; broad DB read + unscoped email" },
{ "rank": 3, "surface": "inter_agent_edges", "reason": "mesh-wide; billing agent has write access" }
]
}
untrusted_entry_points, canonical_attack, and blast_radius for each surface based on the specific Acme Support Agent architecture from Phase 1 (not just the generic surface).blast_radius_score is 1–7 (1 = session-wide/lowest, 7 = mesh-wide/highest). Score each surface's worst case for THIS agent. The sample scores are a starting point — adjust if the Acme architecture changes the worst case.remediation_priority list: rank the surfaces to fix first, following the blast-radius principle (containment-failure surfaces before single-session surfaces). Justify each rank in the reason field.python3 -m json.tool agent_surface_map.json.For the Acme agent specifically, answer in prioritization_justification.md:
execute_python sandbox has no syscall allowlist and no egress filter. What is the specific blast radius if an injection causes the agent to run subprocess.run("curl ...") in the container? Does the containment hold?acme-agent-svc identity has broad DB read access AND unscoped email sending. If the agent is goal-hijacked (surface 1), which surfaces does the excessive agency (surface 5) amplify? Trace the path.acme-billing-agent holds write access to the billing DB and communicates via plain JSON. If the support agent is compromised, can it forge a message to the billing agent that causes a write? What is the blast radius?Record your reasoning. These three judgments are the difference between a copy-pasted map and a real threat model.
classify_untrusted_content() function (20 min)Implement the function the harness calls on every piece of content about to enter the context window. This is the engineering realization of "every retrieved byte is untrusted until tagged." A harness that does not classify inbound content is the ~50% InjecAgent vulnerability rate.
from typing import Literal
TrustLevel = Literal["trusted", "untrusted", "semi_trusted"]
ContentOrigin = Literal[
"user_direct", # direct user input (the prompt)
"tool_output_external", # fetch_url, external API — fully untrusted
"tool_output_internal", # fetch_kb_article, internal API — semi-trusted
"tool_output_data", # query_customer_db result rows — semi-trusted (may contain PII)
"memory_retrieved", # vector store retrieval — untrusted (may be poisoned)
"agent_generated", # the agent's own prior output — semi-trusted
"peer_agent_message", # inter-agent message — untrusted (forgeable)
"system_prompt", # the privileged instruction layer — trusted
]
def classify_untrusted_content(
content: str,
origin: ContentOrigin,
surface: str,
) -> dict:
"""Classify content about to enter the agent's context window.
Returns a tag the harness inserts alongside the content so the model
can distinguish instruction from data. The default posture: every
origin except system_prompt is untrusted or semi-trusted.
Args:
content: the text about to enter the context window.
origin: where it came from (see ContentOrigin).
surface: which of the seven surfaces it entered through.
Returns:
{
"trust_level": TrustLevel,
"tag": str, # the marker the harness prepends, e.g. "[UNTRUSTED TOOL OUTPUT]"
"surface": str,
"origin": str,
"requires_isolation": bool, # should this be wrapped in a data fence?
"rationale": str,
}
"""
pass
Fill in the body. The classification rules:
system_prompt → trusted, no tag, no isolation. This is the privileged instruction layer.user_direct → untrusted (direct injection vector), tag [UNTRUSTED USER INPUT], isolation recommended. Direct injection is the crudest but real vector.tool_output_external → untrusted (the indirect-injection vector — InjecAgent's ~50%), tag [UNTRUSTED EXTERNAL CONTENT], isolation required.tool_output_internal → semi_trusted (internal API, but if the KB is externally editable it degrades to untrusted), tag [SEMI-TRUSTED INTERNAL], isolation recommended.tool_output_data → semi_trusted + PII flag (DB result rows may contain customer PII — this is also a data-handling concern, surface 4/provider-adjacent), tag [SEMI-TRUSTED DATA — MAY CONTAIN PII], isolation required, DLP check required.memory_retrieved → untrusted (may be poisoned — the sleeper attack), tag [UNTRUSTED RETRIEVED MEMORY], isolation required.agent_generated → semi_trusted (the agent's own prior output, but if the agent was hijacked in a prior turn this is contaminated), tag [SEMI-TRUSTED PRIOR OUTPUT], isolation recommended.peer_agent_message → untrusted (forgeable — surface 7), tag [UNTRUSTED PEER MESSAGE], isolation required, signature verification required.Write at least six test calls covering each origin type. For each, print the classification. Verify:
# 1. system_prompt → trusted, no tag
# 2. tool_output_external (a fetched web page with "ignore instructions...") → untrusted, tagged, isolation required
# 3. memory_retrieved (a ticket summary from a prior session) → untrusted, tagged, isolation required
# 4. tool_output_data (customer DB rows) → semi_trusted, PII flag, DLP check required
# 5. peer_agent_message (a message from acme-billing-agent) → untrusted, signature verification required
# 6. user_direct ("summarize my last ticket") → untrusted, tagged
All six must produce the correct trust level, tag, and isolation/verification flags.
This function is the single most important harness control for surfaces 1, 2, 3, and 7. If every piece of content entering the context window is tagged with its trust level, the model has a fighting chance of distinguishing instruction from data — and the harness has a hook to refuse to act on untrusted content that requests high-impact actions. A harness without this function is the ~50% InjecAgent vulnerability rate. B2 implements the full instruction-isolation layer that consumes these tags; B3 implements the memory-specific tagging; B6 implements the peer-message signature verification. This lab builds the shared primitive.
If time remains, write prioritization_report.md that:
execute_python can reach the network and the host filesystem").The point: prioritization is not abstract. It is "which component, which control, which reference architecture, which gap." A report that names all four is a report a CISO can act on.
agent_surface_map.json — the JSON threat model, seven surfaces, validated (Phase 2)prioritization_justification.md — the three key judgments about the Acme agent (Phase 2.3)classify_content.py — the classify_untrusted_content() function + six test calls (Phase 3)prioritization_report.md — the blast-radius prioritization report (Phase 4)agent_surface_map.json validates, has all seven surfaces, each with trust boundary, entry points, canonical attack, ASI risk, blast radius, and destination module.untrusted_entry_points for each surface are specific to the Acme Support Agent (not generic).remediation_priority ranks containment-failure surfaces (sandbox, identity, inter-agent edges) before single-session surfaces (the loop), with reasons.prioritization_justification.md trace specific attack paths through the Acme architecture (sandbox egress, identity amplification, forged peer message).classify_untrusted_content() returns the correct trust level, tag, and isolation/verification flag for all six test origins.system_prompt is the only trusted origin; every other origin is untrusted or semi_trusted.# Lab Specification — Module B1: Threat Model of Agentic Systems
**Course**: Course 2B — Securing & Attacking Harnesses and LLMs
**Module**: B1 — Threat Model of Agentic Systems
**Duration**: 60–75 minutes
**Environment**: No special tooling. A text editor, a JSON linter, and Python 3.10+ (for the `classify_untrusted_content()` function). No GPU, no model API calls. This lab builds the *surface map* — the JSON threat model and the content classifier — that every subsequent module (B2–B12) references for scope and testing.
---
## Learning objectives
By the end of this lab you will have:
1. **Built a JSON threat model of a sample agent** that enumerates all seven attack surfaces (loop, tools, memory, provider, identity, sandbox, inter-agent edges), the trust boundary for each, the untrusted-content entry points, the canonical attack, the OWASP ASI risk, and a blast-radius score.
2. **Mapped each surface to its destination defense module** (B2–B12) and the Course 1 defense architecture (DD-09, DD-16, DD-19, DD-20) it attacks — turning the seven-surface map into a remediation roadmap.
3. **Implemented a `classify_untrusted_content()` function** that the harness uses to tag content crossing any trust boundary into the context window — the engineering realization of "every retrieved byte is untrusted until tagged."
4. **Scored and prioritized remediation** by blast radius, applying the principle that containment-failure surfaces (sandbox, identity, inter-agent edges) take priority over single-session surfaces (the loop).
This lab is deliberately low-execution. The point is to build the *map* before any technique from B2 onward is pointed at a target. A red-team without this map tests one surface and misses six; a defender without it hardens one and leaves the rest open.
---
## Phase 0 — Setup (2 min)
```bash
mkdir b1-threat-model-lab && cd b1-threat-model-lab
# No dependencies for the core lab. Python 3.10+ for the classifier function.
python3 --version # confirm 3.10+
```
No venv, no GPU, no model API. This is a modeling and code lab.
---
## Phase 1 — The sample agent (5 min)
You will model the following sample agent — a realistic customer-support agent deployed by a fictional company, "Acme Support." Read it carefully; every component maps to a surface.
### 1.1 The sample agent
```text
ACME SUPPORT AGENT — architecture summary
- Reasoning loop: ReAct-style (observe → reason → act → observe), up to 10 iterations.
- Tools:
(a) fetch_kb_article(id) — reads from Acme's knowledge base (internal API)
(b) fetch_url(url) — fetches a public web page (external, untrusted)
(c) query_customer_db(sql) — read-only SQL against the customer DB (PII)
(d) send_email(to, subject, body) — sends email from support@acme.example
(e) execute_python(code) — runs Python in a Docker container for data analysis
- Memory: a vector store of past ticket summaries, persisted across sessions, written by the agent.
- Provider: OpenAI gpt-4o-2026-05-01 (shared API, logs inputs, opt-out of training is set).
- Identity: a single service account "acme-agent-svc" with broad read access to the customer DB
and permission to send email from support@acme.example (no rate limit, no recipient allowlist).
- Sandbox: execute_python runs in a Docker container with no syscall allowlist, no network egress
filter, and a 60-second timeout. CPU/memory unbounded.
- Inter-agent: delegates to one peer agent "acme-billing-agent" via plain JSON messages over an
internal queue. No message authentication. The billing agent holds write access to the
billing DB.
```
### 1.2 Your first task
Before building the JSON, on paper or in a scratch file, identify each component's surface. There are components that map to all seven surfaces. Confirm you can name all seven before proceeding to Phase 2.
---
## Phase 2 — The JSON threat model (25 min)
Build `agent_surface_map.json` — the machine-checkable threat model. This is the artifact every subsequent module references. The schema requires all seven surfaces, each with its trust boundary, entry points, canonical attack, ASI risk, blast radius, and destination module.
### 2.1 The schema
```json
{
"agent": {
"name": "Acme Support Agent",
"version": "1.0.0",
"model": "gpt-4o-2026-05-01",
"loop_type": "ReAct",
"max_iterations": 10
},
"surfaces": [
{
"id": "loop",
"name": "The loop (reasoning cycle)",
"trust_boundary": "between consuming untrusted context and selecting an action, every iteration",
"untrusted_entry_points": ["user input (direct injection)", "retrieved context (via memory/tool outputs re-entering the window)"],
"canonical_attack": "goal hijacking (ASI01) — injected content rewrites the objective mid-loop; multi-step variant spans tool calls",
"asi_risks": ["ASI01"],
"blast_radius": "session_wide",
"blast_radius_score": 2,
"defense_module": "B2",
"attacks_architecture": "CrabTrap DD-19",
"defense_preview": "instruction isolation, untrusted tagging, session-level intent tracking"
},
{
"id": "tools",
"name": "Tools (contracts and tool output)",
"trust_boundary": "(1) tool contract: does the tool do only what its schema claims? (2) tool output: return values flow into the context window",
"untrusted_entry_points": ["fetch_url return value (external web page)", "fetch_kb_article return (if KB is externally editable)", "query_customer_db result rows"],
"canonical_attack": "indirect injection via tool output (ASI01) + tool/skill abuse (ASI05) — e.g. read_file reads /etc/passwd, send_email exfiltrates",
"asi_risks": ["ASI05", "ASI01"],
"blast_radius": "tool_scope_wide",
"blast_radius_score": 4,
"defense_module": "B4",
"attacks_architecture": "CrabTrap egress filter DD-19",
"defense_preview": "least-privilege schemas, output sanitization + untrusted tagging, path/host allowlists, MCP identity verification"
},
{
"id": "memory",
"name": "Memory (long-term and context-window)",
"trust_boundary": "between the user's session and the persistent store; is retrieved content trusted as fact?",
"untrusted_entry_points": ["retrieved ticket summaries (possibly poisoned in prior sessions)", "agent-written memory from untrusted user input"],
"canonical_attack": "sleeper attack / memory poisoning (ASI04) — poison in session 1, activates in session 2; context contamination accumulates across turns",
"asi_risks": ["ASI04", "ASI06"],
"blast_radius": "cross_session_cross_user",
"blast_radius_score": 5,
"defense_module": "B3",
"attacks_architecture": "harness-managed writes (Course 1 Module 4.3)",
"defense_preview": "harness-managed writes, signed entries w/ provenance, retrieval-time tagging, TTL decay"
},
{
"id": "provider",
"name": "The provider (model API boundary)",
"trust_boundary": "between the deployer's harness and the provider's model — the model is a third party: opaque, shared, updated without deployer control",
"untrusted_entry_points": ["outbound context sent to the provider (may contain PII/credentials)", "provider logs / training pipeline"],
"canonical_attack": "prompt leakage (ASI02) — model emits system prompt (full tool map); provider logs hold customer context data",
"asi_risks": ["ASI02"],
"blast_radius": "cross_tenant_if_shared",
"blast_radius_score": 4,
"defense_module": "B2 + B11",
"attacks_architecture": "provider boundary",
"defense_preview": "treat as untrusted boundary, DLP on outbound context, pin model version, self-host for high-sensitivity"
},
{
"id": "identity",
"name": "Identity (non-human credentials)",
"trust_boundary": "between what the task needs and what the credential permits — the gap is excessive agency",
"untrusted_entry_points": ["credential exposure via tool-output disclosure or memory leak", "abuse by a hijacked agent"],
"canonical_attack": "excessive agency / credential abuse (ASI03/ASI10) — compromised agent uses over-scoped DB read + unscoped email to exfiltrate",
"asi_risks": ["ASI03", "ASI10"],
"blast_radius": "credential_scope_wide",
"blast_radius_score": 6,
"defense_module": "B5",
"attacks_architecture": "IronCurtain DD-20 credential quarantine",
"defense_preview": "least privilege, ephemeral/task-scoped creds, per-tool isolation, JIT elevation with human approval"
},
{
"id": "sandbox",
"name": "Sandbox (execution isolation)",
"trust_boundary": "the isolation primitive itself (Docker container) — if it holds, code is contained; if it fails, escape to host",
"untrusted_entry_points": ["execute_python code (agent-authored, possibly injection-driven)", "no syscall allowlist / no egress filter"],
"canonical_attack": "sandbox escape (ASI07) — container breakout reaches host, reads creds/memory/prompt, pivots; resource exhaustion (ASI09) — unbounded CPU/mem",
"asi_risks": ["ASI07", "ASI09"],
"blast_radius": "host_wide",
"blast_radius_score": 7,
"defense_module": "B7",
"attacks_architecture": "NemoClaw OpenShell DD-09 + IronCurtain V8 isolate DD-20",
"defense_preview": "hardened isolation, syscall allowlist (seccomp), CPU/mem/time/budget limits, egress filtering, kill switch"
},
{
"id": "inter_agent_edges",
"name": "Inter-agent edges (mesh communication, cascade)",
"trust_boundary": "between agent A's trust domain and agent B's; messages are unauthenticated plain JSON",
"untrusted_entry_points": ["plain-text messages from acme-billing-agent (forgeable)", "billing agent outputs consumed as fact (cascade)"],
"canonical_attack": "inter-agent trust escalation (forge supervisor/peer message) + cascading hallucination (ASI06) — one poisoned result propagates; blast radius mesh-wide",
"asi_risks": ["ASI06", "ASI08"],
"blast_radius": "mesh_wide",
"blast_radius_score": 7,
"defense_module": "B6",
"attacks_architecture": "ZeroClaw DD-16 HMAC receipts (open gap: ephemeral keys)",
"defense_preview": "HMAC-signed messages per edge, replay protection (nonces), per-agent identity isolation, output verification at every edge, cascade-depth limits"
}
],
"remediation_priority": [
{ "rank": 1, "surface": "sandbox", "reason": "host-wide blast radius; containment failure" },
{ "rank": 2, "surface": "identity", "reason": "credential-scope-wide; broad DB read + unscoped email" },
{ "rank": 3, "surface": "inter_agent_edges", "reason": "mesh-wide; billing agent has write access" }
]
}
```
### 2.2 Your task
- Use the schema above as the starting point. Refine the `untrusted_entry_points`, `canonical_attack`, and `blast_radius` for each surface based on the specific Acme Support Agent architecture from Phase 1 (not just the generic surface).
- The `blast_radius_score` is 1–7 (1 = session-wide/lowest, 7 = mesh-wide/highest). Score each surface's worst case for THIS agent. The sample scores are a starting point — adjust if the Acme architecture changes the worst case.
- Build the `remediation_priority` list: rank the surfaces to fix first, following the blast-radius principle (containment-failure surfaces before single-session surfaces). Justify each rank in the `reason` field.
- Validate the JSON: `python3 -m json.tool agent_surface_map.json`.
### 2.3 The key judgment
For the Acme agent specifically, answer in `prioritization_justification.md`:
- The `execute_python` sandbox has no syscall allowlist and no egress filter. What is the specific blast radius if an injection causes the agent to run `subprocess.run("curl ...")` in the container? Does the containment hold?
- The `acme-agent-svc` identity has broad DB read access AND unscoped email sending. If the agent is goal-hijacked (surface 1), which surfaces does the excessive agency (surface 5) amplify? Trace the path.
- The `acme-billing-agent` holds write access to the billing DB and communicates via plain JSON. If the support agent is compromised, can it forge a message to the billing agent that causes a write? What is the blast radius?
Record your reasoning. These three judgments are the difference between a copy-pasted map and a real threat model.
---
## Phase 3 — The `classify_untrusted_content()` function (20 min)
Implement the function the harness calls on every piece of content about to enter the context window. This is the engineering realization of "every retrieved byte is untrusted until tagged." A harness that does not classify inbound content is the ~50% InjecAgent vulnerability rate.
### 3.1 The spec
```python
from typing import Literal
TrustLevel = Literal["trusted", "untrusted", "semi_trusted"]
ContentOrigin = Literal[
"user_direct", # direct user input (the prompt)
"tool_output_external", # fetch_url, external API — fully untrusted
"tool_output_internal", # fetch_kb_article, internal API — semi-trusted
"tool_output_data", # query_customer_db result rows — semi-trusted (may contain PII)
"memory_retrieved", # vector store retrieval — untrusted (may be poisoned)
"agent_generated", # the agent's own prior output — semi-trusted
"peer_agent_message", # inter-agent message — untrusted (forgeable)
"system_prompt", # the privileged instruction layer — trusted
]
def classify_untrusted_content(
content: str,
origin: ContentOrigin,
surface: str,
) -> dict:
"""Classify content about to enter the agent's context window.
Returns a tag the harness inserts alongside the content so the model
can distinguish instruction from data. The default posture: every
origin except system_prompt is untrusted or semi-trusted.
Args:
content: the text about to enter the context window.
origin: where it came from (see ContentOrigin).
surface: which of the seven surfaces it entered through.
Returns:
{
"trust_level": TrustLevel,
"tag": str, # the marker the harness prepends, e.g. "[UNTRUSTED TOOL OUTPUT]"
"surface": str,
"origin": str,
"requires_isolation": bool, # should this be wrapped in a data fence?
"rationale": str,
}
"""
pass
```
### 3.2 Implement it
Fill in the body. The classification rules:
- `system_prompt` → `trusted`, no tag, no isolation. This is the privileged instruction layer.
- `user_direct` → `untrusted` (direct injection vector), tag `[UNTRUSTED USER INPUT]`, isolation recommended. Direct injection is the crudest but real vector.
- `tool_output_external` → `untrusted` (the indirect-injection vector — InjecAgent's ~50%), tag `[UNTRUSTED EXTERNAL CONTENT]`, isolation required.
- `tool_output_internal` → `semi_trusted` (internal API, but if the KB is externally editable it degrades to untrusted), tag `[SEMI-TRUSTED INTERNAL]`, isolation recommended.
- `tool_output_data` → `semi_trusted` + PII flag (DB result rows may contain customer PII — this is also a data-handling concern, surface 4/provider-adjacent), tag `[SEMI-TRUSTED DATA — MAY CONTAIN PII]`, isolation required, DLP check required.
- `memory_retrieved` → `untrusted` (may be poisoned — the sleeper attack), tag `[UNTRUSTED RETRIEVED MEMORY]`, isolation required.
- `agent_generated` → `semi_trusted` (the agent's own prior output, but if the agent was hijacked in a prior turn this is contaminated), tag `[SEMI-TRUSTED PRIOR OUTPUT]`, isolation recommended.
- `peer_agent_message` → `untrusted` (forgeable — surface 7), tag `[UNTRUSTED PEER MESSAGE]`, isolation required, signature verification required.
### 3.3 Test cases
Write at least six test calls covering each origin type. For each, print the classification. Verify:
```python
# 1. system_prompt → trusted, no tag
# 2. tool_output_external (a fetched web page with "ignore instructions...") → untrusted, tagged, isolation required
# 3. memory_retrieved (a ticket summary from a prior session) → untrusted, tagged, isolation required
# 4. tool_output_data (customer DB rows) → semi_trusted, PII flag, DLP check required
# 5. peer_agent_message (a message from acme-billing-agent) → untrusted, signature verification required
# 6. user_direct ("summarize my last ticket") → untrusted, tagged
```
All six must produce the correct trust level, tag, and isolation/verification flags.
### 3.4 The point
This function is the single most important harness control for surfaces 1, 2, 3, and 7. If every piece of content entering the context window is tagged with its trust level, the model has a fighting chance of distinguishing instruction from data — and the harness has a hook to refuse to act on untrusted content that requests high-impact actions. A harness without this function is the ~50% InjecAgent vulnerability rate. B2 implements the full instruction-isolation layer that consumes these tags; B3 implements the memory-specific tagging; B6 implements the peer-message signature verification. This lab builds the shared primitive.
---
## Phase 4 — Stretch: the blast-radius prioritization report (optional, 10 min)
If time remains, write `prioritization_report.md` that:
1. Lists the seven surfaces in your remediation-priority order (from Phase 2.2).
2. For the top three, names the specific Acme-agent component that makes the blast radius what it is (e.g., "sandbox: the Docker container has no syscall allowlist and no egress filter, so `execute_python` can reach the network and the host filesystem").
3. For each of the top three, names the one control that would most reduce the blast radius if implemented tomorrow (e.g., "sandbox: add a seccomp syscall allowlist and network egress filter — drops blast radius from host-wide to contained").
4. States which Course 1 architecture (DD-09, DD-16, DD-19, DD-20) is the reference implementation for each top-three control, and the specific gap in it that B6/B7/etc. must close.
The point: prioritization is not abstract. It is "which component, which control, which reference architecture, which gap." A report that names all four is a report a CISO can act on.
---
## Deliverables
- `agent_surface_map.json` — the JSON threat model, seven surfaces, validated (Phase 2)
- `prioritization_justification.md` — the three key judgments about the Acme agent (Phase 2.3)
- `classify_content.py` — the `classify_untrusted_content()` function + six test calls (Phase 3)
- (optional) `prioritization_report.md` — the blast-radius prioritization report (Phase 4)
## Success criteria
- [ ] `agent_surface_map.json` validates, has all seven surfaces, each with trust boundary, entry points, canonical attack, ASI risk, blast radius, and destination module.
- [ ] The `untrusted_entry_points` for each surface are specific to the Acme Support Agent (not generic).
- [ ] `remediation_priority` ranks containment-failure surfaces (sandbox, identity, inter-agent edges) before single-session surfaces (the loop), with reasons.
- [ ] The three key judgments in `prioritization_justification.md` trace specific attack paths through the Acme architecture (sandbox egress, identity amplification, forged peer message).
- [ ] `classify_untrusted_content()` returns the correct trust level, tag, and isolation/verification flag for all six test origins.
- [ ] `system_prompt` is the only `trusted` origin; every other origin is `untrusted` or `semi_trusted`.
- [ ] Every artifact ties back to a specific surface, ASI risk, and principle from the teaching document.