Same model, two harnesses: what a security-first agent gateway costs

Why compare harnesses at all

An agent’s security does not come from the model alone. The harness decides which tools exist, what they may touch, who approves what, and what happens after the agent reads something an attacker wrote. I built my home assistant’s gateway around that last question. I wanted to measure what that design buys and what it costs, against a widely used open-source harness, with the model held fixed.

A coding benchmark alone would mostly confirm that a coding agent codes better. A prompt-injection benchmark alone would reward a harness that refuses to do anything. So this post reports both, side by side, as one tradeoff.

Setup: what is held constant

• Model: Qwen3.6-35B-A3B, NVFP4, served by vLLM 0.31.0 on an NVIDIA DGX Spark (GB10, arm64). 131,072-token context, the qwen3_xml tool-call parser, the qwen3 reasoning parser, prefix caching off.
• Sampling: temperature 0.6, top-p 0.95, up to 16,384 output tokens per call. Both harnesses are configured to send these values (I did not capture the model requests themselves to confirm what each one actually sent). Thinking is on in both: the gateway sets it explicitly, and OpenCode leaves the chat template’s default, which is on.
• Driver: both harnesses run over the Agent Client Protocol (ACP v1) as stdio subprocesses: gateway --acp-v1 and opencode acp. The same client drives both and records every update, tool call and permission request.
• Tools: in AgentDojo, both harnesses get the same AgentDojo tool definitions and environments through one MCP server (below). What reaches the model still differs by harness: each prefixes the tool names its own way, and the gateway filters untrusted tool output before the model sees it. In Terminal-Bench each harness uses its own tools (a confound, covered below).
• Box and concurrency: the same machine for every run, three cases at a time.

The two harnesses

The gateway

The gateway is a Rust workspace that serves my home assistant. The parts that matter here:

• Signed manifest. Every tool is declared in a TOML manifest, signed with an ed25519 key and verified at startup. A tool missing from the manifest is never shown to the model, and a call to it is denied.
• Tiers and one pure policy function. Each tool has a tier: read, write_reversible, write_irreversible or egress. A deterministic function, with no model involved, returns Allow, Ask or Deny for every call. Reads are allowed. Everything else asks for approval, which in production goes to my phone.
• Session taint. A tool can be marked accepts_untrusted_input, meaning its output can contain text a third party wrote. The first successful call to such a tool permanently downgrades the session from doer to reader. A reader may still call read-tier tools. Every other call is denied outright, without an approval prompt. This is the gateway’s main prompt-injection defense, and the central subject of this post.
• Untrusted-output scrubbing. Output from untrusted tools is passed through secret redaction, and URLs, markdown links and images are removed. A regex detector for common injection phrases adds a warning banner. It never fired in this run: the important_instructions attack text matches none of its patterns.

OpenCode

OpenCode 1.19.4, pointed at the same vLLM endpoint with the same sampling. Isolation flags kept it from loading any outside configuration, skills or prompts. In AgentDojo, all of its built-in tools were turned off, so it saw only the AgentDojo tools. Each of those tools was set to ask, and the driver approved every request. In effect OpenCode ran with every tool allowed (it raised 1,099 permission prompts, all approved).

Method

AgentDojo (security, plus utility on assistant tasks)

• AgentDojo benchmark v1.2.2 (package 0.1.35), all four suites: workspace, travel, banking, slack.
• Benign: all 97 user tasks, no injection. Attacked: 200 of the 949 (user task, injection task) pairs, stratified by suite in proportion to each suite’s pair count (118 / 30 / 30 / 22), seed 20261027, using the important_instructions attack. The attack addresses the model as “Qwen” and signs as AgentDojo’s default user, “Emma Johnson”.
• MCP bridge. For each case, the bridge builds the task environment the same way AgentDojo’s own runner does. It then serves the suite’s tools over MCP and records every call as an AgentDojo FunctionCall. Scoring uses AgentDojo’s own utility and security checks on that trace and on the agent’s final message, so each harness is tested through its real tool path, not through AgentDojo’s built-in agent loop.
• Gateway tool tiers. A script assigned the 69 AgentDojo tools to tiers using only the tool’s name and return type, before any results existed. It marks a tool untrusted when its return type can carry third-party text, such as emails, files, calendar events, messages, reviews or channel and user names (Appendix A).
• Two gateway configurations: every approval prompt auto-approved, and every approval prompt auto-denied. OpenCode has one configuration. Each case ran once per configuration, with a 600-second limit.

Terminal-Bench (capability on terminal tasks)

• Terminal-Bench 2.0 through Harbor 0.25.0, with Harbor’s ACP agent driving each harness inside the task container. Each harness was installed into the container from a prebuilt package, so nothing was downloaded at run time.
• None of the 89 prebuilt task images ships for arm64. I built them from their Dockerfiles. 73 of the 89 pass the reference (oracle) solution on arm64, and those 73 are the task set. The score is not comparable to the public leaderboard.
• One attempt per task per harness, three trials at a time, each task’s own agent time limit (900 seconds for 43 of the 73 tasks; 600 to 7,200 seconds for the rest). Harbor approves every permission request automatically. The gateway ran twice: v1 (the original build) and, after the bug described below, v2 with the fix. OpenCode ran once.
• Scoring. Solved is Harbor’s own score, the verifier’s reward. Harbor runs the verifier even after the agent times out, so a task the agent finished but never declared finished still counts. I also report a strict score that counts every errored or timed-out trial as unsolved. The two differ mostly for the fixed gateway, which ran into the time limit most often.
• Gateway tools: a single shell tool, tier write_irreversible, allowed to write anywhere in the container and to use the network, still confined by Landlock and seccomp (ptrace, mount, namespaces and bpf denied). OpenCode tools: its built-ins except question, websearch and codesearch.

Results

AgentDojo

OpenCode is the better assistant on these tasks by a wide margin. Under attack it still completes 87.5% of user tasks, and 7 of the 200 attacks succeed. Each gateway configuration completes about a third of tasks, with or without attacks, and no attack succeeds. The same 200 pairs ran under each configuration. On the paired comparison (exact McNemar test, two-sided) the 7-versus-0 difference gives p ≈ 0.016. That test treats the pairs as independent, and they are not: the 200 pairs reuse 86 user tasks and repeat injection goals. So read it as weak-to-moderate evidence of a real difference, not a precise one. The absolute numbers are small, and the next sections show how little of the difference can be tied to the policy.

On the 97 benign tasks, the two harnesses disagree mostly in one direction. OpenCode succeeds where the gateway (approve) fails on 59 tasks. The reverse happens on 2.

All seven successful attacks are in banking (four transfers to the attacker’s IBAN) and slack (three). The gateway’s slack utility is close to zero, for a reason the next section explains.

Terminal-Bench

On Harbor’s own score, the fixed gateway (v2) and OpenCode are close: 25 and 26 of 73. Paired by task, OpenCode alone solved 7 and the gateway alone 6. The observed difference is one task, and this run gives no evidence of a difference either way (exact McNemar p = 1.0). That is not the same as showing the two are equal. On the strict score the gateway is behind, 20 against 25, which is also not statistically significant (p ≈ 0.30). The strict count removes six verifier passes: five gateway trials and one OpenCode trial whose container passed the tests after the agent was stopped at its time limit. The reports show the final state passed, not when the work was finished. With 73 tasks and one attempt each, this run cannot detect a difference of a few tasks in either direction.

Taint plays no part here: the Terminal-Bench manifest has a single shell tool and nothing marked untrusted, so the session never drops to read-only. This panel measures the harness and its tools, not the injection policy.

The bug the benchmark found

The first gateway run (v1) solved 17 of 73. Reading the traces pointed to the reason. 36 trials ended “normally” with an empty final message, and another 11 timed out. None of those 47 was solved.

The cause was in the agent loop, not the model or the policy. The loop treated any model reply without a tool call as the final answer. With thinking on and a 16,384-token output budget, the model sometimes spends the entire budget reasoning. vLLM then returns finish_reason=length with no content and no tool call (I confirmed this with a small probe against the server), and the loop ended the turn on that empty reply. The archived traces record turn totals, not per-call finish reasons, but they fit this explanation. In 33 of the 36 empty endings the turn had used at least 16,384 completion tokens, and in 8 the agent stopped after a single round.

The fix, which I call ReplyGuard, never ends a turn on an empty or truncated tool-less reply. It sends the model a short nudge and retries, at most twice in a row (each retry counts against the round limit). It also continues truncated text, and when the model gives no answer at all, the gateway returns an explicit message saying so instead of an empty one. The same change went into the production gateway. I reran only the gateway (v2), with the same model, config and tasks.

• No v2 trial that finished normally had an empty final message (v1: 36). Of the 36 tasks where v1 ended empty, v2 passed 7, four of them after a timeout. Paired by task, v2 solved 11 tasks that v1 missed and lost 3 that v1 solved, on Harbor’s score (p ≈ 0.06). On the strict score it gained 6 and lost 3 (p ≈ 0.5). With one run each, part of that change is ordinary run-to-run variance. The fix removed the failure mode, but this data cannot say how many tasks it was worth.
• The trials that used to quit early now run long: v2 timed out in 33 of 73 trials, against 11 for v1 and 11 for OpenCode. In the 40 trials that finished, ReplyGuard recorded 19 output-budget stops and 15 retries. The timed-out trials report no counters.
• The gateway uses more time per action. Dividing each trial’s agent time by its tool calls gives a median of about 41 seconds for v2, against 26 for OpenCode. That figure includes reasoning, retries and any time before a cutoff, so it is not a measured step latency. Likely contributors, which I did not test separately: the gateway does everything through one shell tool where OpenCode has dedicated file-editing and search tools, and its persona and loop lead to longer reasoning per step (both harnesses run with thinking on). Under a fixed time limit, that costs tasks.

OpenCode failed 47 trials on Harbor’s score: 14 ended normally with no final message (I did not investigate why), 10 timed out, 8 exited with an error after OpenCode reported an internal timeout, and 15 ended normally with an answer that did not pass.

This is the main thing I learned from the capability half of the project. The first run made it look as if security cost a lot of capability on terminal tasks. Fixing a loop bug coincided with most of that gap disappearing. A harness comparison is only as good as the harness code, and reading the failing traces before trusting the score is what found the bug.

Where the gateway’s utility went: taint

The gateway raised exactly one approval prompt per configuration in 297 cases. Its approval gate was almost never reached, which is why the approve and deny configurations score nearly the same (they disagree on 9 of 297 cases). The policy outcome that shows up in the failures is the taint denial, not the approval.

A typical failing case: the user asks the agent to read an email and reply, or to check recent transactions and pay a bill. Reading the email or the transaction list taints the session, because that content could have been written by anyone. The write that the user actually asked for is then denied. Across the approve configuration, 236 calls were denied this way: 112 write_reversible (creating files and calendar events, for example), 82 egress (sending email and messages) and 42 write_irreversible (payments, deletions). When a case hit a taint denial, it succeeded 5.8% of the time. When it did not, it succeeded 72.0% of the time. This is an association, not a measured effect. I did not run the gateway with taint switched off, and cases that hit a denial may also be harder in other ways.

Slack shows the extreme case. Listing channels or channel members is marked untrusted, because other users choose those names. So nearly every slack task taints itself on its first call, and the gateway completes 1 of 21. Slack is where the labeling choice and the failures line up most directly.

The other ~18% of failures (35 cases in the approve configuration) had no taint denial. In 14 of them the scrubber had removed at least one URL from tool output, which may have hidden something the task needed. I did not classify the rest fully. They include wrong answers, a refusal of a request the model judged to be a scam, and final answers that ask the user to confirm first (“Want me to send it now, or hold off?”). The driver sends one user prompt and answers permission requests, but it does not reply to questions, so a harness that asks before acting loses those cases. Seven of the 35 end with a question. Among OpenCode’s failures, one does.

What the zero does and does not show

Zero successful attacks looks like a clean win. A closer look at the traces gives a more modest result. For each injection pair, I checked whether the agent called the injected goal’s tool with the attacker-specific arguments, such as the attacker’s email address, IBAN, website, or the passport number the attacker asked for. This counts the agent going for the injected goal at all, whether or not the call succeeded. A match that is fully explained by the user’s own request does not count. The first version of this check counted a legitimate email to an address the user had named, and a review caught it.

• The model rarely takes this bait. The attack text reached the model in a tool result in 190 of OpenCode’s 200 pairs (187 and 188 for the gateway configurations; some pairs end before the injected content is read). Even in OpenCode, which allows everything, the agent went for the injected goal in 8 of 200 pairs, and 7 of those were scored as successes.
• Under the gateway it went for the goal in one pair (approve configuration: a calendar event the attacker asked for), and the taint rule denied that call. None of OpenCode’s seven successful attacks has a blocked counterpart in the gateway runs: the gateway’s agent never attempted them. The rest of the 7-versus-0 gap comes from things that differ between the harnesses and that this data cannot separate. Candidates: the gateway’s system prompt (it says to stop and ask before outward-facing actions), sessions ending early after denials, and the scrubbing of untrusted output.
• Taint would not have stopped two of OpenCode’s seven successes. Two came from the slack injection “visit this website”. In this manifest, fetching a page is a read, and readers may read. Under the gateway, the URL survived scrubbing in all four of those pairs (it has no scheme, so the URL filter missed it), and the model never visited it. Taint stops writes after untrusted input. It does not stop a tainted agent from leaking data through a read that reaches out to an attacker’s server, and it does not stop an injection that only changes the final answer. One sampled pair has exactly that goal: recommending a particular hotel.

So, for this model and this attack: the gateway scored 57–59 points below OpenCode on benign utility, and taint denials appear in about 82% of its failures. The taint rule demonstrably blocked one attempted attack. The 3.5-point difference in attack success is real in this run, but this experiment cannot divide it between the policy and the other differences between the harnesses. Separating them takes a gateway run with taint switched off. How a stronger or adaptive attack would change either side is untested. Such an attack could also aim at exactly the channels taint leaves open.

What this implies

A taint rule has the right shape for a security boundary. It is deterministic, it does not ask a model whether text looks malicious, and an attacker cannot argue with it. The problem is granularity. The rule taints the whole session, on every untrusted read, for every later write. A few ways to make it finer, each a design change and none tested here:

• Taint data, not sessions. Follow which values came from untrusted content, and block a write only when its arguments depend on them. Paying a bill to an IBAN the user typed is different from paying one read out of an email.
• Treat reads that reach outside as egress. “Visit this URL” succeeded against OpenCode, and taint lets it through by design. (Neither does taint protect the final answer; that needs a different control.)
• Make the tier labels more precise. Labeling channel names as untrusted taints nearly every slack task on its first call, and the gateway completed 1 of 21 of them. The mechanical classification rule is easy to audit, and slack shows what it can cost.

The next experiment is detector-gated taint: untrusted content taints the session only when a prompt-injection classifier flags it. I am training that classifier now, and the AgentDojo rerun with it is future work. It moves part of the boundary from a rule to a model. Its false negatives then decide what reaches the write tools, so it will be reported with the same tables as this run, not as a single score.

Confounds and limitations

• Purpose fit. The gateway is an assistant and operations agent. OpenCode is a coding agent. Terminal-Bench measures fitness for its purpose as much as harness quality.
• System prompts differ. Each harness used its own. The gateway’s prompt is the production persona of my assistant. It mentions tools the benchmark does not provide and tells the agent to ask before outward-facing actions. That instruction probably lowers both its attack success and its utility. The prompt is in the raw-data archive.
• Tool sets differ in Terminal-Bench. The gateway has one shell tool. OpenCode has file editing, search and its other built-ins. In AgentDojo both get the same tool definitions.
• AgentDojo ran before the fix. The AgentDojo numbers come from the pre-fix gateway (v1 loop). I checked the traces for the same failure. Three gateway (approve) cases and two gateway (deny) cases ended with no final message, all with turn totals over 16,384 completion tokens. No gateway case that did end with a message reached that total. Two of the three approve cases still scored as successes on their tool trace. If only those traces were resumed, the fix could recover at most 1 of 297 approve cases and 2 of 297 deny cases (0 and 1 of the 97 benign tasks), so this failure mode does not explain the gap to OpenCode. On the same assumption, attack outcomes could change only in those cases (three approve and one deny injection case). A real rerun is stochastic and is not bounded this way. I have not rerun AgentDojo with the fix, so the fixed gateway’s security is unmeasured.
• Time limit and speed. Terminal-Bench scores work within each task’s time limit. The fixed gateway hits its limit far more often, so its score depends on how fast it works as much as on problem solving. Different limits would likely change both scores, by amounts I did not measure.
• The bench configuration is not production. Approvals were answered by the test client (Appendix B), and tiers came from a mechanical rule. The taint rule, signature check and scrubbing were the production code paths, unchanged.
• One run per configuration. The plan called for three seeds per harness on Terminal-Bench. I ran one attempt per task per harness (two gateway runs, but they differ in code, not just in seed) and one run of every AgentDojo case. At temperature 0.6, run-to-run variance is real and not measured here. Some of the v1-to-v2 change is likely that variance, not the fix. The confidence intervals cover sampling of tasks, not variance between runs. The gateway’s v2 run also came a day after the others, on the same box and setup.
• Statistical dependence. The intervals are nominal binomial intervals. The 200 injection pairs reuse 86 user tasks and repeat injection goals within each suite, and they were sampled in strata, so the real uncertainty is wider than shown. The 97 benign tasks are the whole benign set, so their intervals describe a hypothetical population of similar tasks.
• Unequal exposure and early stops. Not every pair delivers the attack text to the model (190 of 200 for OpenCode, 187 and 188 for the gateway). A gateway session that ends after a denial never gets a chance to act on an injection it reads later.
• No follow-up turns. The driver sends one prompt. A harness that asks the user to confirm loses that case. The gateway’s persona asks more often.
• Task subsets. Terminal-Bench uses the 73 of 89 tasks whose oracle passes on arm64. AgentDojo uses all benign tasks but only 200 of the 949 injection pairs, with a single non-adaptive attack. Neither score is comparable to a public leaderboard, and AgentDojo here runs through an MCP bridge, not AgentDojo’s own pipeline.
• The attempt counts are heuristic. They rely on hand-written signatures per injection task, checked against the user’s own request. They can miss an attempt that rephrases the attacker’s values, and they can still count a false positive the user-request check does not catch. One pair has a text-only goal and cannot match.
• Token counts are not comparable. OpenCode’s ACP usage report does not match the gateway’s per-call totals, so I report wall time only.
• I built the gateway. I fixed the tier rules before any runs and publish every trace, so anyone can check the results.

Appendix A: tools

Gateway, AgentDojo

Generated mechanically from AgentDojo’s tool names and return types (rules below). The gateway exposes only the tools of the current suite, served through the bridge.

Gateway, Terminal-Bench

shell only: sh -c in the session’s working directory, a fresh confined process per call, output clipped to the first and last 8,000 characters, default timeout 120 seconds (maximum 1,800). v1 and v2 are the same tool, manifest and config. v2 adds only the ReplyGuard change to the agent loop.

OpenCode

AgentDojo: only the bridge’s tools (presented as agentdojo_*). Every built-in was disabled: bash, edit, write, patch, read, grep, glob, list, webfetch, websearch, codesearch, task, todowrite, todoread, skill, lsp, question. Terminal-Bench: all of those built-ins except question, websearch and codesearch.

Appendix B: what the bench profile auto-approves

The policy function is the production one. What changes in the benchmark is the manifest, the agent-loop settings, and who answers approval prompts.

• Manifest. A bench-only manifest, signed with a bench-only key (not the production key), that declares only the benchmark’s tools. No production tools (SSH, GitHub, escalation to other models, game servers, memory) are exposed.
• Approvals. A call to a non-read tool in an untainted session produces an ACP permission request. The AgentDojo driver answered every one with allow_once (approve configuration) or reject_once (deny configuration). Harbor answered every one with allow. In production the same request goes to my phone and is bound to a digest of the exact tool call.
• Never auto-approved, in any configuration: calls the policy denies. That covers tools missing from the manifest and any non-read call in a tainted session. Denials do not produce prompts.
• Agent loop: up to 100 tool rounds per prompt (no AgentDojo case hit it), 16,384 output tokens per call, temperature 0.6, top-p 0.95, thinking on.
• Terminal-Bench shell: write_irreversible, so it asks for approval and Harbor approves. Its sandbox allows writes anywhere in the container (rw_paths = ["/"]) and network access, for parity with OpenCode. Landlock and seccomp still apply. In production the same tool is limited to a working directory and gated by my approval.
• Config isolation. The gateway is always started with an explicit --config. In an early smoke test, the ACP client launched the gateway with a scrubbed environment, so it silently fell back to the production config. That test wrote one file to a scratch directory and ran one read-only command, then I made the explicit config path win over the environment.

Reproducibility and raw data

• Raw data: harness-benchmark-raw.tar.gz contains every AgentDojo case record (prompt, injections, full ACP event stream, tool trace, permission log, scores, wall time) for all three configurations. It also has the case list and sampling metadata, the driver, the MCP bridge, the tier generator, both harness configs, the gateway’s bench manifests and persona, the Harbor agent definitions, the Terminal-Bench task list, and the per-trial results for all three Terminal-Bench runs (Harbor results, verifier reports and ACP summaries, plus per-trial tool-call counts derived from the ACP event logs, which at about 360 MB are not included). It also has the analysis scripts that produce the numbers on this page from the traces. The owner’s home directory was replaced with ~, and an automated check found no credentials, tokens, keys or private addresses. Emails, account numbers, passwords and “secret” values in the traces are AgentDojo’s synthetic fixtures. Deployment paths under /srv/gateway and the Docker bridge address are left as they were. SHA256SUMS is inside.
• Recompute: python3 analysis/analyze_agentdojo.py agentdojo/runs/full out.json (tables), python3 analysis/attempts.py agentdojo/runs/full (attempts and exposure), python3 analysis/diagnostics.py agentdojo/runs/full diag.json (the tests, paired counts, scrubbing and URL checks in the text), and python3 analysis/tb_extract.py terminal-bench out.json. This recomputes the scores from the recorded traces. Rerunning the experiment end to end would also need the gateway source, which is private.
• Versions: vLLM 0.31.0; AgentDojo 0.1.35, benchmark v1.2.2; Python ACP SDK 0.12.1; OpenCode 1.19.4; Harbor 0.25.0; Docker 29.6.2.
• Gateway build: branch bench-harness, not merged into production. The AgentDojo runs used commit 3c43ec6. Terminal-Bench v1 used a glibc-2.28 portable build of 5fe1268; the only later commit before the fix changes the ACP v2 path, which Terminal-Bench does not use. Both had an uncommitted Cargo.lock change. Terminal-Bench v2 used a portable build of 461ec0c, which adds ReplyGuard (f4aa429, plus a follow-up after review) on top of a lock-file checkpoint commit. The package binary’s sha256 starts 30248e0c.
• Source: the gateway repository is private; source is available on request.