Three coding agents, one task

The harness is the variable

Asked which coding agent is best, the honest answer from this data is: whichever one you can still change. We ran Marshall against pi, opencode and aider on identical work — 105 trials, three models, three task shapes — and the widest gap in the whole benchmark wasn't between harnesses. It was between two configurations of Marshall.

Same model, same fixture, same verifier: 91 tool calls at the worst and 5.7 at the best, a 16× spread. The worst burned 455,372 input tokens and $0.0353 a run; the best used 72,296 and $0.0078. pi and opencode both landed inside that range on the same task, at 14.8 calls and 13.0. pi's cost per run was the same as Marshall's least efficient configuration, and 4.5× its best.

So this stopped being a league table partway through. It became a study of what makes a harness good — and because the harness we could rebuild was our own, most of the measurements are of Marshall taking itself apart. Every configuration below was built to test one idea about that; the belt Marshall ships is the one the testing produced, and section 08 traces each of its properties back to the experiment that justified it.

Five properties turned out to be measurable, and they are the spine of the piece: it finishes; it is cheap; it doesn't need a human steering it to be fast; it holds up across task shapes; and it gives the model a cheap way to do the obvious thing. Section 09 scores all four harnesses against them.

The task

A 28-file JavaScript project is halfway through a logging migration. A MIGRATION.md describes the rules. Every remaining call site has to move off the deprecated log() helper, the old module has to be deleted, and node --test has to pass afterwards — 122 tests that grade the result.

Every harness gets the same fixture, the same prompt, and the same check(). No harness is told how to do it: reading every file one at a time and scripting the whole thing in one shot are both allowed, and which one a harness picks turns out to be most of the story.

Three models ran it. GPT-5.6 Luna is the strong general model. GLM-5.3 Flash and Qwen3.8 Flash are two small, fast ones — and they are where this benchmark earns its keep, because a capable model can absorb a mediocre harness and a weak one cannot. Every trial had a 7-minute ceiling.

Results

GPT-5.6 Luna

opencode3/3 passed

120.5 s

Marshall · full belt5/5 passed

40.6 s

Marshall · shell-first belt3/3 passed

32.4 s

pi4/4 passed

31.1 s

GLM-5.3 Flash

Marshall · full belt2/2 passed

186.5 s

Marshall · shell-first belt2/2 passed

126.1 s

opencode2/2 passed

118.9 s

pi2/2 passed

77.7 s

Qwen3.8 Flash

pi⏱ timed out, 0/2

420.1 s ceiling

opencode⏱ timed out, 0/2

420.1 s ceiling

Marshall · shell-first belt3/3 passed

146.5 s

Bars are scaled within each model group; the 7-minute ceiling is the longest bar wherever a harness never finished. On Luna all four plotted configurations pass, and Marshall's shell-first belt is a hair behind pi — 32.4s against 31.1s — at roughly a quarter of the cost. Marshall's second Qwen configuration also finished, at 112.7s median and 3/3.

GPT-5.6 Luna

pi4/4 passed

$0.0353

Marshall · full belt5/5 passed

$0.0104

Marshall · shell-first belt3/3 passed

$0.0078

GLM-5.3 Flash

pi2/2 passed

$0.0266

Marshall · shell-first belt2/2 passed

$0.0099

Marshall · full belt2/2 passed

$0.0087

opencode writes no token log this harness can read, so it isn't plotted. Qwen has no comparison at all: pi and opencode never finished, so they have no cost per completed run. Marshall's two Qwen configurations ran $0.0120 and $0.0155.

The full table, all four harnesses and all three models:

Marshall's cell is its best-performing configuration on that model; both of its belts are plotted in the figures above. ⏱ = every trial hit the 7-minute ceiling. n/a under calls means the harness has no tool-calling concept, not zero: aider applies SEARCH/REPLACE blocks from its own reply rather than calling tools. 105 trials total; means for calls, medians for time.

Marshall is the only harness that finishes on Qwen3.8 Flash

pi and opencode timed out on every attempt between them — four trials, four ceilings, no result. Marshall completed 3/3, twice over: two different configurations of it, three trials each, all passing. This is the single largest gap in the benchmark, and it is the one that a league table of wall-clock seconds hides completely, because a run that never finishes has no median.

The harness matters most where the model is weakest

On Luna, all three of Marshall, pi and opencode finished: 3/3, 4/4, 3/3. On GLM-5.3 Flash, all three finished. On Qwen3.8 Flash, only one did. When the model is capable, harness quality shows up in the price and the call count; when it isn't, harness quality decides whether you get an answer at all. If you are running anything smaller than a frontier model — and most production agent work is — this is the axis to optimise.

On Luna, Marshall is the leanest and by a distance the cheapest

5.7 tool calls against pi's 14.8 and opencode's 13.0, at 3/3 correctness, and 3.7× faster than opencode by median run. On cost the gap is wider than on any other measure: between 22% and 37% of pi's dollars per run on every task and model in this piece where both harnesses logged tokens.

pi is still ahead on GLM-5.3 Flash

7 calls to Marshall's 10, and 77.7s against 126.1s by median run. One model out of three, and we did not close it. What the gap is not, though, is a correctness or cost gap: Marshall passed the same 2/2 on that model and cost 2.7× less per run. And the shell-first belt closed most of it on its own — the same model went from 186.5s to 126.1s when the belt changed, with no other variable touched.

Every configuration measured

The Marshall rows above are not one number, and that is the point of this section. Each configuration was built to test a single idea about harness design — batch the file tools, drop the read-before-edit gate, route bulk reads through the shell, remove the read tools entirely. The spread between them is larger than the spread between the harnesses, which is the strongest argument in this piece for the harness being worth engineering.

luna-hybrid-prompt-light, luna-hybrid-shell-prompt and luna-incremental-prompt ran a single trial each, so their figures are measurements and not averages. The headline figures in section 03 are Marshall's best-performing configuration on each model — the shell-first belt on all three — and every other Marshall row here is an ablation measured against it.

Read the ranges before you read any mean in this piece. Several of them are wider than the differences between harnesses: pi and Marshall finish the Luna task 1.3 seconds apart by median, and the same configuration repeated on the same inputs can vary by far more than that.

• 120.5sopencode's spread on Luna — three passing trials at 58.1s, 137.9s and 120.5s, a 2.4× range within one configuration that never changed.
• 21–47spi's spread on Luna — four trials, and the fastest is less than half the slowest.
• $0.0131–$0.0586pi's spread on cost — a 4.5× range between its cheapest and dearest run of the same task.
• 87–297sMarshall on Qwen — three passing trials of one configuration, 3.4× apart.
• 53–320sMarshall on GLM — two trials, six times apart.

That is the single most important thing to hold in mind when reading any number here. The differences between harnesses that survive it are the ones worth believing: the Qwen timeout, the cost gap, and the 16× spread between Marshall's own configurations, which is far outside anything noise produced.

Every migration trial, individually — 54 runs

Configuration	Trial	Result	Calls	Seconds	In tok	Out tok	In tok/call	$
aider-luna	1	fail	—	86.3	5,100	429	—	$0.0015
aider-luna	2	fail	—	117.7	5,100	595	—	$0.0017
luna-hybrid-batch-prompt	1	pass	14	28.1	113,712	1,636	8,122	$0.0086
luna-hybrid-batch-prompt	2	pass	14	39.1	154,120	2,224	11,009	$0.0119
luna-hybrid-batch-prompt	3	pass	19	35.1	115,238	2,136	6,065	$0.0108
luna-hybrid-batch-prompt-light	1	pass	19	44.0	210,197	2,102	11,063	$0.0132
luna-hybrid-batch-prompt-light	2	pass	13	25.2	117,156	1,941	9,012	$0.0130
luna-hybrid-batch-prompt-light	3	pass	12	22.9	87,696	1,657	7,308	$0.0094
luna-hybrid-prompt	1	pass	23	42.4	168,005	2,208	7,305	$0.0130
luna-hybrid-prompt	2	pass	17	32.3	93,767	1,860	5,516	$0.0080
luna-hybrid-prompt	3	pass	21	31.2	131,091	1,738	6,242	$0.0083
luna-hybrid-prompt-light	1	pass	22	45.1	80,129	2,068	3,642	$0.0076
luna-hybrid-shell-prompt	1	pass	20	38.8	102,407	2,103	5,120	$0.0095
luna-incremental-prompt	1	pass	91	107.5	455,372	13,035	5,004	$0.0353
luna-nomask	1	pass	12	30.7	159,451	1,609	13,288	$0.0108
luna-nomask	2	pass	12	23.3	93,501	1,656	7,792	$0.0080
luna-nomask	3	pass	13	26.8	166,885	1,665	12,837	$0.0129
luna-openrouter	1	pass	17	36.2	119,316	1,794	7,019	$0.0089
luna-openrouter	2	pass	23	60.4	189,627	2,578	8,245	$0.0126
luna-openrouter	3	pass	16	40.6	91,659	1,818	5,729	$0.0076
luna-openrouter	4	pass	14	31.9	159,617	1,736	11,401	$0.0110
luna-openrouter	5	pass	15	52.7	152,931	2,382	10,195	$0.0120
luna-shell-edit	1	pass	5	27.5	61,254	1,854	12,251	$0.0071
luna-shell-edit	2	pass	6	37.0	85,293	2,078	14,216	$0.0090
luna-shell-edit	3	pass	6	32.4	70,342	1,538	11,724	$0.0073
opencode-luna	1	pass	16	58.1	—	—	—	—
opencode-luna	2	pass	14	137.9	—	—	—	—
opencode-luna	3	pass	9	120.5	—	—	—	—
pi-luna	1	pass	16	25.3	67,138	1,720	4,196	$0.0234
pi-luna	2	pass	15	47.0	122,167	3,016	8,144	$0.0462
pi-luna	3	pass	16	36.8	179,014	2,304	11,188	$0.0586
pi-luna	4	pass	12	21.5	36,550	1,324	3,046	$0.0131
qwen38flash-or-shell-edit	1	pass	27	274.4	402,202	12,048	14,896	$0.0174
qwen38flash-or-shell-edit	2	pass	17	101.3	158,085	6,223	9,299	$0.0084
qwen38flash-or-shell-edit	3	pass	25	146.5	259,808	5,954	10,392	$0.0101
qwen38flash-or-solo	1	pass	28	297.3	707,793	13,881	25,278	$0.0247
qwen38flash-or-solo	2	pass	25	112.7	383,915	8,017	15,357	$0.0139
qwen38flash-or-solo	3	pass	23	87.0	212,128	4,465	9,223	$0.0079
opencode-qwen38flash	1	⏱ timeout	0	420.1	—	—	—	—
opencode-qwen38flash	2	⏱ timeout	0	420.1	—	—	—	—
pi-qwen38flash	1	⏱ timeout	0	420.1	—	—	—	—
pi-qwen38flash	2	⏱ timeout	0	420.1	—	—	—	—
glm-hybrid-batch-prompt	1	pass	25	211.7	458,802	9,702	18,352	$0.0254
glm-hybrid-batch-prompt	2	⏱ timeout	49	420.0	423,618	15,076	8,645	$0.0128
glm-openrouter	1	pass	17	320.2	290,841	9,795	17,108	$0.0087
glm-openrouter	2	pass	11	52.8	125,203	4,764	11,382	$0.0086
glm-shell-edit	1	pass	9	102.6	197,672	9,148	21,964	$0.0159
glm-shell-edit	2	pass	11	149.5	113,886	4,400	10,353	$0.0038
glm-shell-only	1	pass	8	75.4	142,936	6,866	17,867	$0.0121
glm-shell-only	2	fail	6	187.6	40,520	2,283	6,753	$0.0018
opencode-glm	1	pass	9	121.0	—	—	—	—
opencode-glm	2	pass	7	116.8	—	—	—	—
pi-glm	1	pass	8	101.2	36,791	2,009	4,599	$0.0215
pi-glm	2	pass	6	54.1	67,960	905	11,327	$0.0317

Two of Marshall's rows are not passes: glm-hybrid-batch-prompt hit the ceiling on its second trial at 49 calls, and glm-shell-only failed outright on one of two. Both are early ablations — the first predates the two changes in section 08, the second removes the read tools entirely — and both are in this table because a benchmark that only shows the configurations that worked is not a benchmark.

What the calls were spent on

Call counts say how many round trips. This says what they were spent on, and it is the part that explains everything else. The distance between 37 calls and 5.7 is not tidiness. It is one harness reading files one at a time — read_file 46 of its 74 calls — and another running a shell loop, run_shell 17 of 17.

No composition recoverable for aider-luna, opencode-luna, opencode-qwen38flash, opencode-glm — opencode writes no transcript this harness can read, so its calls are counted but not categorised. Rows marked ⏱ timed out on every trial; their composition is still real, it is what the harness did before the ceiling, but it is a partial run.

A round trip is what costs

The reason call count matters is mechanical, and it is worth stating because it explains the token numbers rather than just correlating with them. Each tool call ends a turn, and the next turn re-sends the whole conversation so far. A harness that needs ten round trips to do one job pays for that job ten times, and the bill grows with every one of them.

The two extremes in this data make the arithmetic plain. luna-incremental-prompt edited one file per call and took 91 round trips: 455,372 input tokens. luna-shell-edit did the same migration in 5.7: 72,296 input tokens. A 6.3× difference in what the same model read to complete the same task, produced entirely by how the harness let it work. Note that per-call cost went the other way — 5,004 input tokens per call against about 12,700 — because each of the fewer calls carried more. Fewer, larger turns is the whole trick.

pi found this on its own

pi runs almost everything through bash. Its own read tool is a minority of its calls; the bulk is shell, including a for f in $(grep -rl …); do cat "$f"; done that ingests every relevant file in one call. On GLM it was 93% shell. That is the same strategy Marshall's leanest belts converge on, arrived at from the other direction.

Marshall's full belt spreads the same work across four tools

read_file, search, list_dir and run_shell — and the read_file share is where its extra calls live. 49% of calls on luna-hybrid-prompt, 62% on glm-hybrid-batch-prompt. Those are file-at-a-time reads of a 28-file project.

The shell-first belt collapses, but only on some models

100% of calls on Luna, 95% on GLM. On those two the model does what pi does and the call count follows. On Qwen3.8 Flash it does not: the belt still pushes it mostly to shell, but it issues 23 calls doing so rather than 5.7. Same belt, same instruction, different model — which is the clearest evidence in this piece that a harness sets the ceiling and the model decides how much of it you get.

A realistic codebase

Every fixture above is small enough that a harness can hold all of it at once. That flatters any approach which front-loads context and never has to search, so it is worth one task where finding the work is the work.

repo-bugs is 45 real TypeScript files — Marshall's own packages/tools — with three faults planted in them: an off-by-one in a shared line-window primitive, an inverted glob guard in search, and a type error in a function no test executes. 248 tests, of which exactly 2 fail. Nothing in the prompt names the faulty files, the failing tests are not all in the files that cause them, and the type error is invisible to the test run, so the verifier requires both a green suite and a clean tsc.

opencode2/2 passed

33.1 s

pi2/2 passed

28.5 s

Marshall · shell-first belt2/2 passed

25.7 s

Marshall · full belt2/2 passed

19.3 s

aider isn't plotted: it finished in 9.7s because it made no changes at all. Marshall's full belt is the fastest of everything that passed, and the cheapest at $0.0070 a run against pi's $0.0274.

aider does nothing at all

0/2, 5,400 input tokens, 9.7 seconds — and the fixture came back byte-identical, the same 2 of 248 tests failing. It is not running out of budget; it is declining to start. Its repo-map ships with the instruction "treat them as read-only — if you need to edit any of these files, ask me to add them to the chat first", because the map exists for a human to read and then name the files. Given --yes-always -m there is nobody to answer. On the two-file fixtures it worked because there was nothing to choose.

That is the whole of its 20× economy on small tasks: a person doing the file selection. It is also the sharpest illustration in this piece of what a harness is for. aider's design is good and it is well suited to a human at a keyboard. Measured as an autonomous agent on a 45-file repo, it scores zero, and no amount of speed makes that a pass.

And the shell-first belt loses to the full one

luna-shell-edit spends 146,104 input tokens and 25.7s against luna-openrouter's 81,633 and 19.3s, for the same 2/2. On a task where the work is finding three faults in 45 files, having a search tool is worth more than not having one. The belt that won the migration by 3× loses here, the first of the two task shapes in this piece where it does; the second is section 07.

The short tasks, as a control

Everything above is one long, mechanical task. That is the shape where harness overhead is large enough to measure — and also the shape a harness can be over-fitted to. These are the short ones: a one-line bug fix, adding a function, a rename across a few files, and an iterative fix. Same harnesses, same models, same verifiers. This is the control group, and it is where the piece's most useful negative result lives.

Across all four: aider-luna 8/9, luna-openrouter 8/8, luna-shell-edit 6/8, opencode-luna 8/8, pi-luna 8/8.

Marshall's full belt is the fastest of the three full agents here

Worth stating, because the migration task's noise makes it easy to miss: on the four short tasks Marshall's full belt was the quickest of Marshall, pi and opencode on three — bug-fix 11.0s against 12.4s and 20.9s, refactor 9.9s against 10.8s and 27.2s, iterate 21.3s against 24.7s and 27.0s — and tied pi exactly on the fourth at 10.0s. It went 8/8, the same as pi and opencode, and it did it in less time than either. aider is faster still on three of them, at 800–1,100 input tokens, for reasons section 06 makes clear.

This is where the shell-first belt stops looking good

aider-luna and luna-shell-edit are the only configurations to fail anything here, while Marshall's full belt, pi and opencode each went clean. The failures are not subtle: on tasks the full belt finishes in 10–21 seconds and under 1,000 output tokens, the shell-first belt twice ran to about 16,000 output tokens and truncated its own tool call mid-JSON, having made two calls total.

Taking read_file away removes the cheap way to look at a small file, and the model sometimes substitutes one enormous edit instead. It isn't reaching for the shell because the shell is better suited; it is reaching for it because it is the only thing left. That is a design fault, not a model fault, and it is the reason the belt that wins the migration by 3× is an ablation rather than a product: it optimises one task shape by removing a capability three others need.

What actually moved the needle

Starting from 91 tool calls at the worst (luna-incremental-prompt) and 5.7 at the best (luna-shell-edit), three changes account for almost all of it. Each one is now in the belt Marshall ships.

• 01Batching the file tools — edits[] on edit, patterns[] on search, paths[] on list. Real, but small. It targeted search and list_dir, which together were 10 of that config's 74 calls.
• 02Dropping the read-before-edit gate — larger. Marshall required a read_file before any edit; the oldString already has to match exactly once, so the gate bought no safety and cost a round trip per file. On GLM this took a run from 37 calls to 14.
• 03Pointing bulk reads at the shell — largest. One grep -rl PATTERN src | xargs cat replaces thirty read_file calls. This only became honest advice after the gate came out, since shell output is now enough to edit from. Together the two took a GLM run from 37 calls at 1/2 to 14 at 2/2.

Read those three in order and the shape of the lesson is visible. None of them made the model smarter. Two of them removed a rule the harness had imposed for its own convenience, and one of them told the model that a capability it already had — a shell — was the right tool for a job it was doing the slow way. The GLM numbers are the proof that this is about more than speed: the same model on the same fixture went from passing half its trials to passing all of them. A harness that gets in the way doesn't just cost money, it costs correctness.

The fourth result is the one that didn't make it into the product. Removing the read tools outright — luna-shell-edit, glm-shell-only — produced the leanest runs in the entire benchmark and then failed on two of the short tasks, spent 1.8× the tokens on repo-bugs, and went 1/2 on GLM. There is a version of minimalism that works and a version that amputates, and the difference between them is only visible if you test on more than one task shape. Marshall keeps read_file, search and list_dir, and tells the model when the shell is the better route. That combination is what the testing produced, and it is what ships.

What good means

Five properties, all measurable from the tables above. If you are choosing a harness rather than building one, this is the list I would work down.

Two of the five separate the field, and they are the two that don't show up in a screenshot of a successful run. Finishes on the weakest model, because a harness that only works on the frontier model is a demo and not infrastructure, and because Qwen is where two of the three harnesses that ran it scored zero. Holds across task shapes, because the migration task alone would have told us to ship a belt that breaks on a one-line fix.

pi fails one of the five and opencode fails one, and it is the same one for both. That is not a criticism of either; they are good harnesses, and on Luna with a strong model all three finish the same job successfully, with Marshall and pi inside 1.3 seconds of each other. With a capable model the harness mostly decides what the run costs. It decides whether you get a result at all only when the model is small — and that is the case most production agent work actually falls into.

What we did not close

pi is ahead on GLM-5.3 Flash and stayed ahead: 7 calls to Marshall's 10, 77.7s to 126.1s by median. We spent a day on the gap and closed most of it — 186.5s down to 126.1s came from the belt, not the model — and the rest is open. One model out of three.

The other limit is the one in section 04. Trial-to-trial noise on this fixture reaches 3.4×, which means several of the smaller differences in these tables are not findings. The ones we rely on are the Qwen ceiling, the cost gap, and the 16× spread between Marshall's own configurations — all far outside it.

Full trial logs, fixtures, and the harness code for this benchmark are on GitHub alongside Marshall.

github.com/agentionai/marshall →