I wrote two pieces arguing the harness is the bottleneck. This month the counter-evidence arrived, so I built a rig to test my own claim.
A few weeks ago I published “Harness Engineering Is the New Bottleneck, Not the Model.” A week later I followed it with a piece about running a harness eval loop on my own .NET agent. Both pieces leaned the same way: stop obsessing over which model is in the loop, look at what is wrapped around it.
I still think that was a useful correction at the time. But I wrote both pieces before reading the evidence I am going to walk through below, and some of it cuts against me. Not against “the harness matters” as a slogan, but against the way I let that slogan blur two different things together.
The two things are these. A harness piece can move the result : more tasks actually solved. Or it can move the cost : more tokens, more minutes, more money, for the same tasks solved. When I wrote “the harness is the bottleneck,” I was mostly thinking about the first. A lot of the new evidence says the second is far more common.
So this article does three things. It goes through the counter-evidence, carefully, with every number labeled by where it came from. It re-reads my own two pieces through that lens. And it builds an ablation rig in .NET, against a local model through Ollama or llama.cpp, where I can switch individual harness pieces off and see which ones move the result and which ones only move the bill.
The counter-evidence, as Jaroslaw Wasowski put it together
The piece that made me stop and re-read my own work was Jaroslaw Wasowski’s “Harness Engineering: When Agent Harness Moves the Result and When It Only Moves the Cost” on Level Up Coding. It is member-only, and I could read only the opening, so I went to his primary sources directly. Where my reading of a primary source differs from how his intro frames it, I say so.
His core move is simple and, I think, right: judge a harness on two axes, result and cost, and never infer one from the other. He also openly revises his own earlier view that the harness is a moat worth hiding behind. I appreciated that. It is easier to write the next article than to say the last one was partly wrong.
The METR note
METR published a research note in February 2026, “Measuring Time Horizon Using Claude Code and Codex.” The question was whether vendor harnesses beat METR’s own default scaffolds when measuring how long a task a model can complete. METR’s defaults are ReAct (a plain act and observe loop) and Triframe (advisor, actor and rater subagents choosing each action).
What the note reports, as I read it on METR’s site:
- Opus 4.5 with Claude Code beat Opus 4.5 with ReAct in 50.7% of bootstrap samples. That is a coin flip.
- GPT-5 with Codex beat GPT-5 with Triframe in only 14.5% of bootstrap samples. Not statistically significant either way.
- Raising the Opus 4.5 token budget from 8M to 32M for the Claude Code runs barely moved the preliminary time horizon.
METR is careful to call it a research note, not a vetted publication. It covers two models. The vendor harnesses are built for interactive use with a human, and the evaluation was fully autonomous, which METR flags itself.
One discrepancy I want to be upfront about. Wasowski’s intro puts a token gap of up to 40x next to the METR finding. When I read the METR note, I could not find a 40x figure in it. The 40x I could trace comes from the next paper. His full text may attribute it precisely and I only saw the opening, so I am not calling it an error. I am telling you where I found the number.
The Scaffold Effect
“The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation” by Naman Vats and Oleg Golev (Sentient Labs), arXiv 2607.22585, preliminary work under review at the DL4C workshop at ICML 2026. This one I read in full, and it is the cleanest experiment in this whole debate.
Setup: three harnesses (Goose, OpenCode, OpenHands-SDK), two models (Qwen 3.6 Plus and MiniMax M2.5, through OpenRouter), 50 tasks sampled from Terminal-Bench Pro across eight categories, 300 trials in total. Same sandbox, a 900 second wall-time cap.
Findings, from the paper:
- Pass rates sat between 38% and 50%. Within one model, the spread across harnesses was 2 points for Qwen and 8 points for MiniMax. Most pairwise differences were not distinguishable from zero.
- Tokens per solved task differed by roughly 40x between harnesses. Goose used around 28K (Qwen) and 37K (MiniMax) tokens per solve. OpenCode used around 1.15M and 1.55M.
- Upgrading the model changed token cost by only 1.0x to 1.3x. The harness moved the bill far more than the model did.
That is the “0 to 8 point band” and the “40x” in one table. Same tasks, nearly the same outcomes, wildly different bills.
Two details in the paper matter more to me than the headline, though. First, the failure fingerprints were properties of the harness, consistent across both models: Goose tends to stop when stuck, OpenHands-SDK pushes toward closure and sometimes submits plausible wrong answers, OpenCode tends to run into the time cap. Second, on the IMPL category, OpenHands-SDK solved 69% against 38% for the other two. So the averaged band hides at least one place where the harness did move the result, on one kind of work.
The authors are honest about scale: with 50 tasks, every task is worth 2 points, and they read their pass-rate differences as descriptive. Keep that sentence in mind. It is going to come back to bite my own rig.
DeepSWE and the fixed harness
DeepSWE (Datacurve, arXiv 2607.07946) is a benchmark of 113 long-horizon tasks across 91 open-source repositories. The design choice relevant here: every model is evaluated inside the same mini-swe-agent harness, with the same bash tool and the same prompt, explicitly so the leaderboard reflects model capability rather than scaffolding. They acknowledge the trade: real developers use Codex CLI or Claude Code, not mini-swe-agent.
I find that choice telling. People who build benchmarks for a living decided the harness is enough of a confounder that it has to be pinned to a deliberately minimal one. That is not the same as “the harness does not matter.” It is “the harness matters enough that we have to hold it still to see anything else.”
Model-harness fit
Wasowski’s companion piece, “Harness Engineering: Model-Harness Fit: Why a Better Teacher Makes a Worse Agent,” summarizes a Salesforce preprint he cites as arXiv 2609.09134. I could not locate that preprint myself, so treat every number in this paragraph as secondhand, through his summary. As he describes it: a harness built around a weaker model lifted its success on seven agentic tasks from 29.2% to 78.0% without touching the weights. Fine-tuning the same model on a stronger expert’s full trajectories dropped it to 63.1%. Correcting single bad turns instead, one per failed rollout, brought it to 79.7%.
A separate ICML 2026 paper I could read, “The Interplay of Harness Design and Post-Training in LLM Agents” by Kyungmin Kim and co-authors, reaches a compatible conclusion on an ALFWorld-based benchmark: harness and post-training cannot be separated, a harness bolted on after training recovers little, and for the smaller model the harness was a prerequisite for post-training to help at all.
Put those two next to the Scaffold Effect and a picture forms that is more interesting than either camp. Swapping one reasonable harness for another, on a capable model, mostly moves cost. Building a harness that fits a specific, weaker model can move the result enormously. Those are different claims, and I had been making the second while citing evidence for the first.
Two practitioners who measured instead of arguing
Two pieces from the same weeks did something I respect more than any thread: they measured their own setup and reported where it disappointed them. Both are member-only, I could read only their introductions, and every figure below is author-reported from their own runs.
Fareed Khan, “Fixing Over-Engineering in LLM Coding Agents.” He built what he calls a proportional-code system around Qwen3.8–27B, aimed at agents that write far more code than a ticket needs. His opening example is a request that needed one database constraint and got 56 new lines, including a DELETE that silently removes members. His numbers: across 34 trap tickets, an unguided agent's passing answers added a median of 1.5x the reference code (the intro calls it "typical"). With his full system, on 25 held-out tickets, that dropped to 0.88x. The part I keep thinking about: of the 11 pieces he designed, only 4 shipped, and his first hypothesis, H1, missed.
Seven of eleven pieces did not earn their place. That is not a failure of his project. That is what an honest ablation looks like when you run one.
Pasquale Molinaro, “I Had a Free Qwen3.8–27B Agent Copy 10 Web Pages. It Spent 64% of Its Time Measuring Pixels.” Qwen3.8–27B at Q4_K_M on two free Kaggle T4s, rebuilding ten pages from the Design2Code benchmark under three conditions: a one-shot copy, a one-shot copy plus one revision, and an agent that starts from the one-shot page with a renderer and 20 minutes per page. He expected the agent, the only condition that could look at its own output, to win clearly. His numbers: the agent issued 237 measuring commands, and 64% of its 169 minutes went to measuring pixels. Most of the score gain traced back to a single page, through a change you cannot even see in a browser.
That second point is the one that scared me. An aggregate score went up, and the honest explanation was “one task, invisible change.” If he had stopped at the aggregate, he would have published a harness win.
Re-reading my own two pieces
So, what does this do to what I wrote?
In “Harness Engineering Is the New Bottleneck,” my strongest piece of evidence was LangChain moving a fixed model from 52.8% to 66.5% on Terminal-Bench 2.0 by rebuilding the harness. I did flag that it was one receipt, not two. What I did not do was ask what kind of harness change it was. In light of the fit papers, that result reads less like “harnesses beat models” and more like “one team fitted a harness to one model and one benchmark.” Real, but narrower than the headline I hung on it.
In the eval loop piece, I actually had both axes in my own data and did not name them. The semantic search tool kept the pass rate flat and cut tokens by 41%. That was a pure cost move, and I kept it for exactly that reason. The self-review step moved the pass rate by 13 points and held at 68% on a rerun. That looked like a result move. But I scored 12 issues, I did not pair the comparison by issue, and with 12 issues each one is worth about 8 points. I would still bet the review step is real, because it gives the model information it did not have before (test output plus a fresh look at the diff). I would no longer state it as confidently as I did.
My revised position, before any new data: harness pieces reliably move cost and only conditionally move the result. My working guess about the condition is that a piece moves the result when it feeds the model a signal it could not get otherwise (a compiler error, a failing check, a rendered page), and mostly moves cost when it adds ceremony (a plan, another loop, more room to wander). That guess is testable, so I built something to test it.
The rig
I wanted the smallest thing that could tell the two axes apart. Here is what I settled on and why.
A local model. The fit evidence says harnesses matter most for weaker models, and both practitioners above ran a 27B local model. If harness pieces only move cost on a small local model, the strongest version of my original claim is in real trouble. Local also means I can run a few hundred configurations without watching a bill.
No SDK. The rig talks to any OpenAI-compatible /chat/completions endpoint with a plain HttpClient. Ollama exposes one at http://localhost:11434/v1, llama.cpp's llama-server at http://localhost:8080/v1. No NuGet packages at all, so between configurations the only thing that changes is the harness, not a library version.
Eight small tasks with traps. Each task is a C# function or class to implement. Each has visible checks the agent can run and hidden checks used only for scoring. The hidden checks target the edge cases a quick implementation misses: overflow, culture-sensitive lowercasing, SemVer pre-release ordering, a Turkish dotted capital I. A selftest command proves every reference solution passes all hidden checks and every starter fails them.
Four switchable pieces. A planner, a verifier tool, a harness-driven retry loop, and a tool budget.
Ten configurations. “Add one piece to bare” and “remove one piece from full.” I need both directions, because pieces overlap. If the verifier and the retry loop both feed check output back to the model, removing either one from the full harness might show nothing, while adding either one to the bare harness shows a lot. Only one direction would mislead me about which pieces are redundant.
Two cost metrics, not one. Tokens per run and tokens per solve, the metric the Scaffold Effect paper uses. Plus wall time, and the share of wall time spent in builds and checks, because Molinaro’s 64% made me want to know where the minutes go, not just how many there are.
Explicit outcomes. Dragos Bilaniuc’s “Reliability at the Core: Five Ground Rules for Building an Agent Loop” shaped the loop: every way a run can end is a named terminal (completed, budget_exhausted, turn_limit, model_error), the tool budget is checked before a tool fires rather than after, and a failed model call never leaves a half-written turn in the history.
+-----------------+---------+----------+-------+--------+---------------------------------+
| Config | Planner | Verifier | Retry | Budget | Question it answers |
+-----------------+---------+----------+-------+--------+---------------------------------+
| bare | off | off | 0 | 6 | Baseline |
| +plan | on | off | 0 | 6 | Does a plan alone help? |
| +verify | off | on | 0 | 6 | Does the agent use checks well? |
| +retry | off | off | 2 | 6 | Does harness feedback help? |
| +budget | off | off | 0 | 20 | Does more room help? |
| full | on | on | 2 | 20 | Everything |
| full-no-plan | off | on | 2 | 20 | Is the plan carrying weight? |
| full-no-verify | on | off | 2 | 20 | Is the verifier redundant? |
| full-no-retry | on | on | 0 | 20 | Is the retry redundant? |
| full-tight | on | on | 2 | 6 | Do checks eat the budget? |
+-----------------+---------+----------+-------+--------+---------------------------------+
And the tasks, with what the hidden checks go after:
+--------------+------------------------------+------------------------------------------+
| Task | What the agent implements | What the hidden checks target |
+--------------+------------------------------+------------------------------------------+
| 01-duration | ParseDuration("1h30m") | Case, whitespace, repeated/out-of-order |
| | | units, unit without a number |
| 02-slugify | Slugify(text) | Diacritics, "İstanbul", only separators |
| 03-intervals | Merge(intervals) | Touching intervals, unsorted, empty |
| 04-semver | Compare(a, b) | Pre-release precedence, build metadata |
| 05-csv | SplitLine(line) | Escaped quotes, trailing comma, no trim |
| 06-lru | LruCache<TKey, TValue> | Get and update refresh recency, cap 1 |
| 07-backoff | Delays(attempts, base, max) | Cap, zero, negative, overflow at 100 |
| 08-roman | ToRoman(value) | 3999, 944, out-of-range throws |
+--------------+------------------------------+------------------------------------------+
Setting it up
You need the .NET 10 SDK and a local model server. On my Mac:
brew install --cask dotnet-sdk
brew install ollama
# The default context window has been small in the Ollama versions I have used.
# A long agent history gets truncated silently, and that hits "full" harder than "bare".
OLLAMA_CONTEXT_LENGTH=16384 ollama serve
In a second terminal:
ollama pull qwen2.5-coder:7b
If you prefer containers, the Docker route works too. On a Mac, Docker cannot use the GPU, so expect it to be much slower than native Ollama:
docker run -d --name ollama -p 11434:11434 -e OLLAMA_CONTEXT_LENGTH=16384 \
-v ollama:/root/.ollama ollama/ollama
docker exec ollama ollama pull qwen2.5-coder:7b
And the llama.cpp route, which is what I would use to match the GGUF and quantization exactly:
brew install llama.cpp
llama-server -hf Qwen/Qwen2.5-Coder-7B-Instruct-GGUF:Q4_K_M --jinja -c 16384 --port 8080
--jinja matters: without the chat template, tool calls come back as plain text and the rig counts them as fallback writes instead of tool calls.
Any model with tool calling works. A model without it makes Ollama reject the request, and those runs end as model_error, which the report excludes and lists.
The project layout:
harness-ablation/
Ablation.csproj
checklib/Check.cs
src/HarnessConfig.cs
src/TaskSpec.cs
src/ChatClient.cs
src/AgentRunner.cs
src/Report.cs
src/Program.cs
tasks/01-duration.task ... tasks/08-roman.task
The code
Ablation.csproj
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<RootNamespace>Ablation</RootNamespace>
<InvariantGlobalization>true</InvariantGlobalization>
</PropertyGroup>
<!-- Task fixtures and the check helper are data for the rig, not part of its own build. -->
<ItemGroup>
<Compile Remove="tasks/ **;checklib/** ;runs/ **;results/**" />
<None Include="checklib/Check.cs" CopyToOutputDirectory="PreserveNewest" />
</ItemGroup>
</Project>
HarnessConfig.cs: the switches
namespace Ablation;
/// <summary>
/// One harness configuration. Every switch is one piece I can remove on its own.
/// </summary>
/// <param name="Planner">Extra model call that writes a plan before the agent acts.</param>
/// <param name="VerifierTool">The agent may call run_checks (build + visible checks) itself.</param>
/// <param name="Retries">After the agent stops, the harness runs the visible checks and, if they fail,
/// feeds the output back for up to this many more attempts.</param>
/// <param name="ToolBudget">Hard cap on tool calls for the whole run, checked before each call.</param>
public sealed record HarnessConfig(string Name, bool Planner, bool VerifierTool, int Retries, int ToolBudget)
{
public const int TightBudget = 6;
public const int LooseBudget = 20;
// "Add one piece to bare" and "remove one piece from full". Same tasks, same model, same seed.
public static readonly IReadOnlyList<HarnessConfig> All =
[
new("bare", Planner: false, VerifierTool: false, Retries: 0, ToolBudget: TightBudget),
new("+plan", Planner: true, VerifierTool: false, Retries: 0, ToolBudget: TightBudget),
new("+verify", Planner: false, VerifierTool: true, Retries: 0, ToolBudget: TightBudget),
new("+retry", Planner: false, VerifierTool: false, Retries: 2, ToolBudget: TightBudget),
new("+budget", Planner: false, VerifierTool: false, Retries: 0, ToolBudget: LooseBudget),
new("full", Planner: true, VerifierTool: true, Retries: 2, ToolBudget: LooseBudget),
new("full-no-plan", Planner: false, VerifierTool: true, Retries: 2, ToolBudget: LooseBudget),
new("full-no-verify", Planner: true, VerifierTool: false, Retries: 2, ToolBudget: LooseBudget),
new("full-no-retry", Planner: true, VerifierTool: true, Retries: 0, ToolBudget: LooseBudget),
new("full-tight", Planner: true, VerifierTool: true, Retries: 2, ToolBudget: TightBudget),
];
public static HarnessConfig Get(string name) =>
All.FirstOrDefault(c => c.Name.Equals(name, StringComparison.OrdinalIgnoreCase))
?? throw new ArgumentException($"Unknown config '{name}'. Known: {string.Join(", ", All.Select(c => c.Name))}");
}
The difference between the verifier and the retry is deliberate. With the verifier, the model decides when to look at the checks. With the retry, the harness decides, after the model says it is done. Those are different bets about who should own verification, and I wanted to price them separately.
A task file
Each task is one text file with five sections. Here is the one with my favorite trap. A naive baseDelay * Math.Pow(2, i) overflows TimeSpan well before attempt 100.
== prompt ==
Implement `Backoff.Delays(int attempts, TimeSpan baseDelay, TimeSpan max)` in Solution.cs.
Return one delay per retry attempt: attempt i (starting at 0) waits baseDelay * 2^i, but never more than max.
attempts == 0 returns an empty list. A negative attempts value throws ArgumentOutOfRangeException.
It must work for large attempt counts (for example 100) without overflowing; once the cap is reached,
every later delay equals max.
== starter ==
using System;
using System.Collections.Generic;
public static class Backoff
{
public static IReadOnlyList<TimeSpan> Delays(int attempts, TimeSpan baseDelay, TimeSpan max)
{
throw new NotImplementedException();
}
}
== visible ==
Check.Run("doubling", () => Check.Seq(new[] { 100.0, 200.0, 400.0 }, Backoff.Delays(3, TimeSpan.FromMilliseconds(100), TimeSpan.FromSeconds(10)).Select(d => d.TotalMilliseconds)));
== hidden ==
Check.Run("cap", () => Check.Seq(new[] { 1.0, 2.0, 3.0, 3.0 }, Backoff.Delays(4, TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(3)).Select(d => d.TotalSeconds)));
Check.Run("zero attempts", () => Check.Eq(0, Backoff.Delays(0, TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(3)).Count));
Check.Run("negative throws", () => Check.Throws<ArgumentOutOfRangeException>(() => Backoff.Delays(-1, TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(3))));
Check.Run("no overflow", () => { var d = Backoff.Delays(100, TimeSpan.FromMilliseconds(250), TimeSpan.FromSeconds(30)); Check.True(d.Count == 100 && d.All(x => x > TimeSpan.Zero && x <= TimeSpan.FromSeconds(30)) && d[^1] == TimeSpan.FromSeconds(30)); });
== reference ==
using System;
using System.Collections.Generic;
public static class Backoff
{
public static IReadOnlyList<TimeSpan> Delays(int attempts, TimeSpan baseDelay, TimeSpan max)
{
if (attempts < 0) throw new ArgumentOutOfRangeException(nameof(attempts));
var list = new List<TimeSpan>(attempts);
var current = baseDelay;
for (int i = 0; i < attempts; i++)
{
list.Add(current < max ? current : max);
if (current < max) current = current.Ticks > max.Ticks / 2 ? max : current + current;
}
return list;
}
}
The other seven follow the same shape and are in the appendix at the end. If you write your own, keep the visible checks thin on purpose. The gap between visible and hidden is where a verifier can make the agent look better than it is, and that gap is exactly what I want to measure.
checklib/Check.cs: assertions without a test framework
Every check run compiles Solution.cs, this helper and a generated Main. No xUnit, no restore from a feed.
using System;
using System.Collections.Generic;
using System.Linq;
// Tiny assertion helper compiled into every check run. No test framework, no NuGet.
public static class Check
{
private static int _passed, _failed;
public static void Run(string name, Action body)
{
try
{
body();
_passed++;
Console.WriteLine($"PASS {name}");
}
catch (Exception ex)
{
_failed++;
var msg = ex.Message.Length > 200 ? ex.Message[..200] + "..." : ex.Message;
Console.WriteLine($"FAIL {name}: {ex.GetType().Name}: {msg}");
}
}
public static void Eq<T>(T expected, T actual)
{
if (!EqualityComparer<T>.Default.Equals(expected, actual))
throw new CheckFailed($"expected <{expected}> but got <{actual}>");
}
public static void Seq<T>(IEnumerable<T> expected, IEnumerable<T>? actual)
{
var e = expected.ToList();
var a = actual?.ToList() ?? throw new CheckFailed("got null");
if (!e.SequenceEqual(a))
throw new CheckFailed($"expected [{string.Join(", ", e)}] but got [{string.Join(", ", a)}]");
}
public static void True(bool condition, string? what = null)
{
if (!condition) throw new CheckFailed(what ?? "condition was false");
}
public static void Throws<TEx>(Action action) where TEx : Exception
{
try { action(); }
catch (TEx) { return; }
catch (Exception ex) { throw new CheckFailed($"expected {typeof(TEx).Name} but got {ex.GetType().Name}"); }
throw new CheckFailed($"expected {typeof(TEx).Name} but nothing was thrown");
}
public static int Done()
{
Console.WriteLine($"{_passed} passed, {_failed} failed");
return _failed == 0 ? 0 : 1;
}
}
public sealed class CheckFailed(string message) : Exception(message);
TaskSpec.cs: loading tasks and running checks
using System.Diagnostics;
using System.Text;
namespace Ablation;
/// <summary>A task file has five sections: prompt, starter, visible, hidden, reference.</summary>
public sealed record TaskSpec(string Id, string Prompt, string Starter, string Visible, string Hidden, string Reference)
{
public static IReadOnlyList<TaskSpec> LoadAll(string dir, IEnumerable<string>? only = null)
{
var filter = only?.ToHashSet(StringComparer.OrdinalIgnoreCase);
return Directory.GetFiles(dir, "*.task")
.OrderBy(f => f, StringComparer.Ordinal)
.Select(Load)
.Where(t => filter is null || filter.Any(f => t.Id.StartsWith(f, StringComparison.OrdinalIgnoreCase)))
.ToList();
}
public static TaskSpec Load(string path)
{
var sections = new Dictionary<string, StringBuilder>();
StringBuilder? current = null;
foreach (var line in File.ReadAllLines(path))
{
var t = line.Trim();
if (t.StartsWith("== ") && t.EndsWith(" ==")) { current = sections[t[3..^3]] = new StringBuilder(); continue; }
current?.AppendLine(line);
}
string S(string k) => sections.TryGetValue(k, out var sb) ? sb.ToString().Trim() : throw new InvalidDataException($"{path}: missing section '{k}'");
return new(Path.GetFileNameWithoutExtension(path), S("prompt"), S("starter"), S("visible"), S("hidden"), S("reference"));
}
}
public sealed record CheckResult(bool Passed, int ChecksPassed, int ChecksTotal, string Output, long ElapsedMs);
/// <summary>Builds a throwaway console project around Solution.cs and runs a set of checks against it.</summary>
public static class CheckRunner
{
private static readonly string Tfm = $"net{Environment.Version.Major}.0";
public static void WriteProject(string dir, string checkLines)
{
Directory.CreateDirectory(dir);
File.WriteAllText(Path.Combine(dir, "Checks.csproj"), $"""
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>{Tfm}</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>disable</ImplicitUsings>
<NoWarn>$(NoWarn);CS8600;CS8601;CS8602;CS8603;CS8604;CS8618;CS8625;CS0168;CS0219</NoWarn>
</PropertyGroup>
</Project>
""");
File.Copy(Path.Combine(AppContext.BaseDirectory, "checklib", "Check.cs"), Path.Combine(dir, "Check.cs"), overwrite: true);
File.WriteAllText(Path.Combine(dir, "Checks.cs"), $$"""
using System;
using System.Collections.Generic;
using System.Linq;
public static class Program
{
public static int Main()
{
{{checkLines}}
return Check.Done();
}
}
""");
}
public static async Task<CheckResult> RunAsync(string dir, TimeSpan timeout, CancellationToken ct = default)
{
var sw = Stopwatch.StartNew();
var build = await Exec("dotnet", $"build \"{dir}\" -c Release -o \"{Path.Combine(dir, "out")}\" --nologo -v q -clp:NoSummary", timeout, ct);
if (build.ExitCode != 0)
return new(false, 0, 0, "BUILD FAILED\n" + Trim(Errors(build.Output)), sw.ElapsedMilliseconds);
var run = await Exec("dotnet", $"\"{Path.Combine(dir, "out", "Checks.dll")}\"", timeout, ct);
var lines = run.Output.Split('\n');
int passed = lines.Count(l => l.StartsWith("PASS ")), failed = lines.Count(l => l.StartsWith("FAIL "));
var output = run.TimedOut ? "TIMED OUT (infinite loop?)\n" + run.Output : run.Output;
return new(run.ExitCode == 0 && !run.TimedOut, passed, passed + failed, Trim(output), sw.ElapsedMilliseconds);
}
// Compiler output repeats every error; keep the distinct error lines, which is what the model needs.
private static string Errors(string output)
{
var errors = output.Split('\n').Where(l => l.Contains("error", StringComparison.OrdinalIgnoreCase))
.Select(l => l.Trim()).Distinct().ToList();
return errors.Count > 0 ? string.Join('\n', errors) : output;
}
private static string Trim(string s) => s.Length <= 3000 ? s.Trim() : s[..3000] + "\n...(truncated)";
private sealed record ExecResult(int ExitCode, string Output, bool TimedOut);
private static async Task<ExecResult> Exec(string file, string args, TimeSpan timeout, CancellationToken ct)
{
var psi = new ProcessStartInfo(file, args) { RedirectStandardOutput = true, RedirectStandardError = true };
psi.Environment["DOTNET_CLI_TELEMETRY_OPTOUT"] = "1";
psi.Environment["DOTNET_NOLOGO"] = "1";
psi.Environment["MSBUILDDISABLENODEREUSE"] = "1";
using var p = Process.Start(psi)!;
var stdout = p.StandardOutput.ReadToEndAsync(ct);
var stderr = p.StandardError.ReadToEndAsync(ct);
using var cts = CancellationTokenSource.CreateLinkedTokenSource(ct);
cts.CancelAfter(timeout);
try { await p.WaitForExitAsync(cts.Token); }
catch (OperationCanceledException) when (!ct.IsCancellationRequested)
{
p.Kill(entireProcessTree: true);
return new(-1, await stdout + await stderr, TimedOut: true);
}
return new(p.ExitCode, await stdout + await stderr, TimedOut: false);
}
}
A build plus check run takes a couple of seconds, which is usually small next to a model call on a laptop GPU. That ratio is not universal. On a fast model server, the checks can become the dominant cost, which is exactly why the report shows the share of time spent in them.
ChatClient.cs: one HTTP call, no SDK
using System.Diagnostics;
using System.Text;
using System.Text.Json.Nodes;
namespace Ablation;
public sealed record ToolCall(string Id, string Name, JsonObject Arguments);
public sealed record ChatTurn(JsonObject AssistantMessage, string? Content, IReadOnlyList<ToolCall> ToolCalls,
int PromptTokens, int CompletionTokens, bool UsageMissing, long ElapsedMs);
/// <summary>
/// Minimal client for any OpenAI-compatible /chat/completions endpoint:
/// Ollama (http://localhost:11434/v1) or llama.cpp's llama-server (http://localhost:8080/v1).
/// No SDK on purpose, so the only thing that changes between configurations is the harness.
/// </summary>
public sealed class ChatClient(HttpClient http, string baseUrl, string model, double temperature, int seed)
{
public string Model => model;
public async Task<ChatTurn> CompleteAsync(JsonArray messages, JsonArray? tools, CancellationToken ct)
{
var body = new JsonObject
{
["model"] = model,
["messages"] = messages.DeepClone(),
["temperature"] = temperature,
["seed"] = seed,
["stream"] = false,
};
if (tools is { Count: > 0 }) body["tools"] = tools.DeepClone();
var sw = Stopwatch.StartNew();
var json = await PostWithTransportRetryAsync(body.ToJsonString(), ct);
var message = json["choices"]![0]!["message"]!.AsObject();
var calls = new List<ToolCall>();
if (message["tool_calls"] is JsonArray arr)
{
foreach (var c in arr)
{
var fn = c!["function"]!;
// Ollama and llama-server send arguments as a JSON string; some servers send an object.
var rawArgs = fn["arguments"];
var args = rawArgs switch
{
JsonObject o => o,
JsonValue v when v.TryGetValue<string>(out var s) && !string.IsNullOrWhiteSpace(s) => TryParseObject(s),
_ => new JsonObject(),
};
calls.Add(new(c["id"]?.GetValue<string>() ?? Guid.NewGuid().ToString("n")[..8], fn["name"]!.GetValue<string>(), args));
}
}
var usage = json["usage"];
int pt = usage?["prompt_tokens"]?.GetValue<int>() ?? 0, ctok = usage?["completion_tokens"]?.GetValue<int>() ?? 0;
var assistant = new JsonObject { ["role"] = "assistant", ["content"] = message["content"]?.DeepClone() };
if (message["tool_calls"] is JsonArray tc) assistant["tool_calls"] = tc.DeepClone();
return new(assistant, message["content"]?.GetValue<string>(), calls, pt, ctok, usage is null, sw.ElapsedMilliseconds);
}
// Transport failures (dropped connection, reset) are retried twice. Nothing has been added to the
// conversation yet, so a retry cannot duplicate a message. HTTP error statuses are not retried.
private async Task<JsonObject> PostWithTransportRetryAsync(string payload, CancellationToken ct)
{
for (int attempt = 0; ; attempt++)
{
try
{
// Buffered body with Content-Length. PostAsJsonAsync streams a chunked body,
// which some local servers and proxies reject.
using var content = new StringContent(payload, Encoding.UTF8, "application/json");
using var resp = await http.PostAsync($"{baseUrl.TrimEnd('/')}/chat/completions", content, ct);
var text = await resp.Content.ReadAsStringAsync(ct);
if (!resp.IsSuccessStatusCode)
throw new InvalidOperationException($"HTTP {(int)resp.StatusCode}: {text[..Math.Min(text.Length, 500)]}");
return JsonNode.Parse(text)!.AsObject();
}
catch (HttpRequestException) when (attempt < 2 && !ct.IsCancellationRequested)
{
TransportRetries++;
await Task.Delay(500 * (attempt + 1), ct);
}
}
}
public int TransportRetries { get; private set; }
private static JsonObject TryParseObject(string s)
{
try { return JsonNode.Parse(s) as JsonObject ?? new JsonObject(); }
catch { return new JsonObject { ["_unparsed"] = s }; }
}
}
Two comments in there are scars, and I come back to them below.
AgentRunner.cs: the loop
This is the part that matters. Every harness piece is an if on the config, so turning one off removes exactly that piece and nothing else.
using System.Diagnostics;
using System.Text.Json.Nodes;
using System.Text.RegularExpressions;
namespace Ablation;
public sealed record RunRecord(
string Config, string Task, int Rep, string Model, DateTime StartedUtc,
bool Passed, int HiddenChecksPassed, int HiddenChecksTotal, string Outcome,
int PromptTokens, int CompletionTokens, int ModelCalls, int ToolCalls, int VerifierCalls,
int RetriesUsed, int FallbackWrites, int UsageMissing,
long ModelMs, long CheckMs, long TotalMs)
{
public int Tokens => PromptTokens + CompletionTokens;
}
/// <summary>
/// The agent loop. Outcomes are explicit terminals (completed, budget_exhausted, turn_limit, model_error),
/// never exceptions, so every run produces one record.
/// </summary>
public sealed partial class AgentRunner(ChatClient chat, string runsRoot, TimeSpan checkTimeout)
{
private const string SolutionFile = "Solution.cs";
public async Task<RunRecord> RunAsync(TaskSpec task, HarnessConfig cfg, int rep, CancellationToken ct)
{
var started = DateTime.UtcNow;
var total = Stopwatch.StartNew();
var dir = Path.Combine(runsRoot, cfg.Name.Replace('+', 'p'), task.Id, $"rep{rep}");
if (Directory.Exists(dir)) Directory.Delete(dir, recursive: true);
var work = Path.Combine(dir, "work"); // the agent's workspace: visible checks only
var hidden = Path.Combine(dir, "hidden"); // created after the agent stops; the agent never sees it
CheckRunner.WriteProject(work, task.Visible);
File.WriteAllText(Path.Combine(work, SolutionFile), task.Starter);
File.WriteAllText(Path.Combine(work, "TASK.md"), task.Prompt);
int pt = 0, ctok = 0, modelCalls = 0, toolCalls = 0, verifierCalls = 0, retriesUsed = 0, fallbacks = 0, usageMissing = 0;
long modelMs = 0, checkMs = 0;
string outcome = "completed";
void Account(ChatTurn t) { pt += t.PromptTokens; ctok += t.CompletionTokens; modelMs += t.ElapsedMs; modelCalls++; if (t.UsageMissing) usageMissing++; }
var messages = new JsonArray
{
Msg("system", SystemPrompt(cfg)),
Msg("user", $"Task:\n{task.Prompt}\n\nCurrent {SolutionFile}:\n```
csharp\n{task.Starter}\n
```"),
};
var tools = Tools(cfg);
try
{
if (cfg.Planner)
{
var plan = await chat.CompleteAsync(new JsonArray
{
Msg("system", "You plan small C# coding tasks. Write a numbered plan of at most 6 steps, naming every edge case the implementation must handle. Do not write code."),
Msg("user", task.Prompt),
}, null, ct);
Account(plan);
messages.Add(Msg("user", $"Plan to follow:\n{plan.Content}"));
}
for (int attempt = 0; attempt <= cfg.Retries; attempt++)
{
// Inner loop: model turns until the model stops calling tools or a limit is hit.
int turns = 0;
while (true)
{
if (++turns > cfg.ToolBudget + 4) { outcome = "turn_limit"; break; }
var turn = await chat.CompleteAsync(messages, tools, ct);
Account(turn);
messages.Add(turn.AssistantMessage);
if (turn.ToolCalls.Count == 0)
{
// Small local models often answer with a code block instead of a tool call.
// Same lenient rule in every configuration, and it is counted.
if (ExtractCode(turn.Content) is { } code) { File.WriteAllText(Path.Combine(work, SolutionFile), code); fallbacks++; }
break;
}
foreach (var call in turn.ToolCalls)
{
// Check the budget BEFORE firing the tool, not after.
if (toolCalls >= cfg.ToolBudget)
{
messages.Add(ToolMsg(call.Id, "Tool budget exhausted. Stop and reply without tool calls."));
outcome = "budget_exhausted";
continue;
}
toolCalls++;
var (result, ms, isVerifier) = await ExecuteToolAsync(call, work, ct);
checkMs += ms;
if (isVerifier) verifierCalls++;
messages.Add(ToolMsg(call.Id, result));
}
if (outcome == "budget_exhausted") break;
}
if (attempt == cfg.Retries || outcome is "budget_exhausted" or "turn_limit") break;
// Harness-driven retry: the harness, not the model, decides to look at the visible checks.
var visible = await CheckRunner.RunAsync(work, checkTimeout, ct);
checkMs += visible.ElapsedMs;
if (visible.Passed) break;
retriesUsed++;
messages.Add(Msg("user", $"The harness ran the visible checks and they failed:\n{visible.Output}\n\nFix {SolutionFile}."));
}
}
catch (Exception ex) when (ex is HttpRequestException or TaskCanceledException or InvalidOperationException or NullReferenceException)
{
outcome = "model_error";
Console.Error.WriteLine($" [{cfg.Name} {task.Id} rep{rep}] model error: {ex.Message}");
}
total.Stop();
// Scoring happens outside the measured time and outside the agent's workspace.
CheckRunner.WriteProject(hidden, task.Visible + "\n" + task.Hidden);
File.Copy(Path.Combine(work, SolutionFile), Path.Combine(hidden, SolutionFile), overwrite: true);
var score = await CheckRunner.RunAsync(hidden, checkTimeout, ct);
File.WriteAllText(Path.Combine(dir, "transcript.json"), messages.ToJsonString(new() { WriteIndented = true }));
File.WriteAllText(Path.Combine(dir, "hidden-output.txt"), score.Output);
return new(cfg.Name, task.Id, rep, chat.Model, started, score.Passed, score.ChecksPassed, score.ChecksTotal, outcome,
pt, ctok, modelCalls, toolCalls, verifierCalls, retriesUsed, fallbacks, usageMissing,
modelMs, checkMs, total.ElapsedMilliseconds);
}
private async Task<(string Result, long Ms, bool IsVerifier)> ExecuteToolAsync(ToolCall call, string work, CancellationToken ct)
{
switch (call.Name)
{
case "read_file":
{
var name = Path.GetFileName(call.Arguments["path"]?.GetValue<string>() ?? SolutionFile);
var path = Path.Combine(work, name);
return (File.Exists(path) && name is SolutionFile or "TASK.md" or "Checks.cs" ? File.ReadAllText(path) : $"Cannot read '{name}'. Readable: {SolutionFile}, TASK.md, Checks.cs", 0, false);
}
case "write_file":
{
var content = call.Arguments["content"]?.GetValue<string>();
if (string.IsNullOrWhiteSpace(content)) return ("write_file needs a non-empty 'content' with the full file.", 0, false);
File.WriteAllText(Path.Combine(work, SolutionFile), content);
return ($"Wrote {SolutionFile} ({content.Length} chars).", 0, false);
}
case "run_checks":
{
var r = await CheckRunner.RunAsync(work, checkTimeout, ct);
return ((r.Passed ? "ALL VISIBLE CHECKS PASSED\n" : "") + r.Output, r.ElapsedMs, true);
}
default:
return ($"Unknown tool '{call.Name}'.", 0, false);
}
}
private static string SystemPrompt(HarnessConfig cfg) =>
"You are a careful C# developer working in a tiny workspace. " +
$"{SolutionFile} is the only file you may change. Use write_file to save the COMPLETE new contents of {SolutionFile}. " +
(cfg.VerifierTool ? "Use run_checks to compile the project and run the visible checks. " : "") +
"When you are finished, reply with one short sentence and no tool call. Do not add features nobody asked for.";
private static JsonArray Tools(HarnessConfig cfg)
{
var tools = new JsonArray
{
Tool("read_file", "Read a workspace file (Solution.cs, TASK.md or Checks.cs).",
new JsonObject { ["path"] = new JsonObject { ["type"] = "string" } }, ["path"]),
Tool("write_file", "Replace the full contents of Solution.cs.",
new JsonObject { ["content"] = new JsonObject { ["type"] = "string", ["description"] = "Complete C# source of Solution.cs" } }, ["content"]),
};
if (cfg.VerifierTool)
tools.Add(Tool("run_checks", "Build Solution.cs and run the visible checks. Returns compiler errors or PASS/FAIL lines.", new JsonObject(), []));
return tools;
}
private static JsonObject Tool(string name, string description, JsonObject props, string[] required) => new()
{
["type"] = "function",
["function"] = new JsonObject
{
["name"] = name,
["description"] = description,
["parameters"] = new JsonObject
{
["type"] = "object",
["properties"] = props,
["required"] = new JsonArray(required.Select(r => (JsonNode)JsonValue.Create(r)!).ToArray()),
},
},
};
private static JsonObject Msg(string role, string content) => new() { ["role"] = role, ["content"] = content };
private static JsonObject ToolMsg(string id, string content) => new() { ["role"] = "tool", ["tool_call_id"] = id, ["content"] = content };
[GeneratedRegex(@"```
(?:csharp|cs|c#)?\s*\n(?<code>[\s\S]*?)
```", RegexOptions.IgnoreCase)]
private static partial Regex CodeBlock();
private static string? ExtractCode(string? content)
{
if (string.IsNullOrEmpty(content)) return null;
var block = CodeBlock().Matches(content).Select(m => m.Groups["code"].Value)
.LastOrDefault(c => c.Contains("class ", StringComparison.Ordinal));
return block?.Trim();
}
}
The decision I went back and forth on most is the fallback write. Small local models regularly answer “here is the code” in a fenced block instead of calling write_file. If I ignore that, I am measuring tool-calling discipline, not coding ability. If I accept it, I am adding a harness piece of my own. I accepted it, apply it identically in every configuration, and count it, so if one configuration leans on it far more than the others, the report shows that.
Report.cs: turning runs into verdicts
using System.Text;
using System.Text.Json;
namespace Ablation;
/// <summary>
/// Turns run records into one ASCII table and a verdict per configuration:
/// result the paired difference in pass rate vs the baseline has a 95% CI that excludes zero
/// unclear the point estimate is a task's worth or more, but the CI touches zero
/// cost only no detectable result change, but tokens or wall time moved by 25% or more
/// noise neither
/// </summary>
public static class Report
{
public static string Build(IReadOnlyList<RunRecord> allRuns, string baseline, int bootstrapSamples = 5000, int seed = 7)
{
// A model_error run is an infrastructure failure, not a harness result. Exclude it and say so.
var errored = allRuns.Where(r => r.Outcome == "model_error").ToList();
var runs = allRuns.Where(r => r.Outcome != "model_error").ToList();
if (runs.Count == 0) return $"All {allRuns.Count} runs ended in model_error. Is the model server up?\n";
var tasks = runs.Select(r => r.Task).Distinct().OrderBy(t => t).ToList();
var configs = runs.Select(r => r.Config).Distinct()
.OrderBy(c => HarnessConfig.All.Select(x => x.Name).ToList().IndexOf(c) is var i && i < 0 ? 999 : i).ToList();
if (!configs.Contains(baseline)) throw new ArgumentException($"Baseline '{baseline}' has no runs.");
double PassRate(string cfg, string task)
{
var rs = runs.Where(r => r.Config == cfg && r.Task == task).ToList();
return rs.Count == 0 ? double.NaN : rs.Count(r => r.Passed) / (double)rs.Count;
}
var baseRuns = runs.Where(r => r.Config == baseline).ToList();
double baseTokens = baseRuns.Average(r => (double)r.Tokens), baseSecs = baseRuns.Average(r => r.TotalMs / 1000.0);
var sb = new StringBuilder();
var header = $"| {"Config",-15} | {"Pass",-9} | {"95% CI",-9} | {"Tok/run",8} | {"Tok/solve",9} | {"s/run",6} | {"Chk%",4} | {"dPass vs base",-17} | {"Verdict",-9} |";
var rule = "+" + string.Join("+", header.Split('|', StringSplitOptions.RemoveEmptyEntries).Select(c => new string('-', c.Length))) + "+";
sb.AppendLine(rule).AppendLine(header).AppendLine(rule);
foreach (var cfg in configs)
{
// Fresh seeded resampler per config: identical data must get an identical CI and verdict.
var rng = new Random(seed);
var rs = runs.Where(r => r.Config == cfg).ToList();
int n = rs.Count, k = rs.Count(r => r.Passed);
var (lo, hi) = Wilson(k, n);
double tokRun = rs.Average(r => (double)r.Tokens);
string tokSolve = k == 0 ? "n/a" : Short(rs.Sum(r => (double)r.Tokens) / k);
double secs = rs.Average(r => r.TotalMs / 1000.0);
double chk = rs.Sum(r => (double)r.CheckMs) / Math.Max(1, rs.Sum(r => (double)r.TotalMs));
string delta = "-", verdict = "baseline";
if (cfg != baseline)
{
// Pair by task: the unit of resampling is the task, not the run.
var diffs = tasks.Select(t => PassRate(cfg, t) - PassRate(baseline, t)).Where(d => !double.IsNaN(d)).ToArray();
double mean = diffs.Average();
var boots = new double[bootstrapSamples];
for (int b = 0; b < bootstrapSamples; b++)
{
double s = 0;
for (int i = 0; i < diffs.Length; i++) s += diffs[rng.Next(diffs.Length)];
boots[b] = s / diffs.Length;
}
Array.Sort(boots);
double bl = boots[(int)(0.025 * bootstrapSamples)], bh = boots[(int)(0.975 * bootstrapSamples) - 1];
delta = $"{mean * 100,4:+0;-0;0} [{bl * 100:+0;-0;0},{bh * 100:+0;-0;0}]";
double tokRatio = baseTokens > 0 ? tokRun / baseTokens : 1, timeRatio = baseSecs > 0 ? secs / baseSecs : 1;
bool resultMoved = bl > 0 || bh < 0;
bool costMoved = tokRatio >= 1.25 || tokRatio <= 0.8 || timeRatio >= 1.25 || timeRatio <= 0.8;
// "cost only" is a claim that the result did NOT move. If the point estimate is a task's
// worth or more but the CI still touches zero, that claim is not earned: say "unclear".
bool bigButUnsure = !resultMoved && Math.Abs(mean) >= 1.0 / tasks.Count - 1e-9;
verdict = resultMoved ? "result" : bigButUnsure ? "unclear" : costMoved ? "cost only" : "noise";
}
sb.AppendLine($"| {cfg,-15} | {$"{k}/{n} {100.0 * k / n:0}%",-9} | {$"{lo * 100:0}-{hi * 100:0}%",-9} | {Short(tokRun),8} | {tokSolve,9} | {secs,6:0.0} | {chk * 100,3:0}% | {delta,-17} | {verdict,-9} |");
}
sb.AppendLine(rule);
int reps = runs.GroupBy(r => (r.Config, r.Task)).Max(g => g.Count());
sb.AppendLine($" {tasks.Count} tasks x {reps} reps per config. Model: {runs[0].Model}. Baseline: {baseline}.");
sb.AppendLine($" dPass is in percentage points, paired by task, 95% bootstrap CI over tasks.");
sb.AppendLine($" Chk% = share of wall time spent in builds and visible checks (verifier tool + harness retries).");
if (tasks.Count < 30)
sb.AppendLine($" WARNING: with {tasks.Count} tasks each task moves a config's pass rate by {100.0 / tasks.Count:0.#} points. Read this as descriptive.");
var outcomes = runs.GroupBy(r => (r.Config, r.Outcome)).Where(g => g.Key.Outcome != "completed")
.Select(g => $"{g.Key.Config}:{g.Key.Outcome}={g.Count()}").ToList();
if (outcomes.Count > 0) sb.AppendLine($" Non-completed terminals: {string.Join(", ", outcomes)}");
if (errored.Count > 0)
sb.AppendLine($" EXCLUDED {errored.Count} model_error runs: {string.Join(", ", errored.GroupBy(r => r.Config).Select(g => $"{g.Key}={g.Count()}"))}. Re-run them before trusting the table.");
int missing = runs.Sum(r => r.UsageMissing), fallbacks = runs.Sum(r => r.FallbackWrites);
if (missing > 0) sb.AppendLine($" WARNING: {missing} model calls reported no token usage; token columns undercount.");
if (fallbacks > 0) sb.AppendLine($" Fallback writes (code block instead of a tool call): {fallbacks}.");
return sb.ToString();
}
// Last record wins per (task, config, rep), so re-running errored runs replaces them.
public static IReadOnlyList<RunRecord> Load(string path) =>
File.ReadAllLines(path).Where(l => l.Length > 0).Select(l => JsonSerializer.Deserialize<RunRecord>(l)!)
.GroupBy(r => (r.Task, r.Config, r.Rep)).Select(g => g.Last()).ToList();
private static (double Lo, double Hi) Wilson(int k, int n, double z = 1.96)
{
if (n == 0) return (0, 0);
double p = (double)k / n, d = 1 + z * z / n;
double c = (p + z * z / (2 * n)) / d, h = z * Math.Sqrt(p * (1 - p) / n + z * z / (4.0 * n * n)) / d;
return (Math.Max(0, c - h), Math.Min(1, c + h));
}
private static string Short(double v) => v >= 10_000 ? $"{v / 1000:0}k" : v >= 1000 ? $"{v / 1000:0.0}k" : $"{v:0}";
}
The verdict column is the whole point of the rig, so the rules are worth stating plainly. A configuration gets result only when the paired 95% interval for the pass-rate difference excludes zero. It gets cost only only when the pass rate did not move by even one task's worth and tokens or time moved by at least 25%. Anything in between is unclear, and I would rather read unclear ten times than have the table tell me a confident story it has not earned.
Program.cs: selftest, run, report
using System.Text.Json;
using Ablation;
// Usage:
// dotnet run -- selftest
// dotnet run -- run --base-url http://localhost:11434/v1 --model qwen2.5-coder:7b --reps 3 --out results/run.jsonl
// dotnet run -- run ... --configs bare,+verify,full --tasks 01,04
// dotnet run -- report results/run.jsonl [--baseline bare]
var opts = ParseArgs(args.Skip(1).ToArray());
string tasksDir = opts.GetValueOrDefault("tasks-dir", Path.Combine(Directory.GetCurrentDirectory(), "tasks"));
using var cts = new CancellationTokenSource();
Console.CancelKeyPress += (_, e) => { e.Cancel = true; cts.Cancel(); };
switch (args.FirstOrDefault())
{
case "selftest":
return await SelfTest();
case "run":
return await Run();
case "report":
Console.Write(Report.Build(Report.Load(args[1]), opts.GetValueOrDefault("baseline", "bare")));
return 0;
default:
Console.Error.WriteLine("Commands: selftest | run | report <file.jsonl>");
return 2;
}
// Proves the fixtures are fair: the reference passes every hidden check, the starter fails.
async Task<int> SelfTest()
{
var root = Path.Combine(Path.GetTempPath(), "ablation-selftest");
int bad = 0;
foreach (var t in TaskSpec.LoadAll(tasksDir))
{
foreach (var (label, source, mustPass) in new[] { ("reference", t.Reference, true), ("starter", t.Starter, false) })
{
var dir = Path.Combine(root, t.Id, label);
if (Directory.Exists(dir)) Directory.Delete(dir, true);
CheckRunner.WriteProject(dir, t.Visible + "\n" + t.Hidden);
File.WriteAllText(Path.Combine(dir, "Solution.cs"), source);
var r = await CheckRunner.RunAsync(dir, TimeSpan.FromMinutes(2), cts.Token);
bool ok = r.Passed == mustPass;
if (!ok) bad++;
Console.WriteLine($"{(ok ? "ok " : "BAD ")} {t.Id,-14} {label,-9} {r.ChecksPassed}/{r.ChecksTotal} checks");
if (!ok) Console.WriteLine(r.Output);
}
}
return bad == 0 ? 0 : 1;
}
async Task<int> Run()
{
var baseUrl = opts.GetValueOrDefault("base-url", "http://localhost:11434/v1");
var model = opts.GetValueOrDefault("model", "qwen2.5-coder:7b");
int reps = int.Parse(opts.GetValueOrDefault("reps", "3"));
int seed = int.Parse(opts.GetValueOrDefault("seed", "7"));
double temperature = double.Parse(opts.GetValueOrDefault("temperature", "0.2"), System.Globalization.CultureInfo.InvariantCulture);
var outPath = opts.GetValueOrDefault("out", Path.Combine("results", $"run-{DateTime.UtcNow:yyyyMMdd-HHmmss}.jsonl"));
var configs = opts.TryGetValue("configs", out var cs) ? cs.Split(',').Select(HarnessConfig.Get).ToList() : HarnessConfig.All.ToList();
var tasks = TaskSpec.LoadAll(tasksDir, opts.TryGetValue("tasks", out var ts) ? ts.Split(',') : null);
Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(outPath))!);
using var http = new HttpClient { Timeout = TimeSpan.FromMinutes(10) };
var chat = new ChatClient(http, baseUrl, model, temperature, seed);
var runner = new AgentRunner(chat, Path.Combine(Directory.GetCurrentDirectory(), "runs"), TimeSpan.FromMinutes(2));
// Interleave configs in a seeded random order so a slow afternoon or a warm KV cache
// does not land on one configuration.
var plan = (from t in tasks from c in configs from r in Enumerable.Range(1, reps) select (t, c, r))
.ToList();
var rng = new Random(seed);
plan = plan.OrderBy(_ => rng.Next()).ToList();
// Resume: skip anything already in the output file, except runs that died on a model error.
var done = File.Exists(outPath)
? Report.Load(outPath).Where(r => r.Outcome != "model_error").Select(r => (r.Task, r.Config, r.Rep)).ToHashSet()
: [];
int i = 0;
foreach (var (t, c, r) in plan)
{
i++;
if (done.Contains((t.Id, c.Name, r))) continue;
if (cts.IsCancellationRequested) break;
var rec = await runner.RunAsync(t, c, r, cts.Token);
await File.AppendAllTextAsync(outPath, JsonSerializer.Serialize(rec) + "\n");
Console.WriteLine($"[{i}/{plan.Count}] {c.Name,-15} {t.Id,-14} rep{r} {(rec.Passed ? "PASS" : "fail")} " +
$"{rec.HiddenChecksPassed}/{rec.HiddenChecksTotal} {rec.Tokens,7} tok {rec.TotalMs / 1000.0,6:0.0}s {rec.Outcome}");
}
Console.WriteLine();
Console.Write(Report.Build(Report.Load(outPath), opts.GetValueOrDefault("baseline", "bare")));
return 0;
}
static Dictionary<string, string> ParseArgs(string[] a)
{
var d = new Dictionary<string, string>();
for (int i = 0; i < a.Length; i++)
if (a[i].StartsWith("--") && i + 1 < a.Length) d[a[i][2..]] = a[++i];
return d;
}
Running it
cd harness-ablation
dotnet run -c Release -- selftest
# Ollama
dotnet run -c Release -- run --base-url http://localhost:11434/v1 \
--model qwen2.5-coder:7b --reps 3 --out results/qwen7b.jsonl
# llama.cpp (the model name is ignored by llama-server but recorded in the results)
dotnet run -c Release -- run --base-url http://localhost:8080/v1 \
--model qwen2.5-coder-7b-q4km --reps 3 --out results/qwen7b-llamacpp.jsonl
# Two views of the same data
dotnet run -c Release -- report results/qwen7b.jsonl
dotnet run -c Release -- report results/qwen7b.jsonl --baseline full
Every run leaves its full transcript and its hidden-check output under runs///rep/. Read them. The table tells you where to look, the transcripts tell you why.
What building the rig taught me before any model touched it
Before pointing the rig at a real model, I tested the plumbing against a tiny scripted fake server: a Python script that speaks the OpenAI format and behaves badly on purpose. One task always answers with a code block instead of a tool call, one re-reads the same file forever, a couple write a wrong first draft. It was meant to be a five-minute sanity check. It found four bugs, and every one of them would have quietly bent a real ablation.
Every request failed on the first try. HttpClient.PostAsJsonAsync streams the JSON body chunked, without a Content-Length header, and the fake server could not read it. Ollama and llama.cpp handle chunked bodies as far as I know, but a proxy in between might not. One StringContent later, it was fine.
A configuration where every run crashed was labeled “cost only.” With zero tokens recorded everywhere, my token ratio fell below 0.8 and the verdict logic happily called it a cost move. Infrastructure failures now get excluded from the statistics and listed loudly at the bottom of the report, instead of being scored as harness failures.
Dropped connections were scored as harness failures. The fake server closed every connection, and as far as I can tell .NET now and then reused one that was already dead. Real model servers do this under load too. Because a turn is only added to the history after a successful response, a transport retry cannot duplicate a message, which is exactly the property Bilaniuc’s rules ask for. Before that fix, those runs would have counted against whichever configuration made more calls, which is the full one. A harness with more model calls would have looked worse purely because it rolled the network dice more often.
Two configurations with identical results got different verdicts. Same pass counts, same tasks, one labeled result, the other one task short of it. The bootstrap shared one random stream across configurations, and with eight tasks the lower bound of the interval sat right on zero, so the resampler's luck decided the label. Each configuration now gets its own seeded resampler. But I want to say the uncomfortable part out loud: the fix made the verdict reproducible, not robust. At eight tasks, a verdict at the boundary is still a coin flip with a fixed seed.
None of this is about models or harnesses. All of it is about the measurement, and all of it would have shown up in a results table looking like a finding.
What I am betting on
I prefer to write down my expectations before looking at the table, for the same reason I set my noise threshold before looking in the eval loop piece: afterward, I will find a story for whatever comes out. These are my bets for a 7B coding model on these eight tasks, with the reasoning, so I can be wrong in public.
+-----------------+-------------------+-----------------+-------------------+----------------+
| Config | Pass rate | Tokens | Wall time | Expected label |
+-----------------+-------------------+-----------------+-------------------+----------------+
| bare | baseline | baseline | baseline | baseline |
| +plan | flat | up, modest | up, modest | cost only |
| +verify | up | up | up, a lot | result |
| +retry | up | up on failures | up | result |
| +budget | flat | up | flat to up | cost only |
| full | up, like +verify | up the most | up the most | result |
| full-no-plan | same as full | below full | below full | (vs full) cost |
| full-no-verify | close to full | below full | below full | (vs full) ? |
| full-no-retry | close to full | below full | below full | (vs full) ? |
| full-tight | below full | below full | below full | unclear |
+-----------------+-------------------+-----------------+-------------------+----------------+
The reasoning, piece by piece:
The planner moves cost. These tasks are small enough that the prompt already is the plan. My guess is that a separate planning call costs one extra round trip and some prompt tokens and changes very little. If the planner earns a result verdict on a 7B model, that is the most interesting thing the rig could tell me, because it would mean smaller models need structure that larger ones do not, which is the fit papers' claim in miniature.
The verifier and the retry move the result. They are the only pieces that give the model information it cannot produce itself: the compiler’s opinion and the checks’ opinion. That matches the self-review result from my own eval loop and the one place the Scaffold Effect paper showed a harness moving outcomes. If either one comes back cost only, my revised position from earlier in this article is wrong too, not just the original one.
The verifier and the retry overlap. Removing one from the full harness should show much less than adding it to bare, because the other one covers for it. That is the reason for having both ablation directions.
More budget mostly buys more reading. With a looser budget, I expect the model to re-read files and wander, the way Molinaro’s agent measured pixels. Tokens up, result flat.
The tight budget is where the verifier eats the agent. Six tool calls, and each check costs one. I expect runs where the model spends its budget checking a broken draft and ends with budget_exhausted before it writes the fix. That is the configuration where the share of time spent in checks should be highest.
What would change my mind: the verifier landing on cost only, or the planner landing on result, on more than one model. A single model will not settle either.
Is the sample big enough? No.
I promised myself I would say this plainly, so here it is.
Eight tasks with three repetitions is 24 runs per configuration. A configuration that solves 12 of 24 has a 95% Wilson interval of roughly 31% to 69%. One that solves 15 of 24, a full 12.5 points better, has an interval of roughly 43% to 79%. Those overlap almost entirely. Pairing by task helps, but pairing does not create tasks: the unit of resampling is still eight tasks, and every single task is worth 12.5 points of pass rate.
To detect a 10-point difference around a 50% pass rate, with the usual 95% confidence and 80% power, a simple unpaired comparison needs on the order of 390 runs per configuration. I am not going to get there with hand-written tasks on a laptop.
So what can this rig actually say?
- Large result effects. If a piece turns a model that cannot get the code to compile into one that can, that shows up even at eight tasks.
- The size of cost effects. Tokens and seconds are measured, not estimated. A 3x token increase is a 3x token increase whether I have eight tasks or eight hundred.
- Not a strong “only cost.” This is the asymmetry I did not appreciate before building it. “This piece moved the cost” is easy to show. “This piece did not move the result” is a claim about the absence of an effect, and eight tasks cannot certify an absence. In my report, cost only really means "no task flipped on average, and the bill changed." That is weaker than the phrase sounds, which is why the unclear label exists.
The Scaffold Effect authors said the same about their 50 tasks, where each task was worth 2 points. I have a quarter of their tasks. Anything I conclude from this rig on its own should be read as a direction to investigate, not as a finding.
What I believe now
I still think the harness is where most of the engineering is. I no longer think “the harness is the bottleneck” is a useful sentence on its own, because it hides the question that matters: bottleneck for what?
For cost, the evidence is strong and consistent. Harness choice moved tokens per solve by around 40x in the Scaffold Effect paper while the model moved it by at most 1.3x. Khan’s system shipped 4 pieces out of 11. Molinaro’s agent spent most of its time measuring. If you care about the bill, the harness is the first place to look, and an ablation will tell you quickly which pieces are rent.
For the result, the evidence is conditional. Swapping one reasonable harness for another on a capable model mostly lands inside the noise. Fitting a harness to a weaker model, or adding a piece that hands the model real information, can move it a lot. That is a narrower claim than the one I made a month ago, and I think it is the honest one.
If you take one thing from this: whenever someone, including me, says a harness piece “made the agent better,” ask which axis moved and how many tasks it was measured on. If the answer is “the score went up” and “about ten,” read the transcripts before you believe it. Molinaro did, and found one page.
Every file of the rig is in this post, with the remaining task files in the appendix below. Point it at whatever you run locally, and if one of my bets above is wrong on your model, I would genuinely like to hear which one.
Appendix: the other seven task files
Save each block as tasks/.task. Run dotnet run -c Release -- selftest afterwards; every line should start with ok.
01-duration.task
== prompt ==
Implement `Durations.ParseDuration(string s)` in Solution.cs.
It parses compact durations made of hours, minutes and seconds, like "1h30m", "45s" or "1h2m3s",
and returns a TimeSpan. Units are h, m, s. Each unit may appear at most once, in the order h, m, s.
Parsing is case-insensitive and ignores leading and trailing whitespace. Values are non-negative integers
and may exceed the next unit (for example "90m" is valid).
Anything else (empty input, unknown units, repeated units, units out of order, a unit with no number)
must throw FormatException.
== starter ==
using System;
public static class Durations
{
public static TimeSpan ParseDuration(string s)
{
throw new NotImplementedException();
}
}
== visible ==
Check.Run("1h30m", () => Check.Eq(TimeSpan.FromMinutes(90), Durations.ParseDuration("1h30m")));
Check.Run("45s", () => Check.Eq(TimeSpan.FromSeconds(45), Durations.ParseDuration("45s")));
Check.Run("2h", () => Check.Eq(TimeSpan.FromHours(2), Durations.ParseDuration("2h")));
== hidden ==
Check.Run("upper case", () => Check.Eq(new TimeSpan(1, 2, 3), Durations.ParseDuration("1H2M3S")));
Check.Run("whitespace", () => Check.Eq(TimeSpan.FromMinutes(10), Durations.ParseDuration(" 10m ")));
Check.Run("90m", () => Check.Eq(TimeSpan.FromMinutes(90), Durations.ParseDuration("90m")));
Check.Run("empty throws", () => Check.Throws<FormatException>(() => Durations.ParseDuration("")));
Check.Run("garbage throws", () => Check.Throws<FormatException>(() => Durations.ParseDuration("abc")));
Check.Run("unknown unit throws", () => Check.Throws<FormatException>(() => Durations.ParseDuration("5x")));
Check.Run("repeated unit throws", () => Check.Throws<FormatException>(() => Durations.ParseDuration("1h1h")));
Check.Run("out of order throws", () => Check.Throws<FormatException>(() => Durations.ParseDuration("5m1h")));
Check.Run("unit without number throws", () => Check.Throws<FormatException>(() => Durations.ParseDuration("m5")));
== reference ==
using System;
using System.Text.RegularExpressions;
public static class Durations
{
private static readonly Regex Pattern =
new(@"^(?:(\d+)h)?(?:(\d+)m)?(?:(\d+)s)?$", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant);
public static TimeSpan ParseDuration(string s)
{
var text = (s ?? "").Trim();
var m = Pattern.Match(text);
if (text.Length == 0 || !m.Success) throw new FormatException($"Invalid duration '{s}'.");
long Part(int i) => m.Groups[i].Success ? long.Parse(m.Groups[i].Value) : 0;
return TimeSpan.FromHours(Part(1)) + TimeSpan.FromMinutes(Part(2)) + TimeSpan.FromSeconds(Part(3));
}
}
02-slugify.task
== prompt ==
Implement `Slugs.Slugify(string text)` in Solution.cs.
Rules: remove diacritics (é becomes e), lower-case with the invariant culture, keep only ASCII letters a-z
and digits 0-9, replace every run of other characters with a single '-', and trim '-' from both ends.
If nothing is left, return an empty string. A null input returns an empty string.
== starter ==
using System;
public static class Slugs
{
public static string Slugify(string text)
{
throw new NotImplementedException();
}
}
== visible ==
Check.Run("hello world", () => Check.Eq("hello-world", Slugs.Slugify("Hello World")));
Check.Run("spaces", () => Check.Eq("a-b", Slugs.Slugify(" A B ")));
== hidden ==
Check.Run("diacritics", () => Check.Eq("creme-brulee", Slugs.Slugify("Crème Brûlée!")));
Check.Run("only separators", () => Check.Eq("", Slugs.Slugify("---")));
Check.Run("symbols", () => Check.Eq("c-net-10", Slugs.Slugify("C# & .NET 10")));
Check.Run("dotted capital I", () => Check.Eq("istanbul", Slugs.Slugify("İstanbul")));
Check.Run("null", () => Check.Eq("", Slugs.Slugify(null!)));
== reference ==
using System;
using System.Globalization;
using System.Text;
public static class Slugs
{
public static string Slugify(string text)
{
if (string.IsNullOrEmpty(text)) return "";
var sb = new StringBuilder();
bool dash = false;
foreach (var ch in text.Normalize(NormalizationForm.FormD))
{
if (CharUnicodeInfo.GetUnicodeCategory(ch) == UnicodeCategory.NonSpacingMark) continue;
var c = char.ToLowerInvariant(ch);
if ((c >= 'a' && c <= 'z') || (c >= '0' && c <= '9')) { sb.Append(c); dash = false; }
else if (!dash && sb.Length > 0) { sb.Append('-'); dash = true; }
}
return sb.ToString().TrimEnd('-');
}
}
03-intervals.task
== prompt ==
Implement `Intervals.Merge(IEnumerable<(int Start, int End)> items)` in Solution.cs.
Return the merged intervals sorted by Start. Intervals are closed, so intervals that touch
(for example (1,2) and (2,3)) merge into one. The input may be unsorted and may be empty.
Every input interval has Start <= End.
== starter ==
using System;
using System.Collections.Generic;
public static class Intervals
{
public static List<(int Start, int End)> Merge(IEnumerable<(int Start, int End)> items)
{
throw new NotImplementedException();
}
}
== visible ==
Check.Run("overlap", () => Check.Seq(new[] { (1, 6), (8, 10) }, Intervals.Merge(new[] { (1, 3), (2, 6), (8, 10) })));
Check.Run("disjoint", () => Check.Seq(new[] { (1, 2), (5, 6) }, Intervals.Merge(new[] { (1, 2), (5, 6) })));
== hidden ==
Check.Run("touching", () => Check.Seq(new[] { (1, 3) }, Intervals.Merge(new[] { (1, 2), (2, 3) })));
Check.Run("unsorted", () => Check.Seq(new[] { (1, 4), (6, 9) }, Intervals.Merge(new[] { (6, 9), (3, 4), (1, 3) })));
Check.Run("contained", () => Check.Seq(new[] { (1, 10) }, Intervals.Merge(new[] { (1, 10), (2, 3), (4, 5) })));
Check.Run("empty", () => Check.Seq(Array.Empty<(int, int)>(), Intervals.Merge(Array.Empty<(int, int)>())));
Check.Run("single", () => Check.Seq(new[] { (7, 7) }, Intervals.Merge(new[] { (7, 7) })));
== reference ==
using System;
using System.Collections.Generic;
using System.Linq;
public static class Intervals
{
public static List<(int Start, int End)> Merge(IEnumerable<(int Start, int End)> items)
{
var result = new List<(int Start, int End)>();
foreach (var (s, e) in items.OrderBy(i => i.Start))
{
if (result.Count > 0 && s <= result[^1].End)
result[^1] = (result[^1].Start, Math.Max(result[^1].End, e));
else
result.Add((s, e));
}
return result;
}
}
04-semver.task
== prompt ==
Implement `SemVer.Compare(string a, string b)` in Solution.cs.
Compare two Semantic Versioning 2.0.0 strings by precedence and return -1, 0 or 1.
Follow the SemVer 2.0.0 precedence rules exactly, including pre-release identifiers
(a pre-release version has lower precedence than the normal version, identifiers are compared
one by one, numeric identifiers compare numerically and are lower than alphanumeric ones,
a longer set of identifiers wins when all preceding ones are equal). Build metadata after '+' is ignored.
You can assume both inputs are valid SemVer strings.
== starter ==
using System;
public static class SemVer
{
public static int Compare(string a, string b)
{
throw new NotImplementedException();
}
}
== visible ==
Check.Run("major", () => Check.Eq(-1, SemVer.Compare("1.0.0", "2.0.0")));
Check.Run("numeric minor", () => Check.Eq(-1, SemVer.Compare("1.2.0", "1.10.0")));
Check.Run("equal", () => Check.Eq(0, SemVer.Compare("3.4.5", "3.4.5")));
== hidden ==
Check.Run("prerelease lower", () => Check.Eq(-1, SemVer.Compare("1.0.0-alpha", "1.0.0")));
Check.Run("longer prerelease wins", () => Check.Eq(-1, SemVer.Compare("1.0.0-alpha", "1.0.0-alpha.1")));
Check.Run("alpha.beta < beta", () => Check.Eq(-1, SemVer.Compare("1.0.0-alpha.beta", "1.0.0-beta")));
Check.Run("numeric ids", () => Check.Eq(-1, SemVer.Compare("1.0.0-beta.2", "1.0.0-beta.11")));
Check.Run("numeric below alpha", () => Check.Eq(-1, SemVer.Compare("1.0.0-1", "1.0.0-alpha")));
Check.Run("rc below release", () => Check.Eq(1, SemVer.Compare("1.0.0", "1.0.0-rc.1")));
Check.Run("build ignored", () => Check.Eq(0, SemVer.Compare("1.0.0+build.5", "1.0.0")));
== reference ==
using System;
using System.Numerics;
public static class SemVer
{
public static int Compare(string a, string b)
{
static (string Core, string? Pre) Split(string v)
{
v = v.Split('+')[0];
var i = v.IndexOf('-');
return i < 0 ? (v, null) : (v[..i], v[(i + 1)..]);
}
var (ca, pa) = Split(a);
var (cb, pb) = Split(b);
var xa = ca.Split('.'); var xb = cb.Split('.');
for (int i = 0; i < 3; i++)
{
var c = BigInteger.Parse(xa[i]).CompareTo(BigInteger.Parse(xb[i]));
if (c != 0) return Math.Sign(c);
}
if (pa == null && pb == null) return 0;
if (pa == null) return 1;
if (pb == null) return -1;
var ia = pa.Split('.'); var ib = pb.Split('.');
for (int i = 0; i < Math.Min(ia.Length, ib.Length); i++)
{
bool na = BigInteger.TryParse(ia[i], out var va), nb = BigInteger.TryParse(ib[i], out var vb);
int c = (na, nb) switch
{
(true, true) => va.CompareTo(vb),
(true, false) => -1,
(false, true) => 1,
_ => string.CompareOrdinal(ia[i], ib[i]),
};
if (c != 0) return Math.Sign(c);
}
return Math.Sign(ia.Length.CompareTo(ib.Length));
}
}
05-csv.task
== prompt ==
Implement `Csv.SplitLine(string line)` in Solution.cs.
Split one CSV line (RFC 4180 style) into fields. Fields are separated by commas. A field may be wrapped
in double quotes, in which case it can contain commas, and a doubled quote ("") inside it means one quote.
Do not trim whitespace. Empty fields are kept, so a trailing comma produces an empty last field.
An empty line returns a single empty field.
== starter ==
using System;
using System.Collections.Generic;
public static class Csv
{
public static List<string> SplitLine(string line)
{
throw new NotImplementedException();
}
}
== visible ==
Check.Run("simple", () => Check.Seq(new[] { "a", "b", "c" }, Csv.SplitLine("a,b,c")));
Check.Run("quoted comma", () => Check.Seq(new[] { "a,b", "c" }, Csv.SplitLine("\"a,b\",c")));
== hidden ==
Check.Run("escaped quotes", () => Check.Seq(new[] { "a", "he said \"hi\"", "c" }, Csv.SplitLine("a,\"he said \"\"hi\"\"\",c")));
Check.Run("all empty", () => Check.Seq(new[] { "", "", "" }, Csv.SplitLine(",,")));
Check.Run("trailing comma", () => Check.Seq(new[] { "a", "" }, Csv.SplitLine("a,")));
Check.Run("empty quoted", () => Check.Seq(new[] { "" }, Csv.SplitLine("\"\"")));
Check.Run("no trim", () => Check.Seq(new[] { "x", " y" }, Csv.SplitLine("x, y")));
Check.Run("empty line", () => Check.Seq(new[] { "" }, Csv.SplitLine("")));
== reference ==
using System;
using System.Collections.Generic;
using System.Text;
public static class Csv
{
public static List<string> SplitLine(string line)
{
var fields = new List<string>();
var sb = new StringBuilder();
bool quoted = false;
for (int i = 0; i < line.Length; i++)
{
char c = line[i];
if (quoted)
{
if (c == '"' && i + 1 < line.Length && line[i + 1] == '"') { sb.Append('"'); i++; }
else if (c == '"') quoted = false;
else sb.Append(c);
}
else if (c == '"') quoted = true;
else if (c == ',') { fields.Add(sb.ToString()); sb.Clear(); }
else sb.Append(c);
}
fields.Add(sb.ToString());
return fields;
}
}
06-lru.task
== prompt ==
Implement the class `LruCache<TKey, TValue>` in Solution.cs.
- Constructor `LruCache(int capacity)`; capacity must be at least 1, otherwise throw ArgumentOutOfRangeException.
- `void Put(TKey key, TValue value)` inserts or updates. Updating an existing key also makes it the most recently used.
- `bool TryGet(TKey key, out TValue value)` returns the value and marks the key as most recently used.
- `int Count` is the number of stored entries.
- When a Put would exceed capacity, evict the least recently used entry.
Get and Put should be O(1).
== starter ==
using System;
using System.Collections.Generic;
public sealed class LruCache<TKey, TValue> where TKey : notnull
{
public LruCache(int capacity) => throw new NotImplementedException();
public int Count => throw new NotImplementedException();
public void Put(TKey key, TValue value) => throw new NotImplementedException();
public bool TryGet(TKey key, out TValue value) => throw new NotImplementedException();
}
== visible ==
Check.Run("put get", () => { var c = new LruCache<string, int>(2); c.Put("a", 1); Check.True(c.TryGet("a", out var v) && v == 1); });
Check.Run("evicts oldest", () => { var c = new LruCache<string, int>(2); c.Put("a", 1); c.Put("b", 2); c.Put("c", 3); Check.True(!c.TryGet("a", out _) && c.Count == 2); });
== hidden ==
Check.Run("get refreshes", () => { var c = new LruCache<string, int>(2); c.Put("a", 1); c.Put("b", 2); c.TryGet("a", out _); c.Put("c", 3); Check.True(c.TryGet("a", out _) && !c.TryGet("b", out _)); });
Check.Run("update refreshes", () => { var c = new LruCache<string, int>(2); c.Put("a", 1); c.Put("b", 2); c.Put("a", 10); c.Put("c", 3); Check.True(c.TryGet("a", out var v) && v == 10 && !c.TryGet("b", out _)); });
Check.Run("update keeps count", () => { var c = new LruCache<string, int>(2); c.Put("a", 1); c.Put("a", 2); Check.Eq(1, c.Count); });
Check.Run("capacity one", () => { var c = new LruCache<int, int>(1); c.Put(1, 1); c.Put(2, 2); Check.True(!c.TryGet(1, out _) && c.TryGet(2, out _)); });
Check.Run("zero capacity throws", () => Check.Throws<ArgumentOutOfRangeException>(() => new LruCache<int, int>(0)));
== reference ==
using System;
using System.Collections.Generic;
public sealed class LruCache<TKey, TValue> where TKey : notnull
{
private readonly int _capacity;
private readonly Dictionary<TKey, LinkedListNode<(TKey Key, TValue Value)>> _map = new();
private readonly LinkedList<(TKey Key, TValue Value)> _order = new();
public LruCache(int capacity)
{
if (capacity < 1) throw new ArgumentOutOfRangeException(nameof(capacity));
_capacity = capacity;
}
public int Count => _map.Count;
public void Put(TKey key, TValue value)
{
if (_map.TryGetValue(key, out var node)) { _order.Remove(node); _map.Remove(key); }
else if (_map.Count == _capacity) { var last = _order.Last!; _order.RemoveLast(); _map.Remove(last.Value.Key); }
_map[key] = _order.AddFirst((key, value));
}
public bool TryGet(TKey key, out TValue value)
{
if (_map.TryGetValue(key, out var node))
{
_order.Remove(node); _order.AddFirst(node);
value = node.Value.Value; return true;
}
value = default!; return false;
}
}
08-roman.task
== prompt ==
Implement `Roman.ToRoman(int value)` in Solution.cs.
Convert an integer from 1 to 3999 to a standard Roman numeral using subtractive notation
(4 is IV, 9 is IX, 40 is XL, 90 is XC, 400 is CD, 900 is CM).
Values outside 1..3999 throw ArgumentOutOfRangeException.
== starter ==
using System;
public static class Roman
{
public static string ToRoman(int value)
{
throw new NotImplementedException();
}
}
== visible ==
Check.Run("3", () => Check.Eq("III", Roman.ToRoman(3)));
Check.Run("4", () => Check.Eq("IV", Roman.ToRoman(4)));
Check.Run("1994", () => Check.Eq("MCMXCIV", Roman.ToRoman(1994)));
== hidden ==
Check.Run("3999", () => Check.Eq("MMMCMXCIX", Roman.ToRoman(3999)));
Check.Run("40", () => Check.Eq("XL", Roman.ToRoman(40)));
Check.Run("944", () => Check.Eq("CMXLIV", Roman.ToRoman(944)));
Check.Run("zero throws", () => Check.Throws<ArgumentOutOfRangeException>(() => Roman.ToRoman(0)));
Check.Run("4000 throws", () => Check.Throws<ArgumentOutOfRangeException>(() => Roman.ToRoman(4000)));
== reference ==
using System;
using System.Text;
public static class Roman
{
private static readonly (int Value, string Symbol)[] Table =
{
(1000, "M"), (900, "CM"), (500, "D"), (400, "CD"), (100, "C"), (90, "XC"),
(50, "L"), (40, "XL"), (10, "X"), (9, "IX"), (5, "V"), (4, "IV"), (1, "I"),
};
public static string ToRoman(int value)
{
if (value is < 1 or > 3999) throw new ArgumentOutOfRangeException(nameof(value));
var sb = new StringBuilder();
foreach (var (v, s) in Table)
while (value >= v) { sb.Append(s); value -= v; }
return sb.ToString();
}
}
Sources
Primary sources I read directly:
- METR, “Measuring Time Horizon Using Claude Code and Codex,” research note, February 2026: https://metr.org/notes/2026-02-13-measuring-time-horizon-using-claude-code-and-codex/
- Naman Vats and Oleg Golev, “The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation,” arXiv 2607.22585 (preliminary, DL4C workshop at ICML 2026): https://arxiv.org/pdf/2607.22585
- Wenqi Huang, Charley Lee, Leonard Tng, Serena Ge (Datacurve), “DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks,” arXiv 2607.07946: https://arxiv.org/pdf/2607.07946
- Kyungmin Kim et al., “The Interplay of Harness Design and Post-Training in LLM Agents,” ICML 2026: https://icml.cc/virtual/2026/78142
- Dragos Bilaniuc, “Reliability at the Core: Five Ground Rules for Building an Agent Loop,” Towards AI: https://pub.towardsai.net/reliability-at-the-core-five-ground-rules-for-building-an-agent-loop-d99df50f9dbf
Member-only pieces I could read only in part; their figures are author-reported or secondhand as labeled above:
- Jaroslaw Wasowski, “Harness Engineering: When Agent Harness Moves the Result and When It Only Moves the Cost,” Level Up Coding: https://medium.com/gitconnected/harness-engineering-when-agent-harness-moves-the-result-and-when-it-only-moves-the-cost-3f14c46a8210
- Jaroslaw Wasowski, “Harness Engineering: Model-Harness Fit: Why a Better Teacher Makes a Worse Agent,” Level Up Coding (source of the Salesforce preprint figures, arXiv 2609.09134, which I did not locate myself): https://levelup.gitconnected.com/harness-engineering-model-harness-fit-why-a-better-teacher-makes-a-worse-agent-8c07de6f7dcf
- Fareed Khan, “Fixing Over-Engineering in LLM Coding Agents,” Level Up Coding: https://levelup.gitconnected.com/fixing-over-engineering-in-llm-coding-agents-12e31ec83e9b
- Pasquale Molinaro, “I Had a Free Qwen3.8–27B Agent Copy 10 Web Pages. It Spent 64% of Its Time Measuring Pixels,” AI Advances: https://medium.com/ai-advances/i-had-a-free-qwen3-8-27b-agent-copy-10-web-pages-it-spent-64-of-its-time-measuring-pixels-7e9dbd4daf99
My earlier pieces:
- “Harness Engineering Is the New Bottleneck, Not the Model”: https://topuzas.medium.com/harness-engineering-is-the-new-bottleneck-not-the-model-2d446ec31ff2
- “I Ran a Harness Eval Loop on My Own .NET Agent. Here’s What Changed and What Didn’t”: https://topuzas.medium.com/i-ran-a-harness-eval-loop-on-my-own-net-agent-heres-what-changed-and-what-didn-t-9646d6526db2
Tags: ai-agents, harness-engineering, dotnet, csharp, llm, evaluation, ollama, llama-cpp, local-llm, benchmarking
Top comments (0)