How an LLM Benchmark Produces Comparable Numbers: Inside LFORLA's LLM Evaluation

TL;DR: LFORLA's Capability Retention benchmark runs 20 deterministic tasks through every model with identical prompts, scores them with one fixed judge against per-axis rubrics, and publishes every answer verbatim so you can re-read exactly what the model said. The result is a leaderboard where a 90 and a 35 mean the same thing across models.

How the evaluation works

The benchmark, called Capability Retention (Normal vs Uncensored), asks a simple question: what does uncensoring cost in raw capability? To answer it comparably, the pipeline is deliberately rigid.

1. Same prompts for every model. The suite contains 20 deterministic tasks spanning reasoning, code, math, instruction following, and reading. Deterministic means the expected answer is fixed, so there is no room for a judge to invent a new standard mid-run.
2. Same fixed judge. One judge configuration scores every submission. No model gets a friendlier grader, and no run swaps judges halfway through.
3. Per-axis rubrics. Each task is scored against explicit criteria for the relevant axis. A math task is checked for the correct final value; a code task for behavior, not style; an instruction task for compliance with the stated constraint. The rubric is the contract, not the judge's mood.
4. Public verbatim trail. Every prompt and every model answer is published. Anyone can re-read what the model actually wrote and check the score against the rubric.

That combination is what makes the numbers comparable. A score is not a vibe; it is the output of a repeatable procedure applied identically to each model.

The real numbers

We also submitted our own model to this benchmark. GLM 5.2 scored 78.0 overall — real results, published alongside everyone else's.

Why the ranking looks this way

The top of the table is not a clean story about censorship. Qwen2.5 7B Instruct leads at 90, but its abliterated sibling sits at 80 — a 10-point retention cost. Llama 3.1 8B Abliterated at 85 actually beats its instruct counterpart at 70, which suggests the abliteration procedure preserved more of the underlying capability for that model family. Mistral 7B Instruct at 60 and Dolphin Mistral 7B at 35 show the widest gap in the set: 25 points between a base instruct model and its uncensored derivative.

In practice, a 90 means the model handled roughly nine in ten of the deterministic tasks correctly under the rubric. A 35 means it missed most of them. The scores are not percentages of human preference; they are task-level pass rates on a fixed suite. That is why the gap between 85 and 80 matters more than the gap between 90 and 85 — both are single-task differences on a 20-task set.

What this means for practitioners choosing a model

If you are picking a model for a task with a verifiable answer — math, code, structured instructions — the retention gap is the number to watch. An uncensored model that drops 25 points may still be fine for open-ended chat but will fail deterministically checkable work. Our own GLM 5.2 at 78.0 sits between the abliterated Llama and the abliterated Qwen, which is a useful calibration point: a general model can land mid-table without being the best or worst at any single axis.

How to read a leaderboard

• Check the task count. A 20-task suite is directional, not definitive. One task is five points.
• Find the judge. If the judge is not fixed and described, the numbers are not comparable across models.
• Look for the verbatim trail. If you cannot read the model's actual answer, you cannot verify the score.
• Compare within families. Instruct vs abliterated pairs tell you the retention cost; cross-family comparisons tell you less.
• Treat your own submission as data. We published GLM 5.2 at 78.0 because a benchmark you only win is not a benchmark.

Honest limitations

Twenty tasks is a small suite. Deterministic scoring rewards tasks with checkable answers and underweights open-ended quality. The judge, however fixed, is still a single configuration. And local providers mean hardware differences can affect reproducibility even with identical prompts. These are real constraints, not footnotes.

Conclusion

A comparable LLM evaluation is mostly discipline: same prompts, same judge, explicit rubrics, public answers. The Capability Retention leaderboard is one implementation of that discipline, and you can inspect every answer behind every score. See the full leaderboard and the verbatim trail at lforla.org.