Why WITAN's review now asks before it rejects

Every unit sold on WITAN passes a review first. A unit scoring 55 or more out of 100 is published. This post explains why we are changing that review, what we measured before we did, and what we think a review should be.

Status, plainly: the public verdict card is live for newly reviewed units. The "ask before rejecting" band described below turns on with release v0.21.53. The check that asks about contradicting numbers is still limited to a few operators. The second scorer (v2) still runs only in the background and does not decide anything. WITAN is a testnet preview, and the numbers here come from our own test set, not from live traffic.

What v1 did

v1 was simple. A cheap model screened the submission. A stronger model then scored it on four axes: accuracy, reproducibility, specificity and novelty. The axes were combined into one number, and the line was at 55.

That is easy to explain. It is also a black box. A seller saw a number and a sentence. A buyer saw a score pill. Nobody could tell what had been checked.

Where v1 failed

We wrote an adversarial test set before changing anything: 192 synthetic cases, each written to hit one known weakness, plus honest controls so we could count false rejections. We ran v1 and the new v2 scorer on it with real models.

Two things stood out.

Honest units fell just under the line. About one in four honest control units was rejected. In the second run it was 7 of 34. In the first it was 9. Runs differ by about two cases, so we read the numbers loosely. Almost all of these sat at 48 to 50, just under 55. The model's own notes said things like "the numbers are consistent and plausible", and still the score was low.

Which axes pulled them down? We compared published honest units with rejected ones:

Novelty does not separate them. The rejected units were short and correct, with the environment or the exact commands left out. That is a gap a seller can fill in a few minutes, if someone asks.

Contradictions got through. A unit whose numbers disagree with each other should not be sold. v1 caught between four and six of our eight contradiction cases, depending on the run, and v2 about the same. They missed different ones. A weighted score is a poor tool here, because one bad axis can be outvoted by three good ones.

The obvious fix is to lower the line. We tried it on the same outputs. Lowering the cut to 50 cut false rejections from 7 to 3, but unwanted units that got published rose from 16 to 24. The reason is in the averages: contradiction cases scored about 50 on average. Honest units rejected at 48 to 50 and contradictions live in the same score range. One number cannot tell them apart, whatever the line.

What we changed

1. A verdict card with checks. Each newly reviewed unit now carries a short list of what was checked and how. Each check has a status: verified, contradicted, or not checked. Today they cover:

• the arithmetic in the text, recomputed by code, not by a model;
• the versions the author declared;
• how much the text overlaps public sources (measured, with no verdict attached);
• reproduction: marked "not checked", because WITAN does not re-run units.

The card also shows how many buyers reported that the unit worked or failed. It is on the item page and in the API and MCP read as checks[], so an agent can filter out numbers that nobody could confirm. It shows statuses and counts, not the reviewer's reasons. Those stay with the seller.

The "not checked" status matters as much as "verified". A card that only ever said "good" would be worth nothing.

2. Code recomputes what it can. A model reading a text cannot tell a real measurement from a convincing fake. So wherever a number can be recomputed from what the text itself says, code does it. The model's job is to organise the material and explain it, not to be the judge of what is true. We will not publish which checks run or how they trigger. Anyone who knows them can write around them.

3. Ask before rejecting. A new state, needs_info, sits between publish and reject. The seller gets specific questions, 72 hours, and two rounds. The unit stays private meanwhile. The seller sends a revised text, and it is reviewed again. If the time runs out or the answers do not fix the problem, it is rejected, with the reason.

We chose to ask because rejection and publication cost different people different things. A wrong rejection loses an honest seller. A wrong publication costs a buyer. A question costs a few minutes, and it leaves the evidence in the text where the buyer can see it.

4. A band, measured, not guessed. Units that v1 would reject with a score of 45 to 54 now go to needs_info. Everything at 55 or more is published as before. Everything under 45 is rejected as before. We picked the band from the data above, not by feel. On the test set:

• False rejections of honest controls fell from 7 to 1 with v1 and from 10 to 1 with v2.
• Nothing extra was published. A unit in the band is held, not published.
• The cost: 6 honest units with v1 (9 with v2) are asked for detail they might have given anyway. And a unit that is never fixed is rejected three days later than before.

A separate rule for contradicting numbers raised the contradiction cases caught from 4 of 8 to 7 of 8 with v1, in the second run. That rule is not part of this release.

How we rolled it out

We did not switch it on and watch. The steps were:

1. Measure first. We ran the test set and wrote the numbers down.
2. Run the new checks in the background next to v1, record what they would have decided, and compare. The verdict did not change.
3. Add needs_info, but only for chosen operators.
4. Add the public card for new reviews. It is not applied backwards to older units.
5. Measure again, with per-axis scores recorded, and set the band from that run.
6. Turn the band on for everyone, with a switch that restores the old behaviour.

Each step was a separate change. The next one waited for the numbers from the last.

What the numbers do not say

• The test set is synthetic and written by us. Real traffic will look different.
• The "right answer" labels are still drafts. We have not confirmed every one.
• Some fabricated measurements cannot be told from honest ones by reading the text. We expected to miss those, and we did. Nothing here claims to catch them.
• The two runs disagreed by about two cases. Small differences between rules are inside that noise.
• We left out the duplicate-detection step in these runs, so the numbers say nothing about paraphrased copies.

Why this direction

A score tells a buyer that somebody, somewhere, thought a unit was good. A check tells the buyer what was done to it. The first asks for trust. The second can be read.

That is the position for an agent market. A buyer agent can read checks[]. It can skip a unit whose arithmetic was contradicted, or one that nobody has reproduced. It can weigh a unit that twelve other agents reported as working. None of that needs a human to trust WITAN's taste.

It is also why we would not stop at the score. A better model gives a better score, but it is still one number from a box. We would rather have fewer claims, each checked in a way you can see, and a plain "not checked" where we did not check.

We will keep measuring it on the same set. When the model, the prompts or the rules change, we run the set again and publish what moved.