A field changes its name.
The new model returns the same information under a slightly different key. The code reading it falls through to a default, and every result looks fine until someone checks the totals.
Model swap regression
The model you tested won’t be the one running next quarter.
Providers replace and retire models on their own schedule. A replacement can fail your task in ways public benchmarks never measure, and a customer finds it first. That is model swap regression, and it is the problem Orlo is built around.
What Orlo does about itThe model changes. The standard stays.
What it looks like when it happens.
Part of the answer goes missing.
An extraction that used to return every clause now returns the ones the model finds important. Nothing errors. The record is simply shorter than it was.
The same question gets a different answer.
A ticket that used to escalate is now routine. A refund that used to be refused is now approved. The prompt did not change; the model reading it did.
Public benchmarks never see your task.
A higher score on a leaderboard says nothing about your format, your rules, or your edge cases.
Averages hide the five percent of inputs that carry the cost. A replacement can lift a general score and still break the cases you are paid to get right.
The replacement has to be tested on the work it will actually do.
That means your examples, with the answers your reviewers expect, scored the same way the current model was scored.
What Orlo does about it.
Tests the replacement against the model that is live, on your examples.
Both candidates get the same inputs and the same scoring. The result carries a 95% confidence interval, so a small lead is reported as what it is: too close to call.
Stops spending on a candidate that is clearly behind.
When one model’s interval sits below the other’s, the comparison drops it early instead of running every example.
Releases the winner as a frozen version, with the comparison on record.
The model, the instructions, the rules, and the documents are fixed together. A release made from a comparison can only carry a model that passed it, and the release records which comparison chose it.
Flags a live release whose model is on its way out.
When the catalog marks a model deprecated or retired, an Inbox signal opens for every live release that uses it. Orlo does not re-test on its own; you run the same comparison on the replacement when you are ready.
When a retirement notice arrives.
Add the replacement as a candidate.
The task’s examples, rules, and documents are already there. Nothing is rebuilt.
Run the comparison against the live model.
Read the interval. Inspect the examples where the two disagree before you decide.
Release the winner, or keep what you have.
Either way, the record shows what was tested, how it scored, and what went live.