Disclosure: we make Loquelle, the copilot measured here. Every number below comes from our own replay log, 3 to 8 November 2026. Nobody who was on these calls is named.
A live meeting copilot has one job that a notetaker does not: it hands you a sentence to say while the other person is still waiting for it. If that sentence contains a fact nobody stated, you are the one who said it. So the question that matters is not how fluent the suggestions are but how often one is invented, and how often a correct one is thrown away by the thing that guards against inventions. We measure both. Neither number is where we want it, and both are below.
What does an invented card look like?
The dangerous kind is not nonsense. It is a line on the right subject with one false step in it: the call discussed telling clients about a change, and the card offers you a sentence saying it has already been done. The subject was raised, the names are right, and the claim is untrue. A check that only asks whether the topic came up lets it through.
How do you measure invention on a live call?
By replaying real calls. Each call in the corpus is stored as the transcript Loquelle heard and every card it showed, and each card is labelled by hand as faithful or invented by reading it against the transcript. A change to a prompt, a rule or a model is replayed against every stored call several times before it is switched on, because one run proved to be worth very little (more on that below).
| Call | Cards | Invented | What made it hard |
|---|---|---|---|
| 16 October | 30 | 8 | A one-to-one call with a document attached |
| 2 November | 16 | 3 | Four people, three languages; one invention was shown live and we added two of the same kind by hand, because one example is not a test |
| 6 November | 28 | 8 | A group call through laptop speakers, no headset |
The bar a change must clear, per call: every card labelled invented is blocked, and at most 1 in 10 faithful cards is dropped.
What are the three gates a card passes?
- Grounding in code. Every figure in a line, digits or spelled out, and every name must be found in what the copilot was actually given: your document, your brief, a page you linked, your saved notes, or the transcript. A line that fails is not shown. No model is involved, so this gate gives the same answer every time.
- The subject judge. A second model reads each surviving card against the source and must quote the words that raise the card's subject. Its verdict alone is not trusted: code then checks that the quote is really there (at least 70 percent of its words, and one three-word run in order). A judge that waves a card through with a paraphrase changes nothing.
- Question direction. Added on 8 November after the third call: a question card is kept only if the question was put to you. Your own questions, questions addressed to someone else by name, and questions quoted inside a story are dropped with their answers. This gate is also plain code.
What did the judge score, run after run?
These are totals over repeated replays of the same two calls on 6 November. The model names are the ones we run: gpt-oss-20b and gpt-oss-120b.
| Judge | 16 Oct: invented blocked | 16 Oct: faithful dropped | 2 Nov: invented blocked | 2 Nov: faithful dropped |
|---|---|---|---|---|
| gpt-oss-20b, 4 runs | 29 of 32 | 43 of 88 (49%) | 7 of 12 | 23 of 52 (44%) |
| gpt-oss-120b, 4 runs | 31 of 32 | 40 of 88 (45%) | 9 of 12 | 17 of 52 (33%) |
| gpt-oss-120b plus a second pass for "already done" claims, 5 runs | 35 of 40 | 52 of 110 (47%) | 13 of 15 | 21 of 65 (32%) |
The last row is what runs today. It blocks 88 percent of the labelled inventions on one call and 87 percent on the other, and it does so by also holding back between a third and a half of the cards that were correct. Against a target of 1 in 10, that is the number we are least happy with.
What did we try that did not work?
- Trusting one run. On 3 November the same judge, on the same cards, blocked 3 of 3 inventions in one run and 0 of 3 in another. Every figure since is a total over at least four runs.
- A majority of three judges. Three votes in parallel, a card dropped when two agree. Over five runs it blocked fewer inventions on the October call (32 of 40 against 38 of 40 for the single judge) and dropped more faithful cards on the November call (29 of 65 against 23 of 65). Off.
- Rewording the judge to ask "does this line invent something?". It blocked 40 of 40 on the October call and dropped 93 of 110 faithful cards to do it. Off.
- A looser quote check. No measurable change in either direction. Reading single cards showed why: the small model often quoted the card it was judging instead of the transcript. Off.
- A larger judge model. The only change that helped on every column, so it is the one that shipped.
What did the third call break?
Something no judge could fix. Of 9 question cards on the 6 November call, 6 were not questions put to the user at all: they were the user's own questions, or questions quoted inside a story, each answered back as if someone had asked. The desktop app hears one mixed stream through the speakers, and the prompt had told the model that everything it heard was the other side. The fix was the third gate above, in code, pinned by 26 checks, with the call added to the corpus so that no later change can quietly undo it.
What does it still get wrong?
- Between 32 and 47 percent of faithful cards are held back, against a target of 10. A held-back card costs you a useful line; it does not put a false one in your mouth, which is why we chose this side of the trade first.
- About 1 in 8 of the labelled inventions still gets through the judge on both calls (5 of 40 and 2 of 15 in the last row above).
- Six suggestions from the October call are called "never raised" by every judge we have tried, although a person reading the transcript finds them. That is a wording problem we have not solved.
- One microphone in a room with several people cannot tell their voices apart. A headset fixes it; no server change does.
What does the check cost?
The larger judge moved our cost for a live minute from $0.0014 to $0.0045. We took it, because the alternative was a cheaper product that is wrong more often in front of your client.
When will we call it done?
After three real calls in a row of different kinds (an interview, a sales call, an internal call with several people) with no invention said aloud, no promise filed under the wrong person and no transcript in the wrong language, while every earlier call in the corpus still passes. We publish these numbers because a copilot you cannot audit is not one you should use on a sales call.
Loquelle is free for 45 minutes a month: nextiqumai.com/loquelle.
Questions people ask
Why not just use a bigger model for the suggestions?
The model that writes the card and the check that guards it fail differently. A bigger writer is more fluent, which makes an invented line more convincing, not less likely to be checked. The gains in the table came from a bigger judge and from rules in code, not from the writer.
Does the check slow the suggestions down?
The code gates take no measurable time. The judge adds one model call per batch of cards, and a second short one when a card claims something is already done; if it is unavailable the cards that passed the code gates are shown, because a dead check must not silence a working meeting, and the outage is logged.
Why drop a correct card at all?
Because the two errors are not equal. A missing suggestion leaves you where you were without the tool. An invented one said aloud costs you credibility with the person you are talking to. Until the judge can tell them apart better, it leans towards silence.
Can I see what was held back on my own call?
Not card by card today. A moment where a line was withheld shows at most one grounded prompt instead, such as a question to ask back, or nothing.