becos memory benchmark

How we ran the experiment

Every question comes from a real investigation in our own work. A teammate asks it again a few days later, so memory has the earlier discussion and the code still holds the answer.

1. A frozen memory. A local copy of the becos stack loads an export of our own team's becos memory, taken on 31 October 2026. It holds 467 captured coding-agent turns from 10 August to 31 October 2026 and the 13,141 facts extracted from them. Most of those facts are about the becos plugins (Claude Code, OpenCode, Cursor, Codex), our server code and our planning notes. For each question the memory is cut at the moment the question was asked, so it holds only what existed then. Nothing touches becos.ai.
2. A workspace from that day. The repositories are checked out at the commits current when the question was asked, so a later fix can't give the answer away.
3. Three sides. With memory: Claude Code with the becos plugin, which recalls past sessions into each prompt. Without memory: no plugin. Redo the investigation: no plugin, but Claude first repeats the original request, then gets the question, to price "just do the work again".
4. The same solver everywhere. Claude Code on Sonnet 5.5, with read, search and git tools only, confined to the workspace. A run that reads outside it is thrown out.
5. Follow-ups until correct. After each answer a blind judge grades it. If it isn't fully correct, Claude gets a neutral follow-up ("a reviewer says your answer is incomplete or wrong in places; keep investigating") up to three times. The follow-up never says what is missing.
6. A key of yes/no facts. Each question's key lists only what the question asks, one checkable fact per line, every fact verified in the code at that commit or, for a plan or decision, in the session where it was made. An answer is correct when every fact is there. A wrong claim is a statement the code shows to be false; leaving something out, or adding a true fact the key doesn't mention, doesn't count. A person approves each key before any run.
7. A blind judge. Before grading, every answer on every side is rewritten by a small model to remove any mention of where information came from (past sessions, memory, dates, who said it), changing no technical claim. Claude Opus 5.5 then marks each fact yes or no and quotes the sentence it matched.
8. Two keys per question. A session key records what the original discussion concluded and grades whether recall delivered it. A truth key grades whether an answer is right. They can differ: Q10's original session said login "may work" on Windows, and the code says it doesn't.
9. What we count. Cost and tokens are running totals up to the first correct answer, including recall text and, for the redo side, the repeated investigation. Token totals count cached and uncached tokens alike, so the order runs happen in can't change them. We repeat each side three times, in alternating order.
10. How savings are computed. The tokens-saved and cost-saved bars compare memory's median cost (and tokens) to its first correct answer against the cheapest other side that also reached a correct answer: without memory, or redoing the investigation, whichever cost less. Picking the cheaper side makes the saving as small as it can honestly be. Runs that never got a correct answer are left out, because a cheap wrong answer isn't a saving.