Does becos memory help Claude Code? We took questions our engineering team had already answered in previous work and asked Claude Code to answer them again, with and without becos memory. We measured two things: whether Claude got the answer right, and how many tokens and dollars it took to get there.
A small sample: 10 questions from our own team's work, with three runs per setup. The 82% saving includes one question (Q5) measured against a single run, because only 1 of 3 runs without memory got it right. See Limitations.
Every question came from a real investigation in our team's work. We then asked Claude the same question again, as if a teammate had returned to the problem a few days later.
Each question ran on two or three of these setups.
We used a frozen export of our team's becos memory, taken on 31 October 2026. It contained 467 captured coding-agent turns from 10 August to 31 October 2026 and 13,141 facts extracted from them. Most of those facts are about the becos plugins (Claude Code, OpenCode, Cursor, Codex), our server code and our planning notes.
For each question, we cut the memory off at the time the question was originally asked, so it contained only information available at that point. A local copy of the becos stack served it, and nothing in the experiment touched becos.ai.
Repositories were checked out at the commits that existed when each question was originally asked. This prevented later code changes from revealing the answer.
Every run used Claude Code on Sonnet 5.5, with read, search and git tools only, restricted to the relevant workspace. Runs that accessed anything outside it were discarded.
Each question's answer key lists only what the question asks, one checkable yes/no fact per line. Every fact was verified in the code at that commit or, for a plan or decision, in the session where it was made, and a person approved each key before any run. An answer is correct when every fact is there. A wrong claim is a statement the code shows to be false; leaving something out, or adding a true fact the key doesn't mention, doesn't count.
Before grading, a separate small model rewrote every answer on every side to remove references to memory, past sessions, dates and information sources, changing no technical claim. Claude Opus 5.5 then evaluated each answer fact by fact and quoted the sentence it matched.
If an answer wasn't fully correct, Claude received a neutral follow-up ("a reviewer says your answer is incomplete or wrong in places; keep investigating"), up to three times. The follow-up never said what was missing.
Each question had two keys. A session key captured what the original discussion concluded, and graded whether recall delivered it. A truth key captured what was actually correct according to the code or original decision, and graded the answers. The two could differ: Q10's original discussion said login might work on Windows, while the code showed that it didn't.
We counted tokens and dollars up to the first correct answer, including recall text and any repeated investigation. Token counts included cached and uncached tokens alike, so the order of the runs can't change them. We repeated each setup three times in alternating order.
We compared memory's median cost and token usage against the alternatives that also reached a correct answer. We calculated cost savings and token savings separately, using the alternative with the lowest median cost for the cost comparison and the alternative with the lowest median token usage for the token comparison. Choosing the best alternative makes the saving as small as it can honestly be.
Runs that never produced a correct answer were excluded. A cheap wrong answer isn't a saving.
Each card shows the latest complete round: how many runs reached a fully correct answer, and the median cost and tokens to get there. Cost is only compared between runs that got the answer right.
These questions tested whether Claude could recover information that existed only in previous discussions: experimental findings, plans that were never implemented, and decisions made during earlier investigations. The code and documentation didn't contain the answers.
Memory was correct on the first answer in all 3 runs, at $0.09-0.20 and 8 tool calls.
DetailsMemory was correct on the first answer in all 3 runs, at $0.05-0.07 and 4-7 tool calls.
DetailsMemory reached a correct answer in 2 of 3 runs ($0.20 and $0.48). Without memory, no run did, after $0.23-0.25 each.
DetailsThe answer can be worked out from the code, but it took a long cross-repo investigation. Memory carries the conclusion; the other sides have to redo the work.
Every side reached a correct answer; memory did it for about 60% less money and 78% fewer tokens.
DetailsMemory was correct on the first answer in all 3 runs, at $0.12-0.15 and 6-8 tool calls.
DetailsMemory was correct on the first answer in all 3 runs, at $0.10-0.16 and 9-18 tool calls.
DetailsWhy we included these questions: the answers are already recorded in commit messages, changelogs or decision logs. An agent can find them without memory, so these questions help measure whether becos adds value when the information is already accessible.
Both sides were correct on the first answer in every run, at $0.04-0.05.
DetailsMemory was correct on the first answer in all 3 runs, at $0.08-0.15; without memory 3 of 3, one on its second answer, at $0.13-0.24.
DetailsWhy we included these questions: the answers were directly available in the code. These questions tested whether memory adds unnecessary overhead when a simple code search is enough.
Without memory: correct on the first answer in every run, at $0.09-0.20. With memory: every first answer repeated a conclusion the original session had withdrawn, and the runs recovered after 2-3 answers by reading the code, at $0.14-0.23.
DetailsAll three sides were correct on the first answer in every run. Memory cost $0.13, no memory $0.14 and redoing the investigation $0.10, so memory cost 28% more than the cheapest side.
DetailsCategory 1
These questions tested whether Claude could recover information that existed only in previous discussions: experimental findings, plans that were never implemented, and decisions made during earlier investigations. The code and documentation didn't contain the answers.
Session-only: a cause found by experiment
Question. “When I call vonic_query from Claude Code through the becos connector it says outcome answered but there's no answer text. Why, and what's the fix?”
The cause isn't in any repository: Claude Code shows the model only a tool's structured receipt and drops the text that holds the answer. The original session saw it live. Asked with no code, Claude scored 0 of 2.
Original investigation: 2026-10-05. · Workspace: the API server and the Claude Code plugin, at the commits current when the question was asked
Of the 2 facts, recall's 8,896 characters delivered 2 fully, 0 partly, and 0 not at all. Both facts were in the recalled memory.
Yes/no facts; Claude Code's behaviour verified with a local MCP probe on 2.1.316.
| Side | Correct | Cost to correct | Tokens to correct | 1st answer coverage | 1st answer wrong claims | 1st answer cost · tokens | Runs that never got there |
|---|---|---|---|---|---|---|---|
| With memory | 3 of 3 (answer 1, 1, 1) | $0.10, $0.09, $0.20 | 83k, 103k, 120k | 100%, 100%, 100% | 0, 0, 0 | $0.10 · 103k | — |
| Without memory | 0 of 3 | — | — | 0%, 0%, 0% | 0, 0, 0 | $0.18 · 220k | $0.67, $0.65, $0.60 · 1.50M, 1.34M, 1.34M |
| Redo the investigation | 0 of 3 | — | — | 0%, 0%, 0% | 1, 1, 2 | $0.28 · 413k | $0.89, $0.81, $0.85 · 2.39M, 1.90M, 2.06M |
One line per answer. The marks are the key facts in order: ✓ present, ✗ missing. "% of key" counts ✓ as 1 and ½ as a half. An answer is correct when every required point is ✓; the run stops there. Cost, tokens and tool calls are the run's totals.
| Run | Each answer | Run cost | Run tokens | Tool calls |
|---|---|---|---|---|
| Q1-on-r1 | Answer 1 ✓✓ 100% of key · 0 wrong claims · correct | $0.10 | 83k | 8 |
| Q1-on-r2 | Answer 1 ✓✓ 100% of key · 0 wrong claims · correct | $0.09 | 103k | 8 |
| Q1-on-r3 | Answer 1 ✓✓ 100% of key · 0 wrong claims · correct | $0.20 | 120k | 8 |
| Q1-off-r1 | Answer 1 ✗✗ 0% of key · 0 wrong claims Answer 2 ✗✗ 0% of key · 1 wrong claim Answer 3 ✗✗ 0% of key · 2 wrong claims Answer 4 ✗✗ 0% of key · 2 wrong claims | $0.67 | 1.50M | 47 |
| Q1-off-r2 | Answer 1 ✗✗ 0% of key · 0 wrong claims Answer 2 ✗✗ 0% of key · 3 wrong claims Answer 3 ✗✗ 0% of key · 3 wrong claims Answer 4 ✗✗ 0% of key · 3 wrong claims | $0.65 | 1.34M | 44 |
| Q1-off-r3 | Answer 1 ✗✗ 0% of key · 0 wrong claims Answer 2 ✗✗ 0% of key · 2 wrong claims Answer 3 ✗✗ 0% of key · 2 wrong claims Answer 4 ✗✗ 0% of key · 3 wrong claims | $0.60 | 1.34M | 45 |
| Q1-off-full-r1 | Answer 1 ✗✗ 0% of key · 1 wrong claim Answer 2 ✗✗ 0% of key · 3 wrong claims Answer 3 ✗✗ 0% of key · 1 wrong claim Answer 4 ✗✗ 0% of key · 1 wrong claim | $0.89 | 2.39M | 62 |
| Q1-off-full-r2 | Answer 1 ✗✗ 0% of key · 1 wrong claim Answer 2 ✗✗ 0% of key · 2 wrong claims Answer 3 ✗✗ 0% of key · 2 wrong claims Answer 4 ✗✗ 0% of key · 2 wrong claims | $0.81 | 1.90M | 49 |
| Q1-off-full-r3 | Answer 1 ✗✗ 0% of key · 2 wrong claims Answer 2 ✗✗ 0% of key · 2 wrong claims Answer 3 ✗✗ 0% of key · 3 wrong claims Answer 4 ✗✗ 0% of key · 3 wrong claims | $0.85 | 2.06M | 59 |
Session-only: a plan never built
Question. “install.sh just says 'already signed in' even when someone's becos sign-in has expired. What did we plan to do about that?”
The plan was made in a session and never built, so no code or doc contains it. Asked with no code, Claude scored 0 of 2.
Original investigation: 2026-10-31. · Workspace: the public plugin marketplace and the Claude Code plugin, at the commits current when the question was asked
Of the 2 facts, recall's 6,539 characters delivered 1 fully, 1 partly, and 0 not at all. Recall carried the re-sign-in rule in full and the check command in part.
Yes/no facts from the session's final plan.
| Side | Correct | Cost to correct | Tokens to correct | 1st answer coverage | 1st answer wrong claims | 1st answer cost · tokens | Runs that never got there |
|---|---|---|---|---|---|---|---|
| With memory | 3 of 3 (answer 1, 1, 1) | $0.05, $0.07, $0.06 | 37k, 81k, 52k | 100%, 100%, 100% | 0, 0, 0 | $0.06 · 52k | — |
| Without memory | 0 of 3 | — | — | 0%, 0%, 0% | 2, 0, 1 | $0.11 · 93k | $0.38, $0.39, $0.37 · 664k, 709k, 639k |
| Redo the investigation | 0 of 3 | — | — | 0%, 0%, 0% | 2, 0, 0 | $0.22 · 261k | $0.57, $0.72, $0.34 · 1.09M, 1.58M, 625k |
One line per answer. The marks are the key facts in order: ✓ present, ✗ missing. "% of key" counts ✓ as 1 and ½ as a half. An answer is correct when every required point is ✓; the run stops there. Cost, tokens and tool calls are the run's totals.
| Run | Each answer | Run cost | Run tokens | Tool calls |
|---|---|---|---|---|
| Q2-on-r1 | Answer 1 ✓✓ 100% of key · 0 wrong claims · correct | $0.05 | 37k | 4 |
| Q2-on-r2 | Answer 1 ✓✓ 100% of key · 0 wrong claims · correct | $0.07 | 81k | 7 |
| Q2-on-r3 | Answer 1 ✓✓ 100% of key · 0 wrong claims · correct | $0.06 | 52k | 6 |
| Q2-off-r1 | Answer 1 ✗✗ 0% of key · 2 wrong claims Answer 2 ✗✗ 0% of key · 1 wrong claim Answer 3 ✗✗ 0% of key · 1 wrong claim Answer 4 ✗✗ 0% of key · 1 wrong claim | $0.38 | 664k | 39 |
| Q2-off-r2 | Answer 1 ✗✗ 0% of key · 0 wrong claims Answer 2 ✗✗ 0% of key · 0 wrong claims Answer 3 ✗✗ 0% of key · 0 wrong claims Answer 4 ✗✗ 0% of key · 1 wrong claim | $0.39 | 709k | 41 |
| Q2-off-r3 | Answer 1 ✗✗ 0% of key · 1 wrong claim Answer 2 ✗✗ 0% of key · 1 wrong claim Answer 3 ✗✗ 0% of key · 1 wrong claim Answer 4 ✗✗ 0% of key · 0 wrong claims | $0.37 | 639k | 38 |
| Q2-off-full-r1 | Answer 1 ✗✗ 0% of key · 2 wrong claims Answer 2 ✗✗ 0% of key · 2 wrong claims Answer 3 ✗✗ 0% of key · 1 wrong claim Answer 4 ✗✗ 0% of key · 1 wrong claim | $0.57 | 1.09M | 47 |
| Q2-off-full-r2 | Answer 1 ✗✗ 0% of key · 0 wrong claims Answer 2 ✗✗ 0% of key · 1 wrong claim Answer 3 ✗✗ 0% of key · 2 wrong claims Answer 4 ✗✗ 0% of key · 1 wrong claim | $0.72 | 1.58M | 52 |
| Q2-off-full-r3 | Answer 1 ✗✗ 0% of key · 0 wrong claims Answer 2 ✗✗ 0% of key · 1 wrong claim Answer 3 ✗✓ 50% of key · 1 wrong claim Answer 4 ✗✓ 50% of key · 2 wrong claims | $0.34 | 625k | 33 |
Session-only: an approach rejected in discussion
Question. “When we moved the plugin repos (Claude, Codex, OpenCode) to the becos-ai org, which approaches came up, and was each repo actually transferred or copied?”
Git shows the outcome (two repos were copied and froze). Only the session shows that forking was considered and rejected, and that the Codex repo was transferred.
Original investigation: 2026-09-15 to 09-16. · Workspace: the Claude Code plugin, at the commits current when the question was asked
Of the 3 facts, recall's 8,443 characters delivered 2 fully, 1 partly, and 0 not at all. Recall carried the fork rejection and the Codex transfer in full, and the copies in part; the copies are in git anyway.
Commit xxxxxxx's message says all three old repos 'were copied, not transferred'. That is true for Claude and OpenCode and false for Codex: GitHub still redirects the old Codex URL, and its org repo predates the move. The message was written with Claude, so it is no more authoritative than the session.
Memory answers said Codex was transferred, then met the commit and either repeated it or called it a conflict they couldn't settle. We grade the Codex fact as present when an answer states the transfer, even if it also notes the commit disagrees, and count 'Codex was copied' as a wrong claim only when it is stated as fact without the conflict.
With the question asking 'was each repo transferred or copied?', the agent had to choose. In 6 of the 10 memory answers, it weighed the two records and chose the commit ('the copied account is probably right'). It can't open the cited memory record in this benchmark, so it had no way to check which one was right.
The question now asks transferred-or-copied for each repo; graded with the Codex rule below.
| Side | Correct | Cost to correct | Tokens to correct | 1st answer coverage | 1st answer wrong claims | 1st answer cost · tokens | Runs that never got there |
|---|---|---|---|---|---|---|---|
| With memory | 2 of 3 (answer 4, 2) | $0.48, $0.20 | 615k, 211k | 67%, 67%, 67% | 0, 0, 0 | $0.07 · 44k | $0.36 · 479k |
| Without memory | 0 of 3 | — | — | 0%, 0%, 0% | 0, 0, 0 | $0.06 · 62k | $0.24, $0.23, $0.25 · 351k, 291k, 412k |
One line per answer. The marks are the key facts in order: ✓ present, ✗ missing. "% of key" counts ✓ as 1 and ½ as a half. An answer is correct when every required point is ✓; the run stops there. Cost, tokens and tool calls are the run's totals.
| Run | Each answer | Run cost | Run tokens | Tool calls |
|---|---|---|---|---|
| Q3-on-r1 | Answer 1 ✓✗✓ 66% of key · 0 wrong claims Answer 2 ✓✓✗ 66% of key · 0 wrong claims Answer 3 ✓✓✗ 66% of key · 0 wrong claims Answer 4 ✓✓✗ 66% of key · 1 wrong claim | $0.36 | 479k | 18 |
| Q3-on-r2 | Answer 1 ✓✗✓ 66% of key · 0 wrong claims Answer 2 ✓✓✗ 66% of key · 0 wrong claims Answer 3 ✓✓✗ 66% of key · 0 wrong claims Answer 4 ✓✓✓ 100% of key · 0 wrong claims · correct | $0.48 | 615k | 21 |
| Q3-on-r3 | Answer 1 ✓✓✗ 66% of key · 0 wrong claims Answer 2 ✓✓✓ 100% of key · 0 wrong claims · correct | $0.20 | 211k | 11 |
| Q3-off-r1 | Answer 1 ✗✗✗ 0% of key · 0 wrong claims Answer 2 ✗✓✗ 33% of key · 1 wrong claim Answer 3 ✗✓✗ 33% of key · 3 wrong claims Answer 4 ✗✓✗ 33% of key · 1 wrong claim | $0.24 | 351k | 23 |
| Q3-off-r2 | Answer 1 ✗✗✗ 0% of key · 0 wrong claims Answer 2 ✗✓✗ 33% of key · 2 wrong claims Answer 3 ✗✓✗ 33% of key · 3 wrong claims Answer 4 ✗✓✗ 33% of key · 1 wrong claim | $0.23 | 291k | 17 |
| Q3-off-r3 | Answer 1 ✗✗✗ 0% of key · 0 wrong claims Answer 2 ✗✗✗ 0% of key · 0 wrong claims Answer 3 ✗✓✗ 33% of key · 2 wrong claims Answer 4 ✗✗✗ 0% of key · 2 wrong claims | $0.25 | 412k | 27 |
Earlier wording ('which approaches went wrong?'), re-graded with the Codex rule; follow-ups were decided by the stricter first grading.
| Side | Correct | Cost to correct | Tokens to correct | 1st answer coverage | 1st answer wrong claims | 1st answer cost · tokens | Runs that never got there |
|---|---|---|---|---|---|---|---|
| With memory | 3 of 3 (answer 1, 2, 2) | $0.07, $0.14, $0.15 | 42k, 120k, 126k | 100%, 67%, 67% | 1, 0, 0 | $0.05 · 27k | — |
| Without memory | 0 of 3 | — | — | 0%, 33%, 0% | 0, 1, 0 | $0.07 · 59k | $0.41, $0.27, $0.29 · 698k, 348k, 458k |
One line per answer. The marks are the key facts in order: ✓ present, ✗ missing. "% of key" counts ✓ as 1 and ½ as a half. An answer is correct when every required point is ✓; the run stops there. Cost, tokens and tool calls are the run's totals.
| Run | Each answer | Run cost | Run tokens | Tool calls |
|---|---|---|---|---|
| Q3-on-r1 | Answer 1 ✓✓✓ 100% of key · 1 wrong claim · correct Answer 2 ✓✓✓ 100% of key · 0 wrong claims · correct Answer 3 ✓✓✓ 100% of key · 0 wrong claims · correct Answer 4 ✓✓✓ 100% of key · 0 wrong claims · correct | $0.31 | 340k | 13 |
| Q3-on-r2 | Answer 1 ✓✗✓ 66% of key · 0 wrong claims Answer 2 ✓✓✓ 100% of key · 1 wrong claim · correct Answer 3 ✓✓✓ 100% of key · 0 wrong claims · correct Answer 4 ✓✓✓ 100% of key · 1 wrong claim · correct | $0.30 | 278k | 10 |
| Q3-on-r3 | Answer 1 ✓✗✓ 66% of key · 0 wrong claims Answer 2 ✓✓✓ 100% of key · 0 wrong claims · correct | $0.15 | 126k | 6 |
| Q3-off-r1 | Answer 1 ✗✗✗ 0% of key · 0 wrong claims Answer 2 ✗✓✗ 33% of key · 1 wrong claim Answer 3 ✗✓✗ 33% of key · 1 wrong claim Answer 4 ✗✓✗ 33% of key · 2 wrong claims | $0.41 | 698k | 29 |
| Q3-off-r2 | Answer 1 ✗✓✗ 33% of key · 1 wrong claim Answer 2 ✗✓✗ 33% of key · 1 wrong claim Answer 3 ✗✓✗ 33% of key · 1 wrong claim Answer 4 ✗✓✗ 33% of key · 1 wrong claim | $0.27 | 348k | 17 |
| Q3-off-r3 | Answer 1 ✗✗✗ 0% of key · 0 wrong claims Answer 2 ✗✓✗ 33% of key · 1 wrong claim Answer 3 ✗✓✗ 33% of key · 1 wrong claim Answer 4 ✗✓✗ 33% of key · 1 wrong claim | $0.29 | 458k | 19 |
Category 2
The answer can be worked out from the code, but it took a long cross-repo investigation. Memory carries the conclusion; the other sides have to redo the work.
Cross-repo analysis
Question. “Team activity never shows any todo entries. Why aren't todos being stored in becos_xxxxxx, and what is the right fix?”
The cause sits in another repository (becos_xxxxxx), and the fix was designed in a past session. Memory should help most here.
Original investigation: 2026-10-26: 14 tool calls, 2.2 minutes. · Workspace: the Claude Code plugin, the memory server and the deployment repo, at the commits current when the question was asked
Of the 2 facts, recall's 9,375 characters delivered 2 fully, 0 partly, and 0 not at all. Both the cause and the fix were in the recalled memory, which is why the memory runs answered after 5-7 tool calls.
Yes/no facts checked in code; every answer rewritten neutrally before grading.
| Side | Correct | Cost to correct | Tokens to correct | 1st answer coverage | 1st answer wrong claims | 1st answer cost · tokens | Runs that never got there |
|---|---|---|---|---|---|---|---|
| With memory | 3 of 3 (answer 1, 1, 1) | $0.11, $0.10, $0.09 | 97k, 72k, 71k | 100%, 100%, 100% | 0, 0, 0 | $0.10 · 72k | — |
| Without memory | 3 of 3 (answer 1, 2, 1) | $0.25, $0.49, $0.19 | 323k, 873k, 253k | 100%, 0%, 100% | 0, 1, 0 | $0.21 · 253k | — |
| Redo the investigation | 3 of 3 (answer 1, 1, 1) | $0.30, $0.24, $0.26 | 410k, 315k, 363k | 100%, 100%, 100% | 0, 0, 0 | $0.26 · 363k | — |
One line per answer. The marks are the key facts in order: ✓ present, ✗ missing. "% of key" counts ✓ as 1 and ½ as a half. An answer is correct when every required point is ✓; the run stops there. Cost, tokens and tool calls are the run's totals.
| Run | Each answer | Run cost | Run tokens | Tool calls |
|---|---|---|---|---|
| Q4-on-r1 | Answer 1 ✓✓ 100% of key · 0 wrong claims · correct | $0.11 | 97k | 7 |
| Q4-on-r2 | Answer 1 ✓✓ 100% of key · 0 wrong claims · correct | $0.10 | 72k | 6 |
| Q4-on-r3 | Answer 1 ✓✓ 100% of key · 0 wrong claims · correct | $0.09 | 71k | 5 |
| Q4-off-r1 | Answer 1 ✓✓ 100% of key · 0 wrong claims · correct | $0.25 | 323k | 28 |
| Q4-off-r2 | Answer 1 ✗✗ 0% of key · 1 wrong claim Answer 2 ✓✓ 100% of key · 0 wrong claims · correct | $0.49 | 873k | 43 |
| Q4-off-r3 | Answer 1 ✓✓ 100% of key · 0 wrong claims · correct | $0.19 | 253k | 19 |
| Q4-off-full-r1 | Answer 1 ✓✓ 100% of key · 0 wrong claims · correct | $0.30 | 410k | 23 |
| Q4-off-full-r2 | Answer 1 ✓✓ 100% of key · 0 wrong claims · correct | $0.24 | 315k | 18 |
| Q4-off-full-r3 | Answer 1 ✓✓ 100% of key · 0 wrong claims · correct | $0.26 | 363k | 24 |
Cross-repo analysis
Question. “If we build another adapter, how does recalled memory actually reach the model in our Claude Code plugin versus the OpenCode plugin, and is there a size limit on the Claude side we have to design around?”
How each plugin delivers recall is in the code. The Claude Code limit on total hook output was known only from past sessions at the asked time.
Original investigation: 2026-09-28: 69 tool calls, 10.2 minutes. · Workspace: the Claude Code plugin and the OpenCode plugin, at the commits current when the question was asked
Of the 3 facts, recall's 9,206 characters delivered 1 fully, 2 partly, and 0 not at all. Recall carried the hook-output limit in full and the two delivery paths in part; those are in the code anyway.
The hook-output cap is in Claude Code's public docs, and asked with no code and no tools, Claude recalled 'roughly 10,000 characters' in all 3 tries. We expected Q5 to behave like Q10, a control.
It didn't. With the repositories open, the no-memory runs looked for the limit in the repo, didn't find it, and reported it as unknown. Some suggested testing whether 20,000 characters was safe. They trusted the repo over what they knew.
Memory supplied the limit from the original session, so every memory answer had it from the start.
Yes/no facts; F3 checked against Claude Code's hooks documentation.
| Side | Correct | Cost to correct | Tokens to correct | 1st answer coverage | 1st answer wrong claims | 1st answer cost · tokens | Runs that never got there |
|---|---|---|---|---|---|---|---|
| With memory | 3 of 3 (answer 1, 1, 1) | $0.15, $0.12, $0.13 | 120k, 88k, 93k | 100%, 100%, 100% | 0, 0, 0 | $0.13 · 93k | — |
| Without memory | 1 of 3 (answer 4) | $0.61 | 1.09M | 67%, 67%, 67% | 0, 1, 2 | $0.13 · 129k | $0.75, $0.59 · 1.45M, 875k |
| Redo the investigation | 3 of 3 (answer 2, 1, 4) | $1.17, $1.07, $0.98 | 2.24M, 2.22M, 1.76M | 67%, 100%, 67% | 0, 0, 0 | $0.90 · 1.34M | — |
One line per answer. The marks are the key facts in order: ✓ present, ✗ missing. "% of key" counts ✓ as 1 and ½ as a half. An answer is correct when every required point is ✓; the run stops there. Cost, tokens and tool calls are the run's totals.
| Run | Each answer | Run cost | Run tokens | Tool calls |
|---|---|---|---|---|
| Q5-on-r1 | Answer 1 ✓✓✓ 100% of key · 0 wrong claims · correct | $0.15 | 120k | 8 |
| Q5-on-r2 | Answer 1 ✓✓✓ 100% of key · 0 wrong claims · correct | $0.12 | 88k | 6 |
| Q5-on-r3 | Answer 1 ✓✓✓ 100% of key · 0 wrong claims · correct | $0.13 | 93k | 8 |
| Q5-off-r1 | Answer 1 ✓✓✗ 66% of key · 0 wrong claims Answer 2 ✓✓✗ 66% of key · 1 wrong claim Answer 3 ✓✓✗ 66% of key · 0 wrong claims Answer 4 ✓✓✗ 66% of key · 1 wrong claim | $0.75 | 1.45M | 47 |
| Q5-off-r2 | Answer 1 ✓✓✗ 66% of key · 1 wrong claim Answer 2 ✓✓✗ 66% of key · 3 wrong claims Answer 3 ✓✓✗ 66% of key · 2 wrong claims Answer 4 ✓✓✓ 100% of key · 3 wrong claims · correct | $0.61 | 1.09M | 40 |
| Q5-off-r3 | Answer 1 ✓✓✗ 66% of key · 2 wrong claims Answer 2 ✓✓✗ 66% of key · 2 wrong claims Answer 3 ✓✓✗ 66% of key · 0 wrong claims Answer 4 ✓✓✗ 66% of key · 1 wrong claim | $0.59 | 875k | 37 |
| Q5-off-full-r1 | Answer 1 ✓✓✗ 66% of key · 0 wrong claims Answer 2 ✓✓✓ 100% of key · 0 wrong claims · correct | $1.17 | 2.24M | 56 |
| Q5-off-full-r2 | Answer 1 ✓✓✓ 100% of key · 0 wrong claims · correct | $1.07 | 2.22M | 53 |
| Q5-off-full-r3 | Answer 1 ✓✓✗ 66% of key · 0 wrong claims Answer 2 ✓✓✗ 66% of key · 0 wrong claims Answer 3 ✓✓✗ 66% of key · 0 wrong claims Answer 4 ✓✓✓ 100% of key · 0 wrong claims · correct | $0.98 | 1.76M | 56 |
Cross-repo analysis
Question. “What exactly is wrong with how the plugin decides an entity-history answer is empty, and what is the smallest correct fix across the plugin copies?”
The answer can be worked out from the plugin code and its own decision log, but the original took a trace across six repositories. Memory holds the conclusion.
Original investigation: 2026-10-04: 119 tool calls. · Workspace: the Claude Code plugin, the Cursor plugin, the Codex plugin, the OpenCode plugin, the API server and the memory server, at the commits current when the question was asked
Of the 3 facts, recall's 9,219 characters delivered 2 fully, 1 partly, and 0 not at all. Recall carried the bug and the rule in full, and the fix without naming the three copies.
Yes/no facts checked against the code; the shipped fix made the check an exact match only.
| Side | Correct | Cost to correct | Tokens to correct | 1st answer coverage | 1st answer wrong claims | 1st answer cost · tokens | Runs that never got there |
|---|---|---|---|---|---|---|---|
| With memory | 3 of 3 (answer 1, 1, 1) | $0.14, $0.16, $0.10 | 142k, 163k, 81k | 100%, 100%, 100% | 0, 0, 0 | $0.14 · 142k | — |
| Without memory | 0 of 3 | — | — | 33%, 33%, 0% | 2, 1, 2 | $0.20 · 188k | $0.61, $0.64, $0.83 · 1.14M, 1.35M, 1.69M |
| Redo the investigation | 3 of 3 (answer 1, 1, 1) | $0.51, $0.49, $0.57 | 551k, 615k, 617k | 100%, 100%, 100% | 0, 0, 0 | $0.51 · 615k | — |
One line per answer. The marks are the key facts in order: ✓ present, ✗ missing. "% of key" counts ✓ as 1 and ½ as a half. An answer is correct when every required point is ✓; the run stops there. Cost, tokens and tool calls are the run's totals.
| Run | Each answer | Run cost | Run tokens | Tool calls |
|---|---|---|---|---|
| Q6-on-r1 | Answer 1 ✓✓✓ 100% of key · 0 wrong claims · correct | $0.14 | 142k | 14 |
| Q6-on-r2 | Answer 1 ✓✓✓ 100% of key · 0 wrong claims · correct | $0.16 | 163k | 18 |
| Q6-on-r3 | Answer 1 ✓✓✓ 100% of key · 0 wrong claims · correct | $0.10 | 81k | 9 |
| Q6-off-r1 | Answer 1 ✓✗✗ 33% of key · 2 wrong claims Answer 2 ✓✗✓ 66% of key · 0 wrong claims Answer 3 ✓✗✓ 66% of key · 0 wrong claims Answer 4 ✓✗✗ 33% of key · 1 wrong claim | $0.61 | 1.14M | 54 |
| Q6-off-r2 | Answer 1 ✗✗✓ 33% of key · 1 wrong claim Answer 2 ✗✓✓ 66% of key · 1 wrong claim Answer 3 ✗✗✓ 33% of key · 0 wrong claims Answer 4 ✗✓✓ 66% of key · 1 wrong claim | $0.64 | 1.35M | 56 |
| Q6-off-r3 | Answer 1 ✗✗✗ 0% of key · 2 wrong claims Answer 2 ✗✗✗ 0% of key · 3 wrong claims Answer 3 ✗✓✗ 33% of key · 1 wrong claim Answer 4 ✗✓✗ 33% of key · 1 wrong claim | $0.83 | 1.69M | 61 |
| Q6-off-full-r1 | Answer 1 ✓✓✓ 100% of key · 0 wrong claims · correct | $0.51 | 551k | 41 |
| Q6-off-full-r2 | Answer 1 ✓✓✓ 100% of key · 0 wrong claims · correct | $0.49 | 615k | 47 |
| Q6-off-full-r3 | Answer 1 ✓✓✓ 100% of key · 0 wrong claims · correct | $0.57 | 617k | 49 |
Category 3
Why we included these questions: the answers are already recorded in commit messages, changelogs or decision logs. An agent can find them without memory, so these questions help measure whether becos adds value when the information is already accessible.
Control: answer written in the changelog
Question. “What did a teammate decide when the static instructions were moved out of the per-prompt recall hook?”
The whole answer is in the plugin's CHANGELOG 0.28.0 and decision log, so this shows what written records do without memory.
Original investigation: 2026-10-02. · Workspace: the Claude Code plugin, at the commits current when the question was asked
Of the 3 facts, recall's 9,300 characters delivered 1 fully, 2 partly, and 0 not at all. Recall carried part of it; the changelog carried all of it.
The plugin keeps a decision log (docs/DECISIONS.md) and a CHANGELOG, and the team's decision is written in both, in full. Every no-memory run searched git log and the repo for the change, opened the DECISIONS.md entry, and was done in 4 tool calls.
Memory got there in 2-3 tool calls, but recall carried only part of the decision, so the runs still checked the written record. Two tool calls saved is worth less than a cent: $0.05 with memory against $0.04 without.
So a question whose answer a team already wrote down, where the agent naturally looks, is a control: memory can only repeat what the record says.
Written-records control; with and without memory only.
| Side | Correct | Cost to correct | Tokens to correct | 1st answer coverage | 1st answer wrong claims | 1st answer cost · tokens | Runs that never got there |
|---|---|---|---|---|---|---|---|
| With memory | 3 of 3 (answer 1, 1, 1) | $0.06, $0.05, $0.05 | 40k, 40k, 41k | 100%, 100%, 100% | 0, 0, 0 | $0.05 · 40k | — |
| Without memory | 3 of 3 (answer 1, 1, 1) | $0.04, $0.04, $0.04 | 41k, 41k, 41k | 100%, 100%, 100% | 0, 0, 0 | $0.04 · 41k | — |
One line per answer. The marks are the key facts in order: ✓ present, ✗ missing. "% of key" counts ✓ as 1 and ½ as a half. An answer is correct when every required point is ✓; the run stops there. Cost, tokens and tool calls are the run's totals.
| Run | Each answer | Run cost | Run tokens | Tool calls |
|---|---|---|---|---|
| Q7-on-r1 | Answer 1 ✓✓✓ 100% of key · 0 wrong claims · correct | $0.05 | 40k | 2 |
| Q7-on-r2 | Answer 1 ✓✓✓ 100% of key · 0 wrong claims · correct | $0.05 | 40k | 2 |
| Q7-on-r3 | Answer 1 ✓✓✓ 100% of key · 0 wrong claims · correct | $0.05 | 41k | 3 |
| Q7-off-r1 | Answer 1 ✓✓✓ 100% of key · 0 wrong claims · correct | $0.04 | 41k | 4 |
| Q7-off-r2 | Answer 1 ✓✓✓ 100% of key · 0 wrong claims · correct | $0.04 | 41k | 4 |
| Q7-off-r3 | Answer 1 ✓✓✓ 100% of key · 0 wrong claims · correct | $0.04 | 41k | 4 |
Control: answer written in a commit and the team's TODO list
Question. “A single recall answer lists the same fact several times in slightly different words. Why does memory store reworded copies, and what have we done to fix that?”
The cause and the fix are written in becos_xxxxxx's commit xxxxxxx and in our TODO list, so this shows what written records do without memory.
Original investigation: 2026-10-23: 29 tool calls, about 5 minutes. · Workspace: the memory server and our planning repo, at the commits current when the question was asked
Of the 2 facts, recall's 8,904 characters delivered 0 fully, 2 partly, and 0 not at all. Recall carried the cause and the fix in part; the commit and the TODO list carried all of it.
Written-records control; with and without memory only. 'Fixed in the plugins' or 'at both ends' counts as a wrong claim.
| Side | Correct | Cost to correct | Tokens to correct | 1st answer coverage | 1st answer wrong claims | 1st answer cost · tokens | Runs that never got there |
|---|---|---|---|---|---|---|---|
| With memory | 3 of 3 (answer 1, 1, 1) | $0.15, $0.08, $0.11 | 185k, 97k, 156k | 100%, 100%, 100% | 0, 0, 0 | $0.11 · 156k | — |
| Without memory | 3 of 3 (answer 2, 1, 1) | $0.24, $0.14, $0.13 | 267k, 139k, 170k | 50%, 100%, 100% | 0, 0, 0 | $0.13 · 139k | — |
One line per answer. The marks are the key facts in order: ✓ present, ✗ missing. "% of key" counts ✓ as 1 and ½ as a half. An answer is correct when every required point is ✓; the run stops there. Cost, tokens and tool calls are the run's totals.
| Run | Each answer | Run cost | Run tokens | Tool calls |
|---|---|---|---|---|
| Q8-on-r1 | Answer 1 ✓✓ 100% of key · 0 wrong claims · correct | $0.15 | 185k | 15 |
| Q8-on-r2 | Answer 1 ✓✓ 100% of key · 0 wrong claims · correct | $0.08 | 97k | 6 |
| Q8-on-r3 | Answer 1 ✓✓ 100% of key · 0 wrong claims · correct | $0.11 | 156k | 12 |
| Q8-off-r1 | Answer 1 ✗✓ 50% of key · 0 wrong claims Answer 2 ✓✓ 100% of key · 0 wrong claims · correct | $0.24 | 267k | 22 |
| Q8-off-r2 | Answer 1 ✓✓ 100% of key · 0 wrong claims · correct | $0.14 | 139k | 13 |
| Q8-off-r3 | Answer 1 ✓✓ 100% of key · 0 wrong claims · correct | $0.13 | 170k | 13 |
Category 4
Why we included these questions: the answers were directly available in the code. These questions tested whether memory adds unnecessary overhead when a simple code search is enough.
Control, and a case where memory misled
Question. “What did a teammate conclude about how the author on a team-activity entry or fact is determined?”
The answer is in becos_xxxxxx's authorship code. The source session first reached an incorrect early conclusion, then corrected itself seven minutes later.
Original investigation: 2026-10-14. · Workspace: the Claude Code plugin, the memory server and the API server, at the commits current when the question was asked
Of the 2 facts, recall's 9,300 characters delivered 0 fully, 0 partly, and 2 not at all. Recall delivered the session's withdrawn early conclusion, labelled current, and not the correction.
How the author is chosen is a few lines of code in becos_xxxxxx (coding/authorship.py). A search for 'author' across the three repos leads straight to it, and every no-memory run read it and answered correctly in 9-15 tool calls.
At best memory could have saved that one search. What recall actually delivered was the session's first, withdrawn conclusion, so the memory runs had to read the same code anyway (10-16 tool calls), and only after their first answer was graded wrong.
When the code states the answer plainly, memory can at most save a search. When what's stored is out of date, it costs a wrong first answer.
Part of this question's answer key is reworded for publication: the original wording described an internal detail that may be security-sensitive. The runs were graded with the original key.
Written-records control; with and without memory only.
| Side | Correct | Cost to correct | Tokens to correct | 1st answer coverage | 1st answer wrong claims | 1st answer cost · tokens | Runs that never got there |
|---|---|---|---|---|---|---|---|
| With memory | 3 of 3 (answer 3, 2, 2) | $0.23, $0.16, $0.14 | 270k, 144k, 155k | 0%, 50%, 0% | 3, 2, 2 | $0.06 · 57k | — |
| Without memory | 3 of 3 (answer 1, 1, 1) | $0.14, $0.20, $0.09 | 108k, 223k, 72k | 100%, 100%, 100% | 0, 0, 0 | $0.14 · 108k | — |
One line per answer. The marks are the key facts in order: ✓ present, ✗ missing. "% of key" counts ✓ as 1 and ½ as a half. An answer is correct when every required point is ✓; the run stops there. Cost, tokens and tool calls are the run's totals.
| Run | Each answer | Run cost | Run tokens | Tool calls |
|---|---|---|---|---|
| Q9-on-r1 | Answer 1 ✗✗ 0% of key · 3 wrong claims Answer 2 ✗✗ 0% of key · 3 wrong claims Answer 3 ✓✓ 100% of key · 1 wrong claim · correct | $0.23 | 270k | 16 |
| Q9-on-r2 | Answer 1 ✓✗ 50% of key · 2 wrong claims Answer 2 ✓✓ 100% of key · 4 wrong claims · correct | $0.16 | 144k | 10 |
| Q9-on-r3 | Answer 1 ✗✗ 0% of key · 2 wrong claims Answer 2 ✓✓ 100% of key · 2 wrong claims · correct | $0.14 | 155k | 12 |
| Q9-off-r1 | Answer 1 ✓✓ 100% of key · 0 wrong claims · correct | $0.14 | 108k | 12 |
| Q9-off-r2 | Answer 1 ✓✓ 100% of key · 0 wrong claims · correct | $0.20 | 223k | 15 |
| Q9-off-r3 | Answer 1 ✓✓ 100% of key · 0 wrong claims · correct | $0.09 | 72k | 9 |
Control
Question. “Will the becos codecollab Claude Code plugin work on Windows? If not, what blocks it, and what would a port involve?”
The whole answer is in the plugin's own code, and the original took 4 tool calls. Memory can only save reading here, so it should help little.
Original investigation: 2026-10-30: 4 tool calls, 0.7 minutes. · Workspace: the Claude Code plugin, at the commits current when the question was asked
Of the 5 facts, recall's 9,361 characters delivered 0 fully, 1 partly, and 4 not at all. Recall carried little on this question, so the memory runs read the code like the others.
A coding agent already knows what usually breaks Python on Windows: fcntl, os.fork, chmod, /tmp, a python3 launcher. Every no-memory run searched for exactly those, found python3 in hooks/hooks.json and .mcp.json and the fcntl import in capture.py, and was done in 3-8 tool calls.
Without the code it can't answer: asked with no workspace and no tools, Claude scored 0 of 5 facts in both completed tries, because it can't know which of those this plugin uses.
So the only thing memory could add is 'this plugin uses fcntl and python3', which saves a few searches, a few cents at most. Questions that general knowledge plus a quick look at the code can answer are a control, a check that memory doesn't make things worse, not a test of its value. Memory pays off on project-specific facts neither source gives: decisions, plans, results found by experiment, causes buried in another repo (Q4).
Yes/no facts checked in code; every answer rewritten neutrally before grading.
| Side | Correct | Cost to correct | Tokens to correct | 1st answer coverage | 1st answer wrong claims | 1st answer cost · tokens | Runs that never got there |
|---|---|---|---|---|---|---|---|
| With memory | 3 of 3 (answer 1, 1, 1) | $0.12, $0.13, $0.14 | 83k, 94k, 124k | 100%, 100%, 100% | 1, 1, 0 | $0.13 · 94k | — |
| Without memory | 3 of 3 (answer 1, 1, 1) | $0.15, $0.13, $0.14 | 124k, 100k, 113k | 100%, 100%, 100% | 0, 0, 0 | $0.14 · 113k | — |
| Redo the investigation | 3 of 3 (answer 1, 1, 1) | $0.10, $0.14, $0.10 | 102k, 125k, 109k | 100%, 100%, 100% | 0, 0, 0 | $0.10 · 109k | — |
One line per answer. The marks are the key facts in order: ✓ present, ✗ missing. "% of key" counts ✓ as 1 and ½ as a half. An answer is correct when every required point is ✓; the run stops there. Cost, tokens and tool calls are the run's totals.
| Run | Each answer | Run cost | Run tokens | Tool calls |
|---|---|---|---|---|
| Q10-on-r1 | Answer 1 ✓✓✓✓✓ 100% of key · 1 wrong claim · correct | $0.12 | 83k | 8 |
| Q10-on-r2 | Answer 1 ✓✓✓✓✓ 100% of key · 1 wrong claim · correct | $0.13 | 94k | 13 |
| Q10-on-r3 | Answer 1 ✓✓✓✓✓ 100% of key · 0 wrong claims · correct | $0.14 | 124k | 12 |
| Q10-off-r1 | Answer 1 ✓✓✓✓✓ 100% of key · 0 wrong claims · correct | $0.15 | 124k | 10 |
| Q10-off-r2 | Answer 1 ✓✓✓✓✓ 100% of key · 0 wrong claims · correct | $0.13 | 100k | 12 |
| Q10-off-r3 | Answer 1 ✓✓✓✓✓ 100% of key · 0 wrong claims · correct | $0.14 | 113k | 14 |
| Q10-off-full-r1 | Answer 1 ✓✓✓✓✓ 100% of key · 0 wrong claims · correct | $0.10 | 102k | 6 |
| Q10-off-full-r2 | Answer 1 ✓✓✓✓✓ 100% of key · 0 wrong claims · correct | $0.14 | 125k | 12 |
| Q10-off-full-r3 | Answer 1 ✓✓✓✓✓ 100% of key · 0 wrong claims · correct | $0.10 | 109k | 10 |
Three runs per setup is a small sample, so individual runs can significantly affect the median.
Q4 and Q10 can be answered from code, making them tests of how quickly an agent can find existing information. Q5 is mostly answerable from code, but its size limit existed only in past sessions. Its headline saving is measured against a single run, because only 1 of 3 runs without memory reached a correct answer, and that figure feeds the category average.
Q1 and Q2 are different: their answers existed only in past sessions. Without memory, Claude couldn't answer either question.
The memory export, session transcripts, answer keys and repositories used in this experiment are private for now. They contain internal information, and we're working on redacting sensitive details before publishing them alongside the benchmark harness.
For now, some details are redacted in this report: private repository names are masked (for example becos_xxxxxx), workspaces are described by role rather than repository and commit, and Q9's answer key has been reworded.
becos is in private beta. We'll run a pilot on one of your repositories.