Does what a coding agent worked out in earlier sessions change what it does in later ones? EC-Bench is our attempt to measure that.
Most coding benchmarks evaluate what an agent can accomplish inside a single episode: write the code, reason through the problem, plan the change. When the episode ends, so does the evaluation, and the next one starts from the same place as the last.
That leaves a question unmeasured: does the work an agent has already done change what it does later? Answering it takes sequences of sessions, not single ones.
Each run uses a FastAPI repository and 30 prompts, in three sequential sessions: 14 on investigation and architecture, 13 on implementation and debugging, and 3 on planning. Each arm gets its own copy of the repository and its own isolated memory. In the Reverie arm, every prompt starts with a fresh context, so anything the agent knows from earlier work has to come from memory. Between sessions, the harness accepts every review automatically.
One LLM judge reads the full transcripts and scores them on five weighted metrics. The judge is named in the ledger below.
Identical copy of the repository (FastAPI) for each arm
Arm A only: review: auto-accept (harness setting)
Arm A only: review: auto-accept (harness setting)
Both arms’ transcripts go to one judge.
LLM judge (same for both arms, full transcript)Five weighted metrics:
held constant: repository, prompts, agent harness, model, judge
differs: access to Reverie
Where a run departed from this design, the departure is noted in its row of the ledger.
So far, no run shows a clear advantage for Reverie. The first run’s aggregate favoured memory, but the gain came from one metric that loading memory inflates; on the other four, the agent did worse. The second architecture narrowed the gap under a protocol that favoured the baseline. A controlled comparison is next.
Our own write-ups disagree on some exact figures. Until the sources agree, each row below says what happened and why we aren’t printing numbers, and a run’s figures appear only once it has been reconciled.
The aggregate favoured memory, but the gain came from one metric, cognition reuse, which loading memory inflates. On the other four metrics the agent did worse with memory.
Figures under reconciliation.
Sources EC-Bench Technical Report 001 · EC-Bench Short Report
20260813-141550The baseline scored ahead on all five metrics.
Figures under reconciliation.
Sources The paper, section on results · Run 20260813-141550 (product documentation, section on the benchmark)
Not yet run.
Sources Research index: how we test it
Our first version loaded memory before the agent began work, and on this benchmark the agent trusted memory over the code and did worse.
That is why Reverie infers a task mode when your agent asks. How retrieval works
In the first run, memory helped on cognition reuse, the metric that loading memory inflates, and hurt on the other four.
contradicts and no supersedes relationships.The first run’s finding changed the design. Reverie no longer loads memory when a session starts. Your agent asks for it, after reading the code. How your agent gets it back