Evidence review / October 26, 2026
The correction fix worked.
Abstention still needs work.
The later judged run completed every correction and forget operation in its synthetic fixture. Its retrieval results still do not justify recommending automatic extraction or the full biomimetic profile by default.
The review recovered the later source and checked the recorded outputs. It made no provider calls and changed no thresholds. The current work does not establish a new quality improvement. The separate local release-gate result is recorded below.
The release gate is still failing
The required local benchmark gate failed all six comparison pairs in each of two repetitions. Retrieval MRR lift ranged from -0.4 to 0, and the SQLite latency checks also failed. The thresholds remain unchanged. This candidate has not been released to PyPI.
The restart example can pass while the broader release gate fails: they test different things. The example demonstrates explicit fact persistence and protocol access. It does not establish improved retrieval quality or release readiness.
The legacy harness also uses different embedder fitting paths and compares a SQLite read with an engine operation that writes access metadata. Its engine arm inherits the baseline retrieval profile. These limits need a fairer prospective experiment; they do not clear the failed gate.
The latest result, with the baseline beside it
The corpus contains 200 synthetic held-out sessions, each repeated three times. The latest judged full run reused that corpus after the original failures had been inspected. It is a regression rerun, not a fresh blinded evaluation.
The profiles also differ in which captured facts are visible to retrieval. These scores do not isolate the effect of ranking.
| Measure | Original judged baseline | Latest judged full |
|---|---|---|
| Completed observations | 600 / 600 | 600 / 600 |
| Corrections committed | 60 / 60 | 60 / 60 |
| Forgets committed | 60 / 60 | 60 / 60 |
| Exact precision proxy | 28.46%1110 / 3900 | 51.63%1110 / 2150 |
| Critical retention | 52.86%1110 / 2100 | 52.86%1110 / 2100 |
| Exact answerable recall | 38.5%231 / 600 | 38.5%231 / 600 |
| No-answer accuracy | 36.36%240 / 660 | 22.73%150 / 660 |
The precision measure compares frozen text, scope, and evidence exactly. It is not semantic factual precision. The latest rerun covers the full profile only; it does not replace the original seven-profile ablation matrix. No real-host acceptance was measured.
The old correction failure was already repaired
The original judged full run committed 0 of 60 corrections. Many automatic captures phrased as a request to remember were classified as containing no durable facts, so the correction target never existed.
A later September source bundle included a classification description that preserves a durable fact introduced by a request to remember. Its run committed 60 of 60 corrections and 60 of 60 forget operations. Four source files were recovered from that bundle after their hashes were checked against its execution record. Repeating that prompt edit would not fix the remaining abstention problem.
The separate local source-bound run completed 531 of 600 observations and committed 0 of 60 forget operations. Its excluded failures must remain visible in any comparison. The full report includes those runs and their denominators.
What the retrieval scores hide
Exact answer boundaries
A separate read-only diagnostic found the full literal gold sentence somewhere in 586 of 600 returned answers. Only 231 returned texts matched it exactly; 355 added a prefix or suffix. That helps explain the exact recall score. It does not replace the frozen score or establish semantic correctness.
Related text that does not answer the question
For a question about an unmentioned incident commander, the judged profile returned a transient deployment status. The text was related enough to pass semantic filtering, but the requested answer was absent. The latest run got only 150 of 660 no-answer checks right.
Different visible facts across profiles
With extraction disabled, automatically captured records are hidden from the default memory retrieval view. The full profile can promote extracted records into that view. The original comparison therefore changes both the visible facts and the retrieval behavior; it cannot isolate ranking quality.
A smaller next test
Use a separate development fixture and one fixed set of accepted facts for every profile. Keep the embeddings, scopes, sources, and context budget the same. Test whether a candidate actually supports an answer before tuning ranking or thresholds.
The current retrieval code already checks semantic relevance before graph and state boosts. A missing relevance gate is not the explanation. The next diagnostic should distinguish related text from answer-supporting evidence, including absent entities, missing relationships, temporary status, and quoted instructions.
The original ablations did not establish useful gains for the other four modules. Those negative results stay part of the record. In the meantime, explicit memory and restart persistence can be useful on their own.
Inspect the evidence
The report records the corpus identity, recovery method, exact scores, partial-run exclusions, and the limits of the proposed diagnostic. Its companion provenance record binds the source archives and recorded results with SHA256 hashes.