A retrieval system can return plausible memories and still miss the one that answers the question. That made a second-stage reranker worth investigating in METABOLIQ Brain. It did not make enabling one the obvious next step.
We implemented a local cross-encoder, evaluated it against the existing hybrid retriever and kept it disabled by default. The aggregate quality gains were too small and uncertain for our decision rule, some query groups got worse, and the added latency missed the budget on the measured machine.
This article concerns the merged MB-002 and MB-002.1 implementation. The reranker is an optional experimental capability, not an accepted default recommendation. The wider foundation remains a single-owner system for a trusted local environment.
A second opinion on the candidate list
Hybrid retrieval combines keyword and semantic candidates. A cross-encoder takes the query and each candidate's text together and produces a score used to reorder those candidates. It does not create missing evidence, verify a memory's factual accuracy or recover a relevant memory that never entered its candidate pool.
Brain applies reranking to a bounded head of the authorized candidate list. The default depth is 20. Each candidate supplies its title and up to 2,000 characters of content. The remaining tail keeps its fused order; requests beyond the reranking depth skip model scoring.
The shipped configuration disables the reranker provider and leaves reranking off by default. Enabling the capability requires model installation and provider configuration. Under that default request behavior, a caller then opts in. An administrator can change the server default, and a caller can explicitly request reranking off. Keyword mode ignores the reranking flag.
Those distinctions matter: “disabled by default” describes a configuration, not a permanent prohibition.
Authorization comes before model scoring
A reranker needs memory text, which makes the boundary before scoring important. The implementation selects candidates under the caller's scope, reselects the text under that scope before scoring and scopes hydration again before returning results. Reranking does not grant access to a broader pool.
The existing tests use a recording fake reranker to inspect its inputs. They check other owners, ungranted namespaces, over-ceiling memories and archived records. They also check that adding forbidden-but-similar memories does not change the tested agent's response. Model identity diagnostics are withheld from agent responses.
The implemented provider runs ONNX inference locally. The image build can download pinned artifacts, and the pinned loading path verifies file checksums before loading. A standalone unpinned cache path also exists; we are not claiming every possible configuration enforces pinned offline loading.
Our additional real-model checks used existing local pinned files with hub offline mode enabled. No real personal or company memories were used. This is evidence about the inspected provider and controlled checks, not an audit of every dependency or an infrastructure-wide network guarantee.
Set the bar for changing the default
The evaluation used an explicit enablement rule: improvements in both reciprocal-rank and graded-ranking metrics, an uncertainty interval excluding zero for the graded-ranking gain, positive held-out results, bounded category regressions, acceptable regression-set behavior, authorization checks and a latency budget.
At depth 20, the latency bar was no more than 400 milliseconds added median latency and 900 milliseconds added p95 latency on the evaluation hardware. The quality bar required gains of at least 0.05 in both MRR@10 and nDCG@10.
MRR rewards finding a relevant result early. nDCG also accounts for relevance grades and ordering across the result list. A small improvement in an overall average can conceal damage to particular questions, so the protocol examined held-out queries, categories and newly introduced misses as well.
The finalist missed the bar
In the saved primary evaluation, MiniLM-L6 reranked the top 20 candidates. Across 106 answerable queries, nDCG@10 improved by 0.032, with a 95% paired-bootstrap interval from −0.010 to +0.071. MRR@10 improved by 0.021. Both average gains were below the chosen 0.05 thresholds, and the nDCG interval included zero.
On the held-out test split, MRR@10 fell by 0.012. The ambiguous-reference category lost about 0.18 MRR, and seven queries lost their only relevant top-five result. Some query groups improved, but those improvements did not erase the regressions.
The historical paired measurements recorded 930 milliseconds of added median latency and 1,377 milliseconds at p95. These are measurements from one shared laptop environment, not a prediction for every deployment or a fresh timing run for this article.
The larger bge-reranker-base comparison also failed the overall enablement rule. Its nDCG gain was 0.028 with an interval spanning zero, and its recorded added median latency was about 5.6 seconds. That run was an informational quality comparison, not a second independently selected finalist.
We recomputed per-query metrics from the saved ranked results and labels across six result files. Every stored metric matched. We also regenerated the primary MiniLM-L6 and bge decision tables, including bootstrap intervals, from those saved files. That verifies the recorded analysis; it does not reproduce the original model execution or latency measurements.
The protocol changed, and that belongs in the story
The evaluation was not a pristine confirmatory experiment. Before an amendment, developers had already seen preliminary quality results for two models. The amendment introduced length-sorted batching, added smaller candidates and changed the model-selection procedure. The enablement thresholds and answerability rule stayed fixed.
The amended selection procedure used development-split results and a tie rule that preferred a faster model when gains were within 0.01. MiniLM-L6 filled both finalist slots. The larger bge model was additionally evaluated as an informational comparison.
The merged analysis tests reject selection files containing test-split rows, check the finalist tie rule and leave the regression criterion unevaluated when its required baseline file is absent. Those checks help enforce the procedure now. They do not remove the exploratory nature of choices made after preliminary results were seen.
The corpus and questions were synthetic and English-only. Authoring and labeling agents came from one model family. Category samples were small, labels could disagree, and latency was measured on one machine with unrelated containers present. A second-labeler sensitivity run did not establish a reason to enable reranking.
A relevance score is not permission to answer
The primary evaluation contained 38 unanswerable queries. Both hybrid search and its reranked variant returned results for all 38. The reranker reordered candidates; it did not add an abstention mechanism.
Its top score had an answerability AUROC of about 0.784, compared with 0.771 for the cosine baseline. It failed the evaluation's answerability claim rule. Those values do not support treating a raw reranker score as a calibrated probability or introducing a universal cutoff.
Optional keyword and lexical-coverage signals provide additional descriptions of returned results. They remain signals of matching behavior, not proof of truth, sufficient evidence or confidence in an answer. No LLM judge was evaluated.
Keeping the implementation without changing the recommendation
The optional provider gives future evaluations a place to start. The tested default path does not call it. If an explicitly requested reranker is unavailable or raises the handled reranking error, the response falls back to the unreranked order with a content-free notice. This is not a promise to recover from every possible process or model failure.
For this private article review, we ran the merged backend tests excluding the separate real-model suite, and added focused checks of disabled configuration, real pinned MiniLM scoring and refusal of a modified model copy. The detailed verification record distinguishes those fresh checks from the historical benchmark and documents their limits.
The source baseline matches the merged head reported in the Brain acceptance records. Remote freshness could not be independently confirmed because the configured Git credential helper was unavailable. Backup-and-restore work was ongoing separately and is outside this article's verification.
Revisiting the decision would require a fresh evaluation on founder-approved representative data, credible quality gains and acceptable latency on the intended hardware. Model and dataset terms would also need review before any commercial-use or redistribution decision; this article does not provide a licensing clearance.
The useful outcome of an experiment is sometimes a reason to leave the default alone. In this case, we gained an implementation, a reproducible analysis and clearer evidence about its limits. We did not gain enough evidence to recommend enabling it.
← All engineering articles