open-sourceragllm

LongMemEval-S: 96.0% with the official judge, and what top-5 vs top-20 retrieval changes [R]

r/MachineLearning·November 15, 2026·/u/remoteinspace

We ran all 500 LongMemEval-S questions through our memory retrieval (Papr) using the official answer prompt and the unmodified gpt-4o-2024-08-06 judge. Per-question answers and judge labels are public, so the official metrics script reproduces the numbers. Results For comparison, Chronos reports 95.6% using full context and multi-turn agents, Hindsight reports 94.6% at 43.6k tokens per question (with a different judge), and gbrain reports 90.6%. Things I found useful Retrieval recall is close to saturated. All evidence sessions are in the top 5 for 460 of 470 answerable questions (97.9%). Most remaining errors are the reader, not retrieval. Top 20 vs top 5 is a trade, not a free win: 11 gained, 8 lost, net +3. Temporal reasoning went 123 → 130 of 133. Abstention stayed flat (27/30 both). A small reader gets most of the way: gpt-5-mini reaches 94.4% on the same retrieval. Caveats Used Papr-Embed 0.6B embedding model (open source and open weight), gemini flash 2.5 reranker (also tested J

Read full article on r/MachineLearning

Related Stories