A generation and evaluation framework for testing whether memory systems can remember, forget, update, reflect, and track evolving state across long, noisy, multi-session conversations.
Overview · Benchmark Design · Quick Start · Data Generation · Evaluation · Data Format · Citation
Important
Research preview. MemOps uses external LLMs for data generation, answer
generation, verification, and judging. Running the full pipeline can make many
paid API calls. Start with the included artifacts and a small
--max-questions smoke test before launching a full run.
【NEWS!】 Excited to share our team's new work — Metis: Memory Foundation Model, which is the first prototype of a memory foundation model, equipping foundation models with a persistent and dynamically evolving native memory state. [Paper] [HuggingFace] [GitHub]
Most conversational-memory evaluations ask a static question: can the model retrieve a fact from the past? MemOps studies the harder lifecycle question: does the system maintain the right memory state as evidence is added, corrected, withdrawn, inferred, and surrounded by plausible noise?
MemOps constructs evidence-grounded conversations with explicit gold operation traces, injects them into unrelated multi-session dialogue, and evaluates both the final answer and the memory behavior that produced it.
The benchmark is built around four principles:
- Lifecycle-aware memory. Facts are created, superseded, forgotten, inferred, and revisited over time.
- Verifiable provenance. Every gold operation points back to exact user-turn spans instead of relying on an untraceable summary.
- Controlled difficulty. Recency traps, update chains, multi-target state, multi-hop reasoning, negative seeds, and adversarial injections are explicit knobs.
- Matched evaluation settings. The same capability is tested with nearby clean evidence and inside long, distractor-heavy conversation histories.
| Operation | What the system must do | Typical failure |
|---|---|---|
| Remember | Store a newly established user fact and retain its provenance | Missed or fabricated memory |
| Forget | Remove one requested target without deleting related facts | Leakage or over-forgetting |
| Update | Replace a stale value while preserving the correct target binding | Returning an old value as current |
| Reflect | Derive a bounded, evidence-supported pattern from multiple clues | Unsupported generalization or incomplete evidence |
| TrajectoryOps | Follow several operations across intermediate and final states | Wrong order, state, or provenance |
Each sample can contain confirmed, tentative, and retracted operations. This lets MemOps distinguish a genuinely current memory from a recently mentioned but unconfirmed value.
Adjacent operation
└── Clean evidence conversation → question
Longitudinal operation
└── Many unrelated sessions
├── sparse evidence segments
├── near-miss and stale distractors
└── question about the current memory state
- Adjacent evaluation measures whether a model can interpret the clean evidence and operation semantics.
- Longitudinal evaluation measures whether a memory strategy can preserve the same state after evidence is dispersed across a much longer history.
MemOps does not reduce evaluation to a single recall prompt. Generated samples cover six complementary probe types:
| Probe | Evaluated capability |
|---|---|
OperationTrace |
Recover the evidence and operations that led to the state |
TargetBinding |
Apply an operation to the correct entity or attribute |
StateTransition |
Reason about before/after memory state |
CandidateDisambiguation |
Reject stale, tentative, or third-party values |
OperationApplication |
Use the valid memory state in a downstream task |
StateTrajectory |
Reconstruct intermediate and final states across a chain |
Every generated sample carries a machine-readable difficulty_knobs object.
The main controls are:
| Knob | Stress test |
|---|---|
recency_trap |
A later tentative or retracted value should not replace the confirmed state |
update_chain |
Several changes must be applied in the correct order |
multi_target |
Related targets must remain independently addressable |
multi_hop |
The answer requires combining facts from different turns |
negative_seed |
The system must avoid inventing or retaining an invalid item |
adversarial_injection |
Plausible stale or near-miss evidence appears close to the query |
git clone https://github.com/MemTensor/MemOps.git
cd MemOps
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txtMemOps requires Python 3.10 or newer.
export LLM_BASE_URL="https://api.openai.com"
export OPENAI_API_KEY="your-api-key"LLM_BASE_URL must expose an OpenAI-compatible chat-completions API. Model
names in the shell launchers assume that the configured provider or gateway can
serve those models; override the relevant *_MODEL variables when it cannot.
For local development, a key can instead be placed in a git-ignored api.md
file at the repository root. Environment variables are recommended for shared
or production machines.
The repository includes generated artifacts through the injection stage, so a smoke test does not require regenerating the benchmark or downloading UltraChat:
python 5-test_operation_metrics.py \
--output-dir generated_result/5-smoke \
--model gpt-4o \
--adjacent-models gpt-4o \
--long-context-models gpt-4o \
--no-context-models "" \
--rag-methods rag_vanilla \
--rag-retrieval-units turn \
--max-questions 10Judge the resulting answers:
python 5.5-evaluate_operation_metrics.py \
--input-file generated_result/5-smoke/operation_metrics_all_methods.jsonl \
--output-dir generated_result/5.5-smoke \
--judge-model gpt-4o \
--eval-workers 4The main aggregate report is written to
generated_result/5.5-smoke/summary.json.
The full benchmark pipeline is organized as numbered, inspectable stages:
| Stage | Entry point | Output | Purpose |
|---|---|---|---|
| 1 | 1-generate_background.py |
generated_result/1-background/ |
Generate diverse synthetic user backgrounds from info/topics.md |
| 2 | 2-generate_evidence_conversation_w_verity.sh |
generated_result/2-evidence_conversation/ |
Generate evidence conversations, gold operations, questions, and verifier-approved answers |
| 3 | 3-generate_distractor_facts.sh |
generated_result/3-distractor_facts/ |
Create target-aware, state-neutral distractor conversations |
| 4 | 4-inject_evidence_with_distractors.sh |
generated_result/4-inject_evidence_with_distractors/ |
Insert evidence and distractors into sampled UltraChat sessions |
| 5 | 5-test_operation_metrics.sh |
generated_result/5-test_operation_metrics/ |
Run answer models and memory baselines |
| 5.5 | 5.5-evaluate_operation_metrics.sh |
generated_result/5.5-evaluate_operation_metrics/ |
Score answers and produce sliced summary metrics |
Download UltraChat and
point ULTRACHAT_PATH to either its directory or train_0.jsonl:
export ULTRACHAT_PATH="/path/to/UltraChat"
BACKGROUND_MODEL=gpt-4o \
python 1-generate_background.py
EVIDENCE_GENERATION_MODEL=gpt-4o \
EVIDENCE_QUESTION_MODEL=gpt-4o \
EVIDENCE_VERIFY_MODEL=gpt-4o \
./2-generate_evidence_conversation_w_verity.sh
DISTRACTOR_GENERATION_MODEL=gpt-4o \
DISTRACTOR_VERIFY_MODEL=gpt-4o \
./3-generate_distractor_facts.sh
INJECT_SEED=21 \
./4-inject_evidence_with_distractors.sh
RAG_MODEL=gpt-4o \
ADJACENT_MODELS=gpt-4o \
LONG_CONTEXT_MODELS=gpt-4o \
NO_CONTEXT_MODELS=gpt-4o \
./5-test_operation_metrics.sh
EVAL_JUDGE_MODEL=gpt-4o \
./5.5-evaluate_operation_metrics.shGeneration templates live in template/. They define the dialogue
structure, gold-operation schema, question taxonomy, quality gates, and
verifier requirements, making the benchmark recipe auditable rather than
hiding it in a single generation prompt.
The judge produces overall answer accuracy and operation-detection precision/recall/F1, plus lifecycle-specific diagnostics:
- Forget: leakage rate and over-forget rate.
- Update: stale-value rate.
- Reflect: reflection precision and recall.
- Trajectory: order accuracy, intermediate-state accuracy, final-state accuracy, and provenance support.
- All operations: evidence-provenance support.
summary.json additionally slices results by evaluation setting, method,
context mode, retrieval mode and unit, answer model, operation type, probe type,
difficulty, enabled difficulty knob, operation validity, and chain length. It
also includes matched-pair deltas and paired-flip audits.
# Run a bounded shard of questions.
python 5-test_operation_metrics.py \
--question-offset 100 \
--max-questions 100
# Compare all shipped sparse-retrieval variants.
python 5-test_operation_metrics.py \
--rag-methods rag_vanilla,rag_delete,rag_rewrite \
--rag-retrieval-units session,turn
# Run longitudinal methods only.
python 5-test_operation_metrics.py \
--run-adjacent falseRun python 5-test_operation_metrics.py --help and
python 5.5-evaluate_operation_metrics.py --help for the complete CLI.
Each evidence file is a JSON object containing the conversation, its gold memory operations, explicit difficulty settings, and evaluation questions. A condensed example is shown below:
{
"target_fact": "Current city of residence",
"operation_type": "Update",
"difficulty_knobs": {
"recency_trap": {"enabled": true, "trap_type": "tentative"},
"update_chain": {"enabled": true, "num_updates": 3}
},
"conversations": [
{
"segment_index": 1,
"segment_role": "establish the initial state",
"dialogue": [
{"role": "user", "content": "..."},
{"role": "assistant", "content": "..."}
]
}
],
"operations": [
{
"operation_id": "op2",
"type": "update",
"validity": "confirmed",
"trigger_span": {
"segment_index": 2,
"turn_index": 3,
"quote": "..."
},
"target": {
"target_id": "current_city",
"target_name": "current city of residence"
},
"old_value": "Seattle",
"new_value": "Portland",
"evidence_spans": [
{"segment_index": 1, "turn_index": 1, "quote": "..."},
{"segment_index": 2, "turn_index": 3, "quote": "..."}
]
}
],
"answer": [
{
"question_pair_id": "p1_state_transition",
"evaluation_setting": "longitudinal_operation",
"evaluation_type": "StateTransition",
"evaluation_category": "state_reasoning",
"question": "...",
"expected_answer": "...",
"gold_memory_state": "...",
"judge_rubric": {
"must_include": ["..."],
"must_not_include": ["..."]
},
"diagnostic_checks": {
"stale_value": "..."
},
"gold_provenance": [
{"segment_index": 2, "turn_index": 3, "quote": "..."}
]
}
]
}The injection stage keeps the questions while replacing the short evidence
history with a longer conversations list. Its injection_metadata records
where evidence and distractors were inserted, their source sessions, and the
seed-controlled sampling layout.
MemOps/
├── 1-generate_background.py
├── 2-generate_evidence_conversation_w_verity.py
├── 3-generate_distractor_facts.py
├── 4-inject_evidence_into_irrelevant_conversation.py
├── 5-test_operation_metrics.py
├── 5.5-evaluate_operation_metrics.py
├── generated_result/
│ ├── 1-background/
│ ├── 2-evidence_conversation/
│ ├── 2-evidence_trace/
│ ├── 3-distractor_facts/
│ └── 4-inject_evidence_with_distractors/
├── info/
│ └── topics.md
├── template/
│ ├── generation prompts
│ └── verifier prompts
├── img/
│ └── generation_pipeline.png
└── requirements.txt
If you use MemOps in your research, please cite our paper:
@misc{hao2026memops,
title = {MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations},
author = {Hao, Xixuan and Zhang, Zeyu and Lin, Zehao and Sun, Yihang and
Guo, Ziliang and Zhang, Xichong and Liang, Yuxuan and Xiong, Feiyu and
Li, Zhiyu},
year = {2026},
eprint = {2607.12893},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
url = {https://arxiv.org/abs/2607.12893}
}MemOps is released under the MIT License.
MemOps uses UltraChat as the source of unrelated carrier conversations for longitudinal injection.