Multimodal agent memory

Broader recall.
Sharper evidence.

Remember beyond the closest match.
Bring forward only the evidence that matters.

HiVe-Mem: Interleaving Horizontal Memory Expansion and Vertical Evidence Composition for Multimodal Agents

  • Haozhen Zhang1*
  • Bohan Liu1*
  • Jianzhu Bao1
  • Qingyuan Liu1
  • Haodong Yue2
  • Xiao Li3
  • Zi Pong Lim4
  • Wenya Wang1
  • 1 Nanyang Technological University
  • 2 Tsinghua University
  • 3 Sun Yat-sen University
  • 4 Aumovio Singapore Pte. Ltd.

* Equal contribution.

Read on arXiv Explore the code

Episode-centric memory. Graph-guided recall. Learned evidence selection.

~90%

fewer tokens

vs. the most efficient multimodal competitor*
3

memory benchmarks

Mem-Gallery · WorldMemArena · H2HMem
5

complementary evidence forms

composed independently for each memory
2

evaluated base models

Qwen3-VL-4B · GPT-5-mini transfer

*Paper Table 1; benchmark-averaged input + output tokens. See the comparison and protocol.

The idea

Better memory has
two dimensions.

A relevant episode can sit beyond the nearest semantic match. And even a relevant memory rarely needs to be read in full.

01

Preserve the episode

Keep raw text and images alongside summaries, captions, focused visual views, and modality-aware anchors.

Episode-centric memory
02

Expand the recall

Follow textual and visual anchor connections to recover complementary episodes beyond semantic retrieval.

Horizontal expansion →
03

Compose the evidence

A lightweight learned selector chooses which evidence forms each retrieved episode contributes to the answer.

Vertical composition ↓
Figure 1 from the paper showing memory formation, affinity-guided horizontal expansion, and independent evidence-form decisions for each retrieved memory
FIGURE 01 From interaction history to connected memories and query-adaptive evidence.

See the distinction

One memory.
Different questions.

The question changes what is useful. A trip name may need a short summary; a bag’s color may need a focused visual view.

Illustrative example. The selections below explain the mechanism; they are not live model predictions.

EPISODE m₈Recovered through expansion
Illustration of a red tote bag from a Kyoto tripKyoto · a red canvas tote
SCompact summary—
TRaw interaction text—
CImage caption✓
IRaw image—
VGrounded visual view✓

Useful evidenceA caption and a focused view preserve the bag’s color without the full scene.

The evidence

More from memory.
Less to process.

Higher average answer quality with substantially lower token consumption across three multimodal benchmarks.

HiVe-Mem · LLM-Judge57.96
Total tokens · millions0.658M
Token reduction vs. most efficient competitor91.6%
Qwen3-VL-4B-Instruct · three-benchmark average · paper Table 1
Multimodal methodF1 ↑LLM-Judge ↑Tokens (M) ↓Token footprint
M2A28.4243.8918.492
MIRIX27.6647.788.622
MMA28.1948.3515.752
MemVerse24.1445.427.812
HiVe-Mem Ours32.3257.960.658

F1 and LLM-Judge are on a 0–100 scale. Tokens sum input and output consumption in millions for each benchmark; “average” is the arithmetic mean across benchmarks, not a per-query count. Reductions compare HiVe-Mem with the lowest-token competing method in the selected view.

Source and experimental protocol

Values are reported in Table 1 of the HiVe-Mem paper. The comparison displays its five multimodal methods. The selector is trained on Qwen3-VL-4B-Instruct; GPT-5-mini uses the transferred selector without retraining. WorldMemArena uses the Lifelong subset.

These are reported paper results, not new measurements from this checkout. Paper settings use 5 semantic seeds, 2 expanded episodes, graph degree 4, and λ = 0 for the main results. Current release defaults differ; consult the repository’s training settings before reproduction.

Run settings ↗

Build with HiVe-Mem

Give your agent
a wider memory.

Explore the method, build a memory bank, or train an evidence selector on three supported benchmarks.

Installation guide ↗Configuration ↗
Start with the codebash
# Linux · Python 3.10+
git clone https://github.com/ViktorAxelsen/HiVe-Mem.git
cd HiVe-Mem
bash scripts/install.sh
source .venv/bin/activate
cp .env.example .env

Configure model services in .env, then follow the README to prepare data and build memories.

For your research

Build on the idea.

If HiVe-Mem informs your work, please cite the paper.

arXiv:2611.12872 ↗
BibTeX · arXiv:2611.12872
@article{hivemem2026,
  title = {{HiVe-Mem}: Interleaving Horizontal Memory
    Expansion and Vertical Evidence Composition
    for Multimodal Agents},
  author = {Zhang, Haozhen and Liu, Bohan and Bao, Jianzhu
    and Liu, Qingyuan and Yue, Haodong and Li, Xiao
    and Lim, Zi Pong and Wang, Wenya},
  journal = {arXiv preprint arXiv:2611.12872},
  year = {2026},
  url = {https://arxiv.org/abs/2611.12872}
}