MEMORA

MEMORA: Embodied Action Memory from Egocentric Videos
for Reasoning and Planning

Zihao Yu Xiu Yuan Chongjie Zhang

Washington University in St. Louis

Oral Presentation RSS 2026 FM4RoboPlan
OpenReview Workshop website
Paper Demo Code MEMORA-Bench

Editable, consolidated memory from egocentric experience grounds future plans.

View static paper overview

Overview

Embodied intelligence is shaped not only by the scene in front of an agent, but also by accumulated experience. Familiar places carry histories of use: objects change state, actions become procedures, and repeated episodes reveal regularities that no single frame contains. Egocentric video records this experience from the actor's point of view.

While egocentric imitation and cross-embodiment methods use first-person human video to learn how behavior can be executed, MEMORA studies a complementary question: what should experience become for a future planner? We formulate Embodied Action Memory as the capability to form, maintain, and use embodied experience as a persistent memory state.

MEMORA organizes egocentric experience as editable and consolidated embodied memory, rather than retaining it only as episodes to retrieve or demonstrations to imitate. The resulting semantic–procedural state preserves what happened, what changed, and what recurs so that later goals can be grounded in participant-specific experience.

MEMORA-Bench evaluates this lifecycle on 45 hours of egocentric video across 18 participants. Across four open-weight language models, full MEMORA performs best overall among the evaluated memory interfaces. It gains up to 20.5 points on the experience-dependent, memory-grounded EAM-QA subset and up to 16.6% relative Robot-Grounded Plan score on Generalize. A qualitative two-task robot deployment illustrates the path from remembered human experience to physical execution.

Experience becomes memory. Memory guides action.

MEMORA encodes each segment, revises persistent state as experience changes, consolidates repeated evidence, and retrieves what a later goal needs.

Online · each 10-second segment

Encode layered observations

The Segment Encoder turns each 10-second segment into environment, entity, and activity observations while carrying short context across boundaries.

Segment-level observation
Egocentric breakfast-preparation observation
00:10–00:20
Environment Memory
Entity Memory
Activity Memory
Inferred Knowledge

What memory makes possible

When a new goal arrives, the planner does not replay the entire past. It asks memory for the procedures, objects, places, and participant-specific regularities that make an otherwise generic instruction concrete.

Future goal Help P01 wash all the dirty dishes, dry them, and put them away.
A

Memory makes the goal specific

wash, dry, and store dishes dish storage preferences sponge cloth
Entities

Red ceramic dirty bowl, black metal plate, yellow foam sponge, and blue cotton cloth

Environment

Silver stainless steel sink, chrome metal tap, drying rack, counter, and storage cabinet

Procedure

Collect the dishes, wash with soap, rinse, place them on the rack, dry them, and store them.

Participant grounding

Routine and storage-preference retrieval guide the final sequence and destination

B

The planner turns memory into action

12-step MEMORA plan Objects, tools, locations, and procedure are grounded in P01's remembered kitchen.
  1. Collect the red ceramic dirty bowl and the black metal plate from the counter and carry them to the silver stainless steel sink.
  2. Pick up the yellow foam sponge from the edge of the sink with the right hand.
  3. Wet the yellow foam sponge under the chrome metal tap and apply dish soap.
  4. Scrub the red ceramic dirty bowl with the yellow foam sponge using the left hand.
  5. Rinse the red ceramic bowl under the running water from the chrome metal tap.
  6. Scrub the black metal plate with the yellow foam sponge using the right hand.
  7. Rinse the black metal plate under the running water from the chrome metal tap.
  8. Place the washed red ceramic bowl and black metal plate on the drying rack located on the left side of the sink.
  9. Pick up the blue cotton cloth from the counter with the left hand.
  10. Wipe the red ceramic bowl dry with the blue cotton cloth.
  11. Wipe the black metal plate dry with the blue cotton cloth.
  12. Place the dried red ceramic bowl and black metal plate into the storage cabinet.

From memory to physical action

Planning is where remembered experience meets a new embodiment. These two qualitative examples follow memory-supported actions into physical execution.

Human observation
Actions shown in execution4 grounded actions
  1. Navigate to the counter area.
  2. Grasp the dark soda bottle.
  3. Select the red Starbucks cup.
  4. Pour the soda into the cup.
Robot grounding · 2.0× speed
Human egocentric experience→MEMORA plan→Fixed low-level controller

For these qualitative demonstrations, language-level actions are mapped to a fixed rule-based Unitree G1 controller. The drink example shows the actions carried out by the robot; the breakfast example shows the complete generated plan.

MEMORA-Bench

A memory system should be judged twice: by whether it remains faithful to what happened, and by whether what it remembers can help with what comes next. MEMORA-Bench pairs these retrospective and prospective tests over the same participant-specific memory.

MEMORA-Planning

Find target→Ground workspace→Recover action sequence→Compose plan

Help P01 wash the cucumber before preparing it.

Open each call to see how Entity, Environment, and Activity Memory jointly ground the plan.

search("cucumber")
Routed to Entity Memory
Top entity match0.8621

Green cylindrical cucumber

Observed wet and later located on the cutting board.

P01_105 · participant-specific entity state

Planner decision: identify the participant's cucumber as the target rather than leaving the instruction generic.

search("faucet sink")
Routed to Environment Memory · top 3
Rank 1 · P01_1060.8256

Sink area

Sink, faucet, sponge, and drying rack form a shared washing workspace.

faucet above sink · sink adjacent to counter
Rank 2 · P01_1040.8235

Stainless-steel sink area

A chrome faucet sits above the sink; a drying rack occupies the nearby counter.

faucet above sink · sink left of drying rack
Rank 3 · P01_1090.8220

Sink and preparation counter

The cutting board is on the counter beside the washing area.

faucet above sink · cutting board on counter

Planner decision: ground washing at P01's sink and preserve the transition from sink to preparation counter.

search_objects(name="cutting board")
Entity Memory match
Top entity match0.8800

Brown wooden cutting board

Clean and located on the counter relative to the sink.

destination for the washed cucumber

Planner decision: establish where the cucumber should go after washing.

search_activities(query="wash cucumber")
Activity Memory · top 3
Rank 1 · Selected0.8495

Wash cucumber and knife

Pick up cucumber → turn on faucet → scrub → rinse → turn off faucet.

P01_105 · temporally ordered action evidence
Rank 20.8222

Move cucumber to cutting board

Pick up the cucumber from the sink and place it on the cutting board.

P01_105 · supports the post-wash transition
Rank 30.8136

Clean a spatula in the sink

Related sink-side rinsing evidence, but not the requested object.

P01_105 · rejected as less specific

Planner decision: combine the first activity's washing sequence with the second activity's destination transition.

View the generated 7-step plan
  1. Pick up the green cylindrical cucumber from the drying rack next to the sink with the right hand.
  2. Turn on the silver metal faucet located above the sink with the right hand.
  3. Hold the green cylindrical cucumber under the running water from the silver metal faucet using the right hand.
  4. Scrub the surface of the green cylindrical cucumber gently while it is in the sink with the right hand.
  5. Turn off the silver metal faucet above the sink with the right hand.
  6. Carry the washed green cylindrical cucumber from the sink to the cutting board on the counter with the right hand.
  7. Place the green cylindrical cucumber on the cutting board on the counter.

Generated plan · P01Ranked memory evidence

This generated plan and its ranked evidence come from P01's participant memory and the planning tools used in evaluation.

EAM-QA: Memory assessment

When preparing to cook eggs, does P04 prefer a spatula or fork?

A · Varies randomlyB · ForkC · Spatula ✓D · WhiskE · Not available

MEMORA selects C after retrieving repeated evidence of P04 stirring egg mixtures with a wooden spatula.

45 hegocentric video
18participants
2,212 + 551core QA / unanswerable controls
360planning goals
207 / 153Replay / Generalize
Interactive Explorer Benchmark source

Beyond exact replay

Paired ablations isolate editing and consolidation within fixed interfaces; cross-family comparisons evaluate each memory representation with its intended read tools. The strongest planning gain appears when no matching episode can simply be replayed and evidence must be composed across experience.

18×median reduction of the unedited entity-observation stream, with state histories preserved
+14.4memory-assessment points after offline consolidation
+16.6%maximum relative Robot-Grounded Plan gain on Generalize; the highlighted Gemma setting rises from 0.457 to 0.532 (+16.4%)
Online editing→Selective maintenance18× median entity-stream reduction with state histories preserved
Offline consolidation→Memory assessment+14.4 points over the same memory before consolidation
Full memory lifecycle→Generalize planningUp to +16.6% relative RGP across the evaluated backbones
What does Robot-Grounded Plan measure?

RGP is the unweighted mean of procedural-order correctness, key-object coverage, and preference adherence. Object-vocabulary grounding is reported separately in the paper.

Planning results on Replay and Generalize tasks
MEMORA’s largest gain appears on Generalize, where planners must compose remembered evidence.

Resources

Paper Code MEMORA-Bench License

Citation

@misc{yu2026memoraembodiedactionmemory,
      title={MEMORA: Embodied Action Memory from Egocentric Videos for Reasoning and Planning},
      author={Zihao Yu and Xiu Yuan and Chongjie Zhang},
      year={2026},
      eprint={2607.14252},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2607.14252},
}

Evaluation scope. MEMORA evaluates semantic-procedural memory and language-level planning. The two robot demonstrations illustrate downstream execution; closed-loop visuomotor control is not evaluated here.

Output