Embodied Action Memory

MEMORA

Embodied Action Memory from Egocentric Videos for Reasoning and Planning

MEMORA turns accumulated egocentric experience into persistent semantic-procedural memory for grounded planning.

Zihao Yu · Xiu Yuan · Chongjie Zhang

Washington University in St. Louis

Paper preprint arXiv preprint Demo Code soon MEMORA-Bench soon

MEMORA overview: egocentric experience is formed into typed memory and retrieved for reasoning and planning — Experience is encoded, edited, consolidated, and later retrieved as evidence for reasoning and planning.

Core idea

Planning can learn from experience without replaying it verbatim

Embodied experience unfolds at several timescales. Places remain relatively stable, entities change state, actions provide ordered evidence, and repeated episodes reveal reusable procedures and participant-specific regularities.

Experience accumulates

Continuous first-person video records how a participant acts, not only what is visible at one instant.

Memory is maintained

Typed stores preserve distinct continuities while online editing and offline consolidation revise what is retained.

Plans query evidence

Retrieval separates procedural templates from physical grounding, then composes them for the current goal.

Interactive lifecycle

Watch memory form, change, and become useful

This trace combines real outputs from human egocentric video and a real consolidated EPIC-KITCHENS memory. Select a phase to inspect what MEMORA produces.

Online · each 10-second segment

Turn video into layered observations

The Segment Encoder converts the current video segment into environment, entity, and activity observations while carrying short working context across segment boundaries.

Counter area and workspace layout
Yellow bowl, tea packet, spoon, and bread
Items appear and move in temporal order

Egocentric breakfast-preparation observation

00:10–00:20

Environmentshared counter workspace

Entityyellow bowl placed in workspace

Activitytea item moves from hand to bowl

Environment Memory

Places, zones, layouts, and spatial relations

Entity Memory

Identity, attributes, state, location, and history

Activity Memory

Ordered action evidence and procedure traces

Inferred Knowledge

Reusable routines and participant regularities

Memory-grounded planning

A new goal composes evidence across memory

A real Generalize task has no single matching episode to replay. The planner retrieves participant-specific entities, places, procedures, and regularities, then produces a grounded long-horizon plan.

Instruction Help P01 wash all the dirty dishes, dry them, and put them away.

Memory retrieval

Evidence selected for this goal

wash, dry, and store dishes dish storage preferences sponge cloth

Entities

Red ceramic dirty bowl, black metal plate, yellow foam sponge, and blue cotton cloth

Environment

Silver stainless steel sink, chrome metal tap, drying rack, counter, and storage cabinet

Procedure

Collect → wash with soap → rinse → rack → dry → store

Participant grounding

Routine and storage-preference retrieval guide the final sequence and destination

Raw model output

Complete 12-step MEMORA plan

Collect the red ceramic dirty bowl and black metal plate and carry them to the sink.
Pick up the yellow foam sponge from the sink edge.
Wet the sponge under the chrome metal tap and apply dish soap.
Scrub the red ceramic bowl with the sponge.
Rinse the bowl under running water.
Scrub the black metal plate with the sponge.
Rinse the plate under running water.
Place both washed items on the drying rack left of the sink.
Pick up the blue cotton cloth from the counter.
Wipe the red ceramic bowl dry.
Wipe the black metal plate dry.
Place both dried items into the storage cabinet.

Actual tool calls

get_routine_skill("wash and dry dishes and put them away") get_preferences("where dishes and utensils are stored") search_objects("sponge") search_objects("cloth") search_objects("cloth")

The raw response contains the same 12 numbered steps shown above. Source: Gemma-4-31B-it, full MEMORA, task denovo_composition_P01_full_dishwashing_cycle.

Qualitative robot grounding

From remembered human experience to physical action

Two compact demonstrations show that plans generated from human egocentric memory can be grounded on a Unitree G1. The planning interface is the focus; execution provides a qualitative physical check.

Human observationEgocentric video

Memory-grounded planQwen3.6-35B-A3B

Navigate to the counter area.
Grasp the dark soda bottle.
Select the red Starbucks cup.
Pour the soda into the cup.

Robot grounding2.0× speed

MEMORA-Bench

Does formed memory remain faithful and support future action?

Both protocols operate on the same participant-specific memory. Memory assessment checks whether experience was preserved; planning checks whether that memory can guide action.

Retrospective faithfulness

MEMORA-Embodied Memory Assessment

When preparing to cook eggs, does P04 prefer a spatula or fork?

A · Varies randomlyB · ForkC · Spatula ✓D · WhiskE · Not available

MEMORA: C · retrieved repeated evidence of P04 stirring egg mixtures with a wooden spatula.

Prospective utility

MEMORA-Planning

Help P03 clean the washing-up bowl in the sink.

get_routine_skill("clean bowl") get_routine_skill("wash bowl") search_objects("bowl")

Grounded plan: black cooking bowl → green sponge → chrome faucet → metal drying rack.

45 hegocentric video

18participants

2,212memory-assessment items

360planning goals

207 / 153Replay / Generalize

Evidence

The full lifecycle matters most beyond exact replay

Controlled comparisons isolate memory construction and planning behavior across four open-weight backbones.

18×median reduction of the raw entity-observation stream, with state histories preserved

+14.4memory-assessment points from offline consolidation over episodic-only memory

+16.4%relative Robot-Grounded Plan gain on Generalize over the strongest non-MEMORA baseline

Entity-memory compression and online editor decisions — Online editing maintains memory selectively rather than appending every observation.

Planning results on Replay and Generalize tasks — MEMORA’s largest gain appears on Generalize, where planners must compose remembered evidence.

A complementary use of egocentric video

Action learning and memory formation answer different parts of planning

Egocentric policy learningHow can the robot execute an action?

Learns policies, action representations, or cross-embodiment alignment from human video.

MEMORAWhat remembered context should guide the plan?

Forms participant-specific semantic-procedural memory that planners and policies can query.

Resources

Explore MEMORA

Paper preprint arXiv preprint Demo watch above Code coming soon MEMORA-Bench coming soon

Citation

@misc{yu2026memora,
  title  = {MEMORA: Embodied Action Memory from Egocentric Videos for Reasoning and Planning},
  author = {Yu, Zihao and Yuan, Xiu and Zhang, Chongjie},
  year   = {2026}
}