A central challenge is enabling embodied social agents to learn from and effectively leverage diverse information from both visual observations and natural language interactions over extended periods. In this work, we introduce a structured, multi-modal, non-parametric memory system that stores, updates, and retrieves information effectively. Integrating it with the visual grounding and planning capabilities of the foundation models, we build Ella, an embodied social agent who accumulates experiences and acquires knowledge across hours of social interaction in a 3D open world. Ella retrieves relevant information for decision-making, plans daily activities, builds social relationships, and evolves autonomously while coexisting with other intelligent beings in the open world. Experiments with 15 agents in a dynamic 3D open world demonstrate that Ella learns from and effectively utilizes the information from observation and social interaction to achieve goals. Our results suggest that combining non-parametric memory with foundation models is a promising path toward embodied social intelligence.
Ella: Embodied Social Agent
We build non-parametric long-term memory in two forms:
(a) name-centric semantic memory organizes the knowledge in a name-centric graph including a hierarchical scene graph serving as the spatial memory;
(b) spatiotemporal episodic memory stores the experience as a series of events consisting of
time,
location, and multimodal contents.
(c) Ella first generates a daily schedule according to the knowledge and experiences retrieved from the long-term memory,
(d) then updates the memory based on visual observations of the environment, and
(e) social interactions with other agents and
(f) makes reactions accordingly including
(f1) revising the schedule,
(f2) interacting with the environment,
(f3) and engaging in a conversation.