Beijing Institute of Mathematical Sciences and Applications

Mingming Sun , Yaqing Wang

Guiliang Liu

Friday, October 25, 2024 2:00 PM - 4:00 PM

A3-2-301

Zoom 293 812 9202 (BIMSA)

Embodied Artificial Intelligence refers to AI systems that are integrated with physical entities, enabling them to interact with the world in human-like ways. As a cross-disciplinary research field, Embodied AI encompasses topics such as perceptual understanding, interactive decision-making, and automatic policy adaptation, focusing on developing AI agents that consistently solve real-world tasks. Recent advancements in Large Language Models (LLMs) have significantly impacted the developmental trajectory of Embodied AI systems. These advancements facilitate the evolution from task-driven AI agents to Vision Language Agents (VLAs), which are generalist AI agents capable of solving multiple tasks based on various commands. Training such large multi-modality models typically requires extensive datasets. Moreover, unlike LLM datasets which are readily available online, the datasets for agent supervision include control signals conditioned on diverse environmental dynamics. Previous studies typically collect these datasets through manual teleoperation with real robots or by extracting motion features from other entities such as humans or animals. For training large models, the Real-to-Real pipeline is not computationally feasible. In this presentation, I introduce an alternative approach that utilizes a generative simulation to synthesize training data. The environment for this simulator is designed using a generative model capable of producing a wide range of objects, layouts, and contexts for robot training. By setting various goals and objectives through an LLM, we employ motion planning and reinforcement learning algorithms to identify skills represented by problem-solving trajectories. We demonstrate that the learned skills can address realistic problems through experiments conducted with real robots. Moving forward, we are exploring the possibility of extending this simulation approach across multiple embodiments, including robot arms and humanoids, to further enhance the versatility and applicability of Embodied AI systems.