Teaching Robots the Shape of Reality: Hesai’s Kosmo turns real places into 3D assets
Hesai has launched Kosmo, a 3D world data engine designed to convert physical environments into high-fidelity 3D digital spaces, addressing the critical shortage of real-world data for training physical AI models.
Intelligence analysis by Gemini 2.5 Flash
Physical AI systems, unlike their digital counterparts, struggle with a lack of real-world data necessary for training and improving robotic capabilities. Hesai's new Kosmo engine aims to solve this by capturing lidar, imaging, and motion data from actual locations, transforming them into detailed 3D digital assets for robot simulation, world model development, and immersive content c…
Imagine you want to teach a robot how to clean your room, but it doesn't know what your room looks like. Hesai's Kosmo is like a super-smart scanner that takes a picture of your room, but in 3D, like a video game level. It captures every toy, chair, and wall, turning your real room into a perfect digital copy. Now, you can teach the robot in this digital room, letting it practice thousands of times without bumping into anything in the real world. This helps robots learn much faster and become better at understanding and moving around in our messy world.
Analysis
Physical AI
Physical AI faces a significant hurdle compared to digital AI: the scarcity of real-world data. While digital AI thrives on vast online datasets, robots and other physical AI systems require data that accurately reflects the geometry, physics, and constraints of actual places. This data is essential for training models that enable robots to understand, decide, and adapt within dynamic physical environments, moving beyond simple pre-programmed tasks.
The challenge lies in obtaining this real-world data in a manner that is accurate, affordable, and scalable. Without robust datasets, the development of truly intelligent and autonomous physical AI systems is severely limited. The ability to simulate and train robots in digital twins of real environments is a key step towards bridging this data gap and accelerating the progress of embodied AI.
Kosmo
Hesai's Kosmo is introduced as a comprehensive solution to this data infrastructure problem. It functions as a 3D world data engine, leveraging lidar, imaging, and motion data to capture real-world spaces. This raw data is then processed with AI-generated completion and object labeling, transforming physical environments into high-fidelity 3D digital assets. These digital twins are not merely static models but are adapted for robot simulation, capable of generating over 10,000 training trajectories from a single capture.
Dr. Yifan “David” Li, Hesai’s co-founder and CEO, emphasized that "Physical AI needs infrastructure that lets it understand the real world and act within it." Kosmo provides this by creating reusable assets for simulation and training, allowing developers to test and refine robotic behaviors in a controlled yet realistic digital setting. The engine supports open access to raw capture data and 3D assets, fostering an ecosystem with integrations like NVIDIA Isaac Sim, Isaac Lab, Unreal Engine, RenderMan, and Houdini.
Sharpa
The practical applications of Kosmo extend across various sectors, including robotics, visual effects, cultural tourism, and spatial commerce. In a compelling demonstration at a Dairy Queen store, Kosmo significantly streamlined the process of creating a training environment for a robot maker named Sharpa. A 15-minute Kosmo scan replaced approximately 300 hours of manual modeling, showcasing its efficiency.
Following the rapid reconstruction of the store environment, Sharpa’s robot achieved an impressive 99% task-success rate within the simulated setting. This highlights Kosmo's potential to drastically reduce the time and resources required for developing and deploying robots in complex real-world scenarios. Beyond robotics, Kosmo has been used by Pixar for RenderMan tutorials and by Hesai's Amap partnership to bring 3D reconstructions of cultural sites like the Gravity Art Museum to Amap’s "Flying Street View," aiming to enhance indoor navigation.
Key points
- Hesai launched Kosmo, a 3D world data engine, to address the shortage of real-world data for physical AI.
- Kosmo captures physical environments using lidar, imaging, and motion data, converting them into high-fidelity 3D digital spaces.
- The engine enables efficient robot training and simulation, generating over 10,000 training trajectories from a single capture.
- Applications include robotics (e.g., Sharpa's robot achieving 99% task success), visual effects, cultural tourism, and spatial commerce.
- Kosmo supports an open ecosystem with integrations for major platforms like NVIDIA Isaac Sim, Unreal Engine, and RenderMan.
Kosmo's ability to rapidly convert physical spaces into high-fidelity 3D digital assets could significantly accelerate the development and deployment of advanced physical AI and robotics. This infrastructure promises to make robot training more efficient and cost-effective, leading to more capable and adaptable autonomous systems across various industries.
While promising, the widespread adoption of Kosmo could face challenges related to the cost of lidar and imaging hardware, the complexity of processing massive datasets, and ensuring the absolute accuracy and real-time updating of digital twins for highly dynamic environments. These factors might limit its accessibility for smaller developers or niche applications.
