Real-to-Sim Pipeline

Overview

Developed a toolchain to ingest LiDAR/RGB scans of customer facilities, segment them semantically, and auto-generate USD (Universal Scene Description) assets for NVIDIA Isaac Sim.

Problem

Manually recreating customer environments in 3D tools took 2-3 weeks per site. This bottleneck prevented site-specific validation before deployment, leading to costly on-site debugging.

Pipeline Architecture

End-to-end flow from raw scan data to simulation-ready USD stage.

Code Snippet: USD Mesh Generation

Python script using the Pixar USD API to define mesh topology and apply physics colliders programmatically.

from pxr import Usd, UsdGeom, UsdPhysics, Sdf

def create_wall_mesh(stage, path, points, normals):
    # Define mesh prim
    mesh = UsdGeom.Mesh.Define(stage, path)
    
    # Set vertex data
    mesh.CreatePointsAttr(points)
    mesh.CreateNormalsAttr(normals)
    
    # Add collision API
    UsdPhysics.CollisionAPI.Apply(mesh.GetPrim())
    
    # Define material binding
    UsdShade.MaterialBindingAPI.Apply(mesh.GetPrim())
    
    return mesh

# Batch process segmented point clouds
for segment in segments:
    create_wall_mesh(stage, f"/World/Walls/{segment.id}", segment.points, segment.normals)
                    

Approach

• Decimation: Automated downsampling of dense point clouds (50M+ points) to manageable meshes while preserving geometric fidelity for Nav2.
• Semantic Labeling: Used plane fitting (RANSAC) to extract floors and walls, separating them into distinct USD layers.
• Physics Integration: Automatically applied rigid body colliders and friction materials to floor surfaces.

Media & Artifacts

Results

• Reduced environment generation time from 2 weeks to 3 days.
• Enabled site-specific validation of navigation parameters before the robot arrived on site.
• Found 3 critical navigation failures in narrow corridors using the digital twin prior to deployment.