Abstract

Object manipulation in the real world requires robotic policies to be robust to contextual perturbations and capable of safe planning in cluttered scenes. However, the majority of existing robotics benchmarks contain clean tabletop environments with success rate as the sole metric, leaving two critical deployment questions unanswered: does the policy manipulate safely under real conditions, and what leads to its failure? Some benchmarks evaluate robustness by applying perturbations in isolation on uncluttered surfaces and relying on binary or count-based collision detection. Others provide realistic environments but omit perturbation evaluation and safety metrics entirely.

To this end, we present RoboPRO, a Robotics benchmark for Perturbation-Resilient and Obstacle-aware planning, which unifies various axes of robustness in manipulation. RoboPRO comprises 80 diverse bimanual atomic and compositional tasks across four furnished indoor environments, with a total of 16,000 demonstrations and 5.6K diverse language instructions. Scenes are populated with different numbers of distractors, ranging from clean (only target objects) to densely populated, imposing both visual and operational clutter. We propose two novel metrics that detect unsafe contact between the robot and the environment or distractors. Our evaluation protocol consists of a factorial analysis of three key perturbations — visual, linguistic, object-level — at multiple distractor densities. We conduct extensive experiments on RoboPRO using state-of-the-art manipulation policies to analyze the impact of visual and operational clutter, perturbations, and instructional diversity. Our findings highlight the key challenges for manipulation in real-world conditions and motivate future research in this domain.