Improving Robotic Manipulation Robustness via NICE Scene Surgery

Sajjad Pakdamansavoji; Mozhgan Pourkeshavarz; Adam Sigal; Zhiyuan Li; Rui Heng Yang; Amir Rasouli

Summary

Creates new robot training data by editing existing demonstrations — removing, restyling, and replacing distractor objects — so manipulation policies stay robust to clutter without collecting more robot data.

Abstract

Learning robust visuomotor policies for robotic manipulation remains a challenge in real-world settings, where visual distractors can significantly degrade performance and safety. In this work, we propose an effective and scalable framework, Naturalistic Inpainting for Context Enhancement (NICE). Our method minimizes out-of-distribution (OOD) gap in imitation learning by increasing visual diversity through construction of new experiences using existing demonstrations. By utilizing image generative frameworks and large language models, NICE performs three editing operations, object replacement, restyling, and removal of distracting (non-target) objects. These changes preserve spatial relationships without obstructing target objects and maintain action-label consistency. Unlike previous approaches, NICE requires no additional robot data collection, simulator access, or custom model training, making it readily applicable to existing robotic datasets. Using real-world scenes, we showcase the capability of our framework in producing photo-realistic scene enhancement. For downstream tasks, we use NICE data to finetune a vision-language model (VLM) for spatial affordance prediction and a vision-language-action (VLA) policy for object manipulation. Our evaluations show that NICE successfully minimizes OOD gaps, resulting in over 20% improvement in accuracy for affordance prediction in highly cluttered scenes. For manipulation tasks, success rate increases on average by 11% when testing in environments populated with distractors in different quantities. Furthermore, we show that our method improves visual robustness, lowering target confusion by 6%, and enhances safety by reducing collision rate by 7%.

Key Contributions

A scene-level editing framework performing three operations — object replacement, restyling, and removal of non-target objects — using image generative models together with large language models.
Edits preserve spatial relationships and never obstruct target objects, keeping action labels valid, so no new action generation or teleoperation is required.
Realism validated against real-world counterparts on both background consistency and overall generation quality.
Demonstrated gains in visual affordance prediction across varying levels of clutter.
Real-world validation of policy robustness and safety across tasks and distractor counts.

Approach

The NICE pipeline: object parsing, distractor selection, and generative scene editing, with action labels preserved throughout.

The object parsing step: object detection with Florence-2 and segmentation used to isolate target and distractor objects.

Results

Generates scenes realistic enough to significantly improve robot perception and downstream manipulation.
Improves policy robustness in environments populated with varying numbers of distractors.
Requires no additional robot data collection and minimal human involvement.

Data enhancement on the Bridge dataset. In each pair, left is the original image and right is the edited one.

Real-world replication of the editing operations, used to validate the realism of NICE-generated data.

FID scores for the three enhancement strategies on real-world images.

(Left) Performance of the manipulation policy finetuned on three data configurations. (Right) Experiment scenes with varying numbers of distractors.

BibTeX

@inproceedings{pakdamansavoji2025improving,
  title     = {Improving Robotic Manipulation Robustness via NICE Scene Surgery},
  author    = {Sajjad Pakdamansavoji and Mozhgan Pourkeshavarz and Adam Sigal and Zhiyuan Li and Rui Heng Yang and Amir Rasouli},
  booktitle = {CoRL Workshop on Making Sense of Data in Robotics: Composition, Curation, and Beyond},
  year      = {2025}
}