How VLAs (Really) Work In Open-World Environments

Amir Rasouli; Yangzheng Wu; Zhiyuan Li; Rui Heng Yang; Xuan Zhao; Charles Eret; Sajjad Pakdamansavoji

Summary

An audit of the top-performing BEHAVIOR-1K challenge policy, examining whether its reported results reproduce, how consistent it is across trials, and how it fails.

Abstract

Vision-language-action models (VLAs) have been extensively used in robotics applications, achieving great success in various manipulation problems. More recently, VLAs have been used in long-horizon tasks and evaluated on benchmarks, such as BEHAVIOR1K (B1K), for solving complex household chores. The common metric for measuring progress in such benchmarks is success rate or partial score based on satisfaction of progress-agnostic criteria, meaning only the final states of the objects are considered, regardless of the events that lead to such states. In this paper, we argue that using such evaluation protocols say little about safety aspects of operation and can potentially exaggerate reported performance, undermining core challenges for future real-world deployment. To this end, we conduct a thorough analysis of state-of-the-art models on the B1K Challenge and evaluate policies in terms of robustness via reproducibility and consistency of performance, safety aspects of policies operations, task awareness, and key elements leading to the incompletion of tasks. We then propose evaluation protocols to capture safety violations to better measure the true performance of the policies in more complex and interactive scenarios. At the end, we discuss the limitations of the existing VLAs and motivate future research.

Key Contributions

In-depth analysis of the leading VLA policy on the B1K Challenge covering reproducibility across independent runs, consistency across task trials, and sources of failure from recorded executions.
Safety-aware evaluation metrics that go beyond progress-agnostic success measures to capture safety violations during execution.
Refined success criteria that better reflect the robot's actual task progress, plus two additional metrics quantifying safety violations.
Architecture-agnostic remedies: safety-aware data curation, structured curriculum design, and deployment-oriented validation.

Approach

Total count of failure occurrences aggregated across all tasks, by failure type.

Results

Reported results do not fully reproduce from the officially released checkpoints.
Performance varies substantially across trials of the same task, which single-number leaderboards hide.
Progress-agnostic success metrics mask safety violations that the proposed metrics expose.

Per-task Q-score across trials, sorted by average Q-score.

Qualitative examples of safety violations that incur a score penalty.

BibTeX

@inproceedings{rasouli2026how,
  title     = {How VLAs (Really) Work In Open-World Environments},
  author    = {Amir Rasouli and Yangzheng Wu and Zhiyuan Li and Rui Heng Yang and Xuan Zhao and Charles Eret and Sajjad Pakdamansavoji},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  year      = {2026}
}