Multimodal / Agent / Robotics
Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models
** Nikita Kachaev, Andrey Moskalenko, Matvey Skripkin, Nikita Kurlaev, Daria Pugacheva, Albina Burlova, Mikhail Kolosov, Denis Shepelev, Andrey Kuznetsov, Elena Tutubalina, Aleksandr I. Panov, Alexey K. Kovalev, Vlad Shakhuro
Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models
Authors: Nikita Kachaev, Andrey Moskalenko, Matvey Skripkin, Nikita Kurlaev, Daria Pugacheva, Albina Burlova, Mikhail Kolosov, Denis Shepelev, Andrey Kuznetsov, Elena Tutubalina, Aleksandr I. Panov, Alexey K. Kovalev, Vlad Shakhuro
arXiv ID: 2606.19297
Problem: VLA models fine-tuned from powerful VLMs on robotics data may catastrophically forget commonsense and world knowledge, but existing benchmarks conflate knowledge gaps with low-level control failures.
Key Methodology:
- Act2Answer protocol - adapts VLM knowledge benchmarks into embodied tabletop episodes where agents answer by physically placing an object onto one of two candidate images, isolating knowledge from control confounds.
- Layerwise intent probing - linear classifiers trained on per-layer hidden states (VLM backbone + action head) to localize where answer-relevant information is encoded across model depth.
- 1,720 unique binary questions across 12 knowledge categories (emotion, attribute, state, color, shape, symmetry, counting, time, traffic, public info, celebrity, living world), sourced from MMLB-CompBench, IconQA, MMBench, OK-VQA, and VL-Think.
Key Results:
- 7 VLA models (OpenVLA, π₀, Magma, SpatialVLA, Xiaomi-Robotics-R0, InternVLA-M1, SmolVLA) and 9 VLM baselines evaluated. VLAs score strongly on Color (e.g., OpenVLA 89%, π₀ 86%) and Shape (OpenVLA 64%) but at or near random (50%) on Symmetry, Counting, Time, Normative, Cultural, and Biological categories.
- VLM-VLA gap is 20–40 percentage points: source VLMs far outperform their VLA counterparts, e.g., InternVL3.5-8B scores 95% on Emotion vs. OpenVLA's 49%.
- Magma is the best-performing VLA, reaching 94% on Public Info, 81% on Traffic, 77% on Celebrity, and 77% on Emotion - the only model consistently above chance on rich semantic categories.
- Probing reveals answer-relevant signals peak in middle VLM layers but attenuate in upper action layers (Retention scores: Magma 0.87, OpenVLA 0.77, π₀ 0.36).
- VQA co-training (Magma, Xiaomi-Robotics-R0, InternVLA-M1) is associated with substantially better knowledge retention vs. robotics-only training.
Applied Context: If you're building on VLA models, monitor knowledge retention explicitly - robotics-only fine-tuning silently erodes semantic understanding even when manipulation success looks fine. Prefer architectures with VQA co-training (e.g., Magma-style) to preserve the ability to act on world knowledge, and use Act2Answer-style probes as a diagnostic layer between your VLM backbone and action head.