AnyGroundBench: A Specialized-Domain Benchmark for Video Grounding in Vision-Language Models

Authors: Rintaro Otsubo, Ryo Fujii, Reina Ishikawa, Taiki Kanaya, Kanta Sawafuji, Hiroki Kajita, Shigeki Sakai, Hideo Saito, Ryo Hachiuma

arXiv ID: 2607.02269

Problem: Current video grounding benchmarks are confined to general daily-life domains and zero-shot evaluation, creating a critical disconnect from real-world specialized-domain applications where models must adapt to rare visual concepts.

Key Methodology:

Covers 5 specialized domains (animal, industry, sports, surgery, public security) with 2,040 videos and 3,522 QA pairs, pairing newly captured expert-annotated videos (mouse scratching by medical experts, American football by players) with established datasets under unified high-fidelity spatio-temporal annotations.
Provides dedicated per-domain training subsets to systematically measure few-shot domain adaptability, shifting STVG evaluation from zero-shot to domain adaptation.
Decomposes STVG into Spatial Video Grounding (SVG) and Temporal Video Grounding (TVG) to isolate failure modes, evaluating 15 VLMs (GPT-5.1, Gemini-3.1-Pro, Qwen3.5, InternVL3, etc.) under zero-shot and 2-shot In-Context Learning.

Key Results:

Best proprietary model (Gemini-3.1-Pro) achieves only 16.5% vIoU@0.3 (Animal), 7.69% (Industry), 1.22% (Sports), 4.16% (Surgery), 22.8% (Public Security) on STVG. Open-source models completely collapse - most score <1% STVG across all domains, with Qwen3-VL-8B hitting 0% SVG on 4 of 5 domains.
Temporal grounding shows promise (Gemini-3.1-Pro: 69.4% tIoU@0.3 on Public Security), but spatial grounding is the primary bottleneck - collapsing STVG under stricter metrics.
ICL helps temporal localization (e.g., Gemini-2.5-Pro Surgery TVG: 31.4% → 41.2%) but frequently degrades spatial accuracy, showing no reliable adaptation benefit.

Applied Context: AnyGroundBench reveals that current VLMs - proprietary and open-source alike - cannot ground objects in specialized domains, with spatial localization as the core failure mode. Builders deploying VLMs in niche fields (medical, industrial, security) should not expect zero-shot or ICL-based adaptation to work; domain-specific fine-tuning of spatial reasoning is a necessary path forward.