BrainJanus: A Unified Model for Understanding and Generation across Brain, Vision, and Language

Authors: Haitao Wu, Qirui Zhang, Zhouheng Yao, Shangquan Sun, Qihao Zheng, Mianxin Liu, Chi Zhang, Wanli Ouyang, Chunfeng Song, Changqing Zhang, Jiamin Wu

arXiv ID: 2606.30319

Problem: Existing brain encoding and decoding models treat these as separate tasks using unimodal alignment, ignoring the brain's intrinsic multimodal integration nature.

Key Methodology:

Unified Brain Tokenizer - a VQ-VAE trained from scratch to quantize continuous fMRI signals into discrete tokens, aligned with vision and language tokens in a shared Omni space
All-in-One Autoregressive Transformer - a single Janus-7B backbone using next-token prediction across interleaved brain, vision, and language token sequences, enabling any-to-any generation (image↔brain, text↔brain) in one model
Two-stage training - tokenizer pretraining → joint supervised fine-tuning on all four encoding/decoding tasks simultaneously with LoRA

Key Results:

Brain-to-text decoding: BERTScore 38.12 (surpasses prior SOTA by +7.21) and CLIP score 96.2% (+1.5%)
Brain-to-image decoding: CLIP semantic similarity 94.4% - the only autoregressive method, outperforming diffusion-based baselines in high-level semantic alignment
Zero-shot cross-task generalization (e.g., brain→text model can do brain→image without training on image pairs)
Also reveals and warns against evaluation hacking in fMRI encoding metrics (a trivial "padding" baseline achieves perfect scores by leaking visual embeddings)

Applied Context: BrainJanus is the first unified BCI foundation model that doesn't need separate task-specific pipelines - builders can use a single checkpoint for both reading from (decoding) and writing to (encoding) the brain, unlocking general-purpose neural interfaces that flexibly translate between thoughts, images, and text.