LLM / Code / Training
AVTok: 1D Unified Tokenization for Holistic Audio-Video Generation
** Kien T. Pham, I Chieh Chen, Qifeng Chen, Long Chen (HKUST)
AVTok: 1D Unified Tokenization for Holistic Audio-Video Generation
Authors: Kien T. Pham, I Chieh Chen, Qifeng Chen, Long Chen (HKUST)
arXiv ID: 2606.30811
Problem: Existing audio-video generation models use separate per-modality tokenizers, which creates a representation gap, causes semantic misalignment, and requires expensive dual-branch architectures.
Key Methodology:
- AVTok - a dual-stream transformer with shared encoder-decoder and modal-specific learnable queries, encoding an audio-video pair into a compact 1D discrete latent space using a unified codebook (1152 tokens total: 1024 video + 128 audio)
- Video-First-Audio-Later (VFAL) - a 3-stage hierarchical training strategy that progressively trains video reconstruction, then audio reconstruction with frozen video/shared weights, then joint refinement
- Representation alignment loss using CAV-MAE Sync features and a cross-modal AR generative prior (GPT-2) to promote a causally-ordered latent space for autoregressive downstream tasks
Key Results:
- Reconstruction: AVTok achieves 25.62 PSNR, 12.80 rFVD, 0.126 LPIPS (video) and 23.09 SI-SDR, 5.93 rFAD, 1.523 MR-STFT (audio), beating all unimodal baselines (LARP, AdapTok, WavTokenizer, SpectralCodec) on video and competitive on audio
- Audio-to-Video: 150.26 gFVD - 5× lower than TempoTokens (786.61)
- Video-to-Audio: 49.47 gFAD - beats VinTAGe (80.06), V-AURA (126.92), SpecVQGAN (210.07)
- Class-conditional Joint AV Generation: 138.80 gFVD / 56.58 gFAD - 7× better gFVD and 2.3× better gFAD than Ovi and JavisDiT, with a tokenizer+generator that is 23–28× smaller (208M + 632M params) than Ovi (989M + 17.3B)
Applied Context: AVTok replaces the need for separate video and audio codecs in AV generation pipelines, letting builders train a single AR transformer for audio-to-video, video-to-audio, or joint generation - dramatically reducing compute and eliminating cross-modal semantic drift. The architecture is AR-native, so it slots directly into LLM-style generation frameworks with no architectural glue.