BioVITA

BioVITA: Biological Dataset, Model, and Benchmark for Visual-Textual-Acoustic Alignment

CVPR 2026 Main (Highlight)


Risa Shinoda1,2, Kaede Shiohara2, Nakamasa Inoue3, Kuniaki Saito4, Hiroaki Santo1, Fumio Okura1

1The University of Osaka 2The University of Tokyo 3Institute of Science Tokyo

4OMRON SINIC X

arXiv 🤗 Dataset 🤗 Model Code Poster
BioVITA overview: 1.3 million audio clips, 2.3 million images, and 14k species with 34 traits aligned by audio, image, and text encoders for species-level cross-modal retrieval

We introduce BioVITA for biological visual-textual-acoustic alignment. We curate a million-scale dataset, train a unified representation model, and develop a comprehensive species-level retrieval benchmark.

Abstract

Understanding animal species from multimodal data poses an emerging challenge at the intersection of computer vision and ecology. While recent biological models, such as BioCLIP, have demonstrated strong alignment between images and textual taxonomic information for species identification, the integration of the audio modality remains an open problem. We propose BioVITA, a novel visual-textual-acoustic alignment framework for biological applications. BioVITA involves (i) a training dataset, (ii) a representation model, and (iii) a retrieval benchmark. First, we construct a large-scale training dataset comprising 1.3 million audio clips and 2.3 million images, covering 14,133 species annotated with 34 ecological trait labels. Second, building upon BioCLIP2, we introduce a two-stage training framework to effectively align audio representations with visual and textual representations. Third, we develop a cross-modal retrieval benchmark that covers all possible directional retrieval across the three modalities (i.e., image-to-audio, audio-to-text, text-to-image, and their reverse directions), with three taxonomic levels: Family, Genus, and Species. Extensive experiments demonstrate that our model learns a unified representation space that captures species-level semantics beyond taxonomy, advancing multimodal biodiversity understanding.

Video

Dataset

BioVITA Bench

We develop BioVITA Bench, a species-level retrieval benchmark spanning the six cross-modal directions. Our benchmark enables comprehensive analysis from multimodal, ecological, and generalization perspectives. BioVITA Bench retrieval examples: audio-to-image, image-to-audio, audio-to-text, and text-to-audio species retrieval with spectrograms and animal photos

BioVITA Train

We introduce BioVITA Train, a training dataset for VITA alignment. We curate 1.3 million audio clips and 2.3 million images with textual taxonomic annotations, covering 14k species and 34 ecological traits.

Table comparing BioVITA Train with BEANS, AnimalSpeak, iNatSounds, and SSW60 by audio clips, images, taxonomy, species, and traits

BioVITA Model

BioVITA Model

BioVITA Model consists of three encoders. Building upon BioCLIP 2, we train the audio encoder in Stage 1, and jointly train the audio and text encoders in Stage 2.

BioVITA model: audio (HTS-AT), text (Transformer), and image (ViT-L/14) encoders trained in two stages with contrastive losses

Overview of Results

Species-level cross-modal retrieval

Our BioVITA effectively handles all retrieval scenarios and significantly outperforms the tri-modal baselines. Stage 2, which incorporates visual information, further improves all retrieval scenarios by providing complementary cues for robust VITA alignment.

Table and radar charts of top-1 and top-5 species-level retrieval across six audio, image, and text directions, comparing BioVITA with CLIP, CLAP, ImageBind, BioCLIP 2, and TaxaBind

Unseen subset

Despite encountering entirely novel taxa, BioVITA demonstrates robust generalization. Consistent with observations in seen scenarios, the improvement from Stage 1 to Stage 2 underscores the crucial role of incorporating visual modalities for enhancing generalization.

Table of cross-modal retrieval results on the unseen species subset for ImageBind, BioCLIP 2, and BioVITA

Error Analysis

The left plot reports, among all species-level errors, the proportion in which the retrieved sample belongs to the correct genus, and the right plot reports the proportion belonging to the correct family. This suggests that the learned representations successfully capture hierarchical taxonomic structure.

Bar charts of the share of species-level retrieval errors that fall in the correct genus and family, for ImageBind and BioVITA

BibTeX


@article{Shinoda_2026_CVPR,
    author    = {Shinoda, Risa and Shiohara, Kaede and Inoue, Nakamasa and Saito, Kuniaki and Santo, Hiroaki and Okura, Fumio},
    title     = {BioVITA: Biological Dataset, Model, and Benchmark for Visual-Textual-Acoustic Alignment},
    journal   = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2026},
    pages     = {29336-29346}
}