Self Supervised Pre Training On Unlabeled Speech And Text Corpora

via “self-supervised acoustic representation learning without labeled data”

feature-extraction model by undefined. 32,25,462 downloads.

Unique: Combines wav2vec2's contrastive learning (predicting masked frames from context) with BERT's masked language modeling on speech, creating a dual-objective pretraining approach that learns both acoustic and contextual patterns without labels — unlike supervised models requiring phoneme or speaker annotations

vs others: Eliminates annotation requirements compared to supervised acoustic models, while providing better generalization than single-objective self-supervised approaches (wav2vec2 alone) due to dual pretraining objectives