Fine-Tune Whisper For Multilingual ASR with 🤗 Transformers

What changed

Whisper is a pre-trained model for automatic speech recognition (ASR) published in September 2022 by the authors Alec Radford et al. Unlike many of its predecessors, such as Wav2Vec 2.0, which are pre-trained on un-labelled audio data, Whisper is pre-trained on a vast quantity of labelled audio-transcription data, 680,000 hours to be precise. The pre-trained checkpoints achieve competitive results to state-of-the-art ASR systems, with near 3% word error rate (WER) on the test-clean subset of LibriSpeech ASR and a new state-of-the-art on TED-LIUM with 4.7% WER (c.f.

Why it matters

A concrete addition to Practical AI: Tools, Models & Frameworks: it changes what's available to builders today rather than being general commentary.

How it compares

Related prior coverage to compare against:

• Fine-Tune Wav2Vec2 for English ASR in Hugging Face with 🤗 Transformers
• Fine-Tune W2V2-Bert for low-resource ASR with 🤗 Transformers
• Fine-Tune XLSR-Wav2Vec2 for low-resource ASR with 🤗 Transformers

• Fine-Tune Wav2Vec2 for English ASR in Hugging Face with 🤗 Transformers similarity 0.48
• Fine-Tune W2V2-Bert for low-resource ASR with 🤗 Transformers similarity 0.45
• Fine-Tune XLSR-Wav2Vec2 for low-resource ASR with 🤗 Transformers similarity 0.45