Speech recognition and audio AI

In short

Speech recognition (ASR) converts an audio waveform into text; audio AI is the wider family of tasks on sound — text-to-speech, speaker identification, diarization, wake-word spotting, emotion and event detection. Technically it is sequence-to-sequence learning over a variable-length signal, so it introduces alignment problems (CTC, transducers, attention) that no other AI topic covers. Indian exams (GATE DA, university AI/DS electives, speech-processing and DSP papers) ask the front end — sampling, framing, windowing, STFT, mel scale, MFCC — plus HMM-GMM theory and modern end-to-end architectures and WER arithmetic. It is used commercially by Bhashini and AI4Bharat, by voice-bot firms (Skit.ai, Gnani.ai, Uniphore, Ozonetel, Exotel), by Sarvam AI and Krutrim, and in bank/insurance call-centre analytics, where interviewers probe WER, latency and code-mixed Hindi-English handling.

Explained

Speech signals and the front end

• Speech is produced by an excitation source (vocal-fold vibration for voiced sounds, turbulent noise for unvoiced) filtered by the vocal tract — the *source-filter model*. Resonances of the tract are formants (F1, F2 …); the vibration rate is pitch or F0 (roughly 85-180 Hz for adult male, 165-255 Hz female).
• Sampling: by the Nyquist-Shannon theorem, sample above twice the highest frequency. Telephony uses 8 kHz (narrowband, 0-4 kHz, G.711); modern ASR uses 16 kHz mono, 16-bit PCM. Music/audio AI uses 44.1 or 48 kHz.
• Speech is non-stationary overall but quasi-stationary over 20-30 ms, so analysis is done on short frames: typically 25 ms window with a 10 ms hop (overlap), giving 100 frames per second.
• Front-end steps, in order:
  1. Pre-emphasis: y[n] = x[n] − α x[n−1], α ≈ 0.97 — boosts high frequencies that the glottal source rolls off.
  2. Framing and windowing: multiply each frame by a Hamming/Hann window to reduce spectral leakage at frame edges.
  3. FFT → power spectrum of each frame (magnitude squared); together the frames form a spectrogram (time × frequency × energy).
  4. Mel filterbank: 40-80 triangular filters spaced on the mel scale, which is linear below ~1 kHz and logarithmic above, mimicking human pitch perception; take log of each filter's energy → log-mel spectrogram (the standard input to neural ASR).
  5. DCT of the log-mel vector → MFCCs; keep the first 12-13 coefficients plus frame energy. The DCT decorrelates the features so diagonal-covariance Gaussians can be used.
  6. Δ and ΔΔ (velocity, acceleration) coefficients appended → the classic 39-dimensional MFCC vector.
  7. Normalisation: cepstral mean and variance normalisation (CMVN) removes channel/microphone effects.
• Other representations: LPC and LPCC (model the vocal tract as an all-pole filter, used in coding), PLP, filterbank energies, raw waveform fed to a learnt convolutional front end (wav2vec, SincNet), constant-Q transform for music.
• VAD (voice activity detection) segments speech from silence using energy, zero-crossing rate or a small neural net — saves compute and prevents hallucinated text on silence.

Classical ASR: the HMM-GMM pipeline

• Statistical formulation (the noisy channel / fundamental equation of speech recognition): choose the word sequence W maximising P(W|X) ∝ P(X|W)·P(W).
• Acoustic model P(X|W): an HMM per sub-word unit, states emitting feature vectors; emission densities were GMMs (later replaced by a DNN giving state posteriors — the hybrid DNN-HMM).
• Units: phonemes (~40-50 for English; Indian languages are written near-phonetically, which helps), modelled context-dependently as triphones; similar triphone states are clustered by a decision tree (senones) to control parameter count.
• Pronunciation lexicon maps words to phone sequences; a grapheme-to-phoneme (G2P) model handles out-of-vocabulary words.
• Language model P(W): n-gram with smoothing (Kneser-Ney) or a neural LM; measured by perplexity.
• Training: Baum-Welch / EM (forward-backward) for HMM parameters; Viterbi forced alignment to align transcripts to audio; discriminative training (MMI, MPE, later LF-MMI in Kaldi).
• Decoding: Viterbi beam search over a composed WFST (HCLG = HMM ∘ Context ∘ Lexicon ∘ Grammar) — the Kaldi recipe.
• Limitations: many hand-built components, a lexicon per language, independence assumptions, and errors compound across modules. Still used where footprint and interpretability matter.

End-to-end ASR

• One network maps audio features directly to characters/sub-words (BPE, SentencePiece), removing lexicon and HMM.
• CTC (Connectionist Temporal Classification): emits a label or a blank per frame; a *collapse* rule (remove repeats, then blanks) turns the frame-level path into text. The loss sums the probability of all valid alignments via a forward-backward recursion, so no pre-alignment is needed.
  • Assumes conditional independence of outputs given audio → weak implicit LM, so an external LM helps a lot. Output length must be ≤ number of (downsampled) frames.
  • Greedy decoding = argmax per frame; prefix beam search with LM shallow fusion is better.
• RNN-T / Transducer: CTC-style encoder plus a prediction network (LM over previous outputs) and a joint network — removes the independence assumption and is naturally streaming. This is what on-device assistants use.
• Attention encoder-decoder (LAS, Whisper): encoder over audio, autoregressive decoder with cross-attention. Highest accuracy offline; not inherently streaming, and can hallucinate fluent text that was never spoken. Joint CTC-attention training stabilises alignment.
• Architectures: BiLSTM → Transformer → Conformer (convolution + self-attention; convolution captures local phonetic detail, attention captures long context). Self-attention is O(T²), so audio is downsampled (two stride-2 convolutions → 40 ms per frame) before the encoder.
• Streaming vs offline: streaming needs causal/chunked attention and is judged by *latency* and *emission delay*; offline sees the whole utterance and scores better.

Self-supervised learning and foundation audio models

• Labelled speech is expensive (hours of transcription); unlabelled audio is abundant — hence self-supervised pre-training, then fine-tuning on a small labelled set.
• wav2vec 2.0: CNN encoder on the raw waveform, masked latent frames, contrastive loss against quantised targets; fine-tuned with CTC. HuBERT predicts cluster labels of masked frames; WavLM adds denoising/speaker tasks; Whisper instead uses large-scale *weakly supervised* multilingual training (ASR + translation + language ID in one model).
• Practical gain: tens of hours of labelled data can reach what previously needed thousands — the reason low-resource Indian languages became feasible.
• Audio LLMs / speech-to-speech models feed speech encoder outputs into an LLM for spoken question answering and translation; latency and turn-taking replace WER as the key metric.

Evaluation, decoding and deployment

• WER is the standard metric (substitutions + deletions + insertions over reference words), computed from the edit-distance (Levenshtein) alignment; CER is the character version, preferred for agglutinative or heavily inflected languages and for Indic scripts.
• WER can exceed 100% (many insertions). Always state normalisation: case, punctuation, numerals ("₹2,000" vs "two thousand rupees"), filler words — careless normalisation makes WER look better than it is.
• Code-mixing: Hinglish utterances are scored unfairly by WER if transliteration differs; report transliteration-normalised WER or CER.
• Other metrics: perplexity (LM), DER (diarization error rate), EER (speaker verification), MOS and MUSHRA (subjective TTS quality), PESQ (ITU-T P.862) and POLQA (ITU-T P.863) for speech quality, STOI for intelligibility, RTF (real-time factor) for speed.
• Robustness: additive noise, reverberation, channel mismatch, far-field microphones, accents, children's and elderly speech, Lombard effect. Mitigations: SpecAugment (time/frequency masking), speed and volume perturbation, room-impulse-response simulation, multi-condition training, beamforming with mic arrays, speech enhancement front ends.
• Deployment: cloud GPU batch transcription; on-device keyword spotting (a few hundred kB, always-on DSP) gating a larger model; quantisation to int8 and pruning for edge; endpointing to decide when the user stopped speaking.

Beyond ASR: synthesis, speaker and general audio

• TTS pipeline: text normalisation → G2P/phonemisation → acoustic model (Tacotron 2, FastSpeech 2) producing mel spectrograms → vocoder (Griffin-Lim, WaveNet, HiFi-GAN) producing the waveform; VITS is fully end-to-end, generating the waveform directly from text without a separate vocoder. Concatenative and formant synthesis are the older approaches. Prosody (duration, pitch, energy) controls naturalness; voice cloning needs only seconds of reference audio.
• Speaker recognition: identification (1-of-N) vs verification (accept/reject); text-dependent vs text-independent. Embeddings: i-vectors → x-vectors → ECAPA-TDNN, scored by cosine similarity or PLDA; thresholds set from the DET curve at the EER. Anti-spoofing / liveness detection is mandatory against replay and TTS deepfakes.
• Diarization: "who spoke when" — VAD → segmentation → embedding → clustering (spectral/agglomerative) → overlap handling; end-to-end neural diarization (EEND) handles overlap directly.
• Other tasks: spoken language identification, keyword spotting / wake word, speech emotion recognition, audio event detection and acoustic scene classification (AudioSet, UrbanSound; log-mel + CNN is the standard baseline), music information retrieval, source separation, speech enhancement.
• Audio classification trick: treat the log-mel spectrogram as an image and use a CNN or Audio Spectrogram Transformer — but augment in the *time-frequency* domain, not with flips, because frequency axes are not symmetric.

Indian languages, data and law

• Bhashini (Ministry of Electronics and IT, National Language Translation Mission) exposes ASR, TTS and translation APIs for Indian languages through the ULCA data platform; used in government services and the Anuvaad/translation stack.
• AI4Bharat (IIT Madras) releases IndicWav2Vec, IndicASR/IndicConformer, IndicTTS, and datasets Shrutilipi (mined from Prasar Bharati/AIR broadcasts), Kathbath and IndicSUPERB (benchmark), Svarah (accented English). Vakyansh (EkStep/Open Speech) and Mozilla Common Voice add open data; Sarvam AI and Krutrim ship Indic ASR/TTS models.
• Challenges specific to India: 22 scheduled languages and hundreds of dialects, code-mixing with English, Indian-accented English, named entities and numerals, telephony-grade 8 kHz audio, noisy field recordings.
• Legal and compliance: voice is personal data and a voiceprint is biometric data under the Digital Personal Data Protection Act, 2023 — notice and consent, purpose limitation, data-minimisation and breach reporting apply; the detailed DPDP Rules were notified in 2025 and obligations are phasing in, so check current status before designing retention. Call-recording consent, RBI/SEBI record-retention rules for regulated entities, and TRAI telemarketing rules also bite. Separately, call-centre voice biometrics must disclose enrolment, and synthetic-voice (deepfake) misuse is pursued under the IT Act and IT Rules provisions on impersonation and deepfake labelling.

Key points

• Source-filter model - speech = excitation (pitch) × vocal-tract filter (formants); the basis of LPC and synthesis.
• Nyquist rate - sample above 2× the top frequency; 16 kHz for ASR, 8 kHz for telephony.
• Framing - 25 ms window, 10 ms hop ⇒ 100 frames/s; speech is quasi-stationary only over ~20-30 ms.
• Mel scale - perceptual frequency scale, linear below 1 kHz and logarithmic above.
• MFCC - pre-emphasis → window → FFT → mel filterbank → log → DCT; 13 coefficients + Δ + ΔΔ = 39 dims.
• Log-mel spectrogram - preferred over MFCC for neural models, because the DCT's decorrelation is unnecessary and discards information.
• HMM-GMM / DNN-HMM - acoustic model + lexicon + n-gram LM, decoded by Viterbi over a WFST (Kaldi).
• CTC - per-frame labels with a blank, loss = sum over all alignments; conditionally independent outputs, so needs an external LM.
• RNN-T - CTC encoder + prediction + joint network; streaming-friendly and the usual on-device choice.
• Attention encoder-decoder - best offline accuracy (Whisper), but can hallucinate and is not naturally streaming.
• Conformer - convolution for local detail plus self-attention for long context; current standard encoder.
• Self-supervised pre-training - wav2vec 2.0 / HuBERT make low-resource languages viable with tens of labelled hours.
• WER - (S + D + I)/N from the edit-distance alignment; can exceed 100%, and depends heavily on text normalisation.
• CER - character error rate, fairer for Indic scripts and code-mixed text.
• RTF - processing time ÷ audio duration; < 1 means faster than real time.
• Diarization vs recognition - "who spoke when" (DER) versus "what was said" (WER); speaker verification is scored by EER.
• TTS stack - text normalisation → phonemes → acoustic model → vocoder; MOS is the quality metric.
• SpecAugment - time and frequency masking on the spectrogram; the cheapest large WER gain available.

Formulas and examples

Example 1 - framing and feature-tensor size. 16 kHz audio, 2.5 s long, 25 ms window, 10 ms hop, 80 mel filters.

• L_total = 16,000 × 2.5 = 40,000 samples; L_frame = 0.025 × 16,000 = 400 samples; L_hop = 0.010 × 16,000 = 160 samples.
• F = ⌊(40,000 − 400)/160⌋ + 1 = ⌊39,600/160⌋ + 1 = 247 + 1 = 248 frames (≈ 100 frames per second, as expected).
• Log-mel input tensor = 248 × 80 = 19,840 values. After two stride-2 convolutions the encoder sees ⌈248/4⌉ = 62 frames (40 ms each), so a CTC head can emit at most 62 tokens for this utterance.

Example 2 - storage and bit rate. One hour of 16 kHz, 16-bit mono speech.

• R = 16,000 × 16 × 1 = 256,000 bps = 256 kbps = 32,000 bytes/s.
• Size = 32,000 × 3,600 = 115,200,000 bytes ≈ 115.2 MB/hour (≈ 109.9 MiB). A 10,000-hour corpus therefore needs about 1.15 TB raw — why Shrutilipi-scale corpora are distributed as compressed Opus/FLAC.

Example 3 - mel spacing. How far apart are 1 kHz and 4 kHz on the mel scale?

• m(1000) = 2595 × log₁₀(1 + 1000/700) = 2595 × log₁₀(2.4286) = 2595 × 0.38561 = 1,000.7 mel.
• m(4000) = 2595 × log₁₀(1 + 4000/700) = 2595 × log₁₀(6.7143) = 2595 × 0.82724 = 2,146.7 mel.
• A 4× jump in Hz is only a 2.15× jump in mel — so the filterbank places many narrow filters below 1 kHz and few wide ones above 4 kHz, concentrating resolution where phonetic information (formants F1, F2) lives.

Example 4 - WER on a banking utterance.

• Reference (N = 9): "please transfer five thousand rupees to my savings account"
• Hypothesis: "please transfer fifty thousand rupees to my account now"
• Alignment: five → fifty = 1 substitution; "savings" missing = 1 deletion; "now" extra = 1 insertion.
• WER = (1 + 1 + 1)/9 = 3/9 = 33.33%. Note the single substitution is the costly one in business terms (₹5,000 vs ₹50,000) — hence domain-specific numeral post-processing and confirmation prompts in voice-banking flows.

Example 5 - SNR, RTF and CTC alignments.

• Signal power 0.01 W, noise power 1 × 10⁻⁴ W → SNR = 10·log₁₀(100) = 20 dB (clean-office grade; call-centre audio is often 5-15 dB).
• A model transcribes a 30-second clip in 6 seconds → RTF = 6/30 = 0.2, i.e. 5× real time; one GPU can serve about 5 concurrent live streams at this RTF before queuing.
• CTC alignment count: target "ab" with T = 3 frames and alphabet {a, b, ε} gives the paths (a,a,b), (a,b,b), (a,ε,b), (ε,a,b), (a,b,ε) = 5 valid alignments — the loss sums all five, which is why the forward-backward recursion is needed instead of enumeration.

Where it is used

• Voice bots and contact centres: Skit.ai, Gnani.ai, Uniphore, Ozonetel, Exotel and Haptik run Indic ASR + TTS for collections, onboarding and support; call-quality analytics scores agents from transcripts for banks, NBFCs and insurers.
• Government and public digital services: Bhashini APIs in citizen-facing portals, Indian Railways enquiry, agriculture advisories, and speech interfaces for low-literacy users; court and legislature transcription pilots.
• Consumer products: Google and Microsoft India development centres, Jio and Samsung voice assistants, YouTube auto-captions for Indian languages, Android keyboard dictation, media subtitling and dubbing studios.
• Edge and embedded: wake-word chips in TVs, cars and set-top boxes (Tata Elxsi, Bosch, Qualcomm India teams); automotive voice control tuned for cabin noise.
• Security and fintech: speaker verification for telephone banking and KYC, with anti-spoofing against replay and synthetic voice; voice-deepfake detection for fraud teams.
• Job roles: speech/ASR engineer, audio ML engineer, data annotation lead for speech corpora, MLOps for streaming inference, applied research at AI4Bharat, Sarvam AI, Krutrim, Reverie.

Interview questions

• Q: Why are MFCCs computed on 25 ms frames rather than the whole utterance? A: Speech is non-stationary; it is only quasi-stationary over 20-30 ms, so short-time analysis captures a locally valid spectrum while the 10 ms hop tracks change over time.
• Q: Why the mel scale and not linear frequency? A: Human pitch perception is roughly logarithmic above 1 kHz, and phonetic information is concentrated at low frequencies, so mel spacing allocates resolution where it matters and reduces dimensionality.
• Q: Why do neural ASR models prefer log-mel features over MFCCs? A: The DCT in MFCC exists to decorrelate features for diagonal-covariance GMMs; neural networks handle correlated inputs, and the DCT truncation throws away spectral detail.
• Q: What problem does CTC solve? A: It removes the need for frame-level alignment between audio and text by summing the probability of every valid alignment, using a blank symbol and a collapse rule.
• Q: What is the blank token in CTC for? A: It lets the model output "nothing" for a frame and separates genuine repeated characters (as in "letter") from repeats caused by multiple frames on the same sound.
• Q: Why does CTC need an external language model more than an attention model does? A: CTC assumes outputs are conditionally independent given the audio, so it has no internal LM; attention and transducer decoders condition on previous outputs.
• Q: When would you choose RNN-T over Whisper? A: For streaming, low-latency, on-device use — RNN-T emits tokens causally with a small footprint, whereas Whisper is offline, larger and can hallucinate on silence.
• Q: Can WER exceed 100%? A: Yes — insertions are counted but the denominator is only the reference length, so a verbose or hallucinating model can exceed 100%.
• Q: Your model's WER is 8% on a benchmark but 35% on customer calls. Why? A: Domain and channel mismatch: 8 kHz telephony, background noise, accents, code-mixing, domain names and numerals. Fix with in-domain fine-tuning, noise/RIR augmentation and a domain LM or biasing list.
• Q: How do you make an ASR system recognise a product catalogue it has never seen? A: Contextual biasing — boost those sub-word sequences during beam search (shallow fusion with a small list-based LM or an attention-based biasing module) rather than retraining.
• Q: What is diarization and how does it differ from speaker recognition? A: Diarization segments audio by speaker without knowing identities (scored by DER); speaker recognition matches a voice to an enrolled identity (scored by EER).
• Q: What compliance issues arise when you store call recordings and voiceprints in India? A: Voice and voiceprints are personal/biometric data under the DPDP Act, 2023 — consented, purpose-limited collection, minimal retention, breach reporting, plus sector rules from RBI/SEBI and call-recording consent; the DPDP Rules are phasing in, so verify the current obligations.

Common mistakes

• Confusing frame length with frame hop in the frame-count formula, or mixing milliseconds with samples.
• Reporting WER without stating the text normalisation (punctuation, casing, numerals) — it can swing the number by several points.
• Forgetting insertions, or dividing errors by the *hypothesis* length instead of the reference length.
• Claiming CTC does alignment "like an HMM": CTC marginalises over alignments and gives no meaningful single alignment or duration model.
• Applying image-style augmentation (horizontal/vertical flips) to spectrograms; use SpecAugment-style time and frequency masking instead.
• Training at 16 kHz and deploying on 8 kHz telephony audio without resampling or matched fine-tuning — a large, avoidable WER jump.
• Evaluating Indic ASR only by WER on transliterated text, where a script or spelling convention difference is scored as a full word error; report CER as well.

Related topics

Natural language processing basics · Large language models and prompt engineering · Edge AI and model optimisation · Computer vision basics · Text embeddings and semantic search · Time series forecasting