Speech and audio processing basics

In short

Speech and audio processing applies digital signal processing to signals produced by the human vocal tract and to music or environmental sound, so that they can be analysed, compressed, recognised or synthesised. The core idea is the source-filter model: a quasi-periodic or noisy excitation driving a slowly varying vocal-tract filter, which makes short-time analysis (frames of 20-30 ms) the central technique. It is examined in AICTE/CBCS electives such as "Speech and Audio Signal Processing", "Multimedia Signal Processing" and "Digital Speech Processing", and appears in GATE EC/EE under sampling, quantisation, correlation and DFT applications. In industry it underpins telephony codecs (TRAI-regulated voice networks, VoLTE on Jio/Airtel), voice assistants and IVR in Indian languages (Bhashini, AI4Bharat, Reverie, Gnani.ai), hearing aids and automotive voice systems. Interviewers at Qualcomm, Samsung R&D, Texas Instruments, Analog Devices, Sony, Bosch and speech startups ask about MFCC, LPC, pitch detection and codec bit rates.

Explained

Speech production and the source-filter model

• Anatomy: lungs supply airflow; the larynx with vocal folds acts as the source; the vocal tract (pharynx, oral cavity, nasal cavity, 17 cm long in an adult male on average) acts as a time-varying acoustic filter; lips and nostrils radiate the sound.
• Source-filter model: s(n) = excitation e(n) convolved with vocal-tract impulse response v(n) and radiation r(n). Excitation is an impulse train at the pitch period for voiced sounds, white-noise-like for unvoiced sounds.
• Voiced sounds (vowels, /m/, /n/, /z/): vocal folds vibrate, spectrum shows harmonics of F0 spaced at 1/T0, high energy, low zero-crossing rate.
• Unvoiced sounds (fricatives /s/, /f/, /sh/): turbulent noise at a constriction, flat or high-frequency-weighted spectrum, low energy, high zero-crossing rate.
• Plosives/stops (/p/, /t/, /k/): a silence (closure) followed by a burst — transient, not well modelled by steady-state analysis.
• Fundamental frequency F0 (pitch): typically 80-200 Hz (adult male), 150-350 Hz (adult female), up to 400-500 Hz (children). Perceived as voice pitch; carries intonation, stress and emotion, and lexical tone in tone languages.
• Formants: resonances of the vocal tract, labelled F1, F2, F3. Vowel identity is decided mainly by F1-F2 (e.g. /i/ ≈ 300/2300 Hz, /a/ ≈ 700/1200 Hz, /u/ ≈ 350/900 Hz — values vary with speaker). A uniform tube closed at the glottis resonates at (2k−1)c/4L.
• Spectral envelope vs fine structure: envelope (formants) carries phoneme identity; fine structure (harmonic spacing) carries pitch. Cepstral analysis separates the two.
• Classification of Indian speech sounds: phoneme inventories of Indian languages add aspirated stops (kh, gh), retroflex consonants (ṭ, ḍ, ṇ) and nasalised vowels — one reason Western-trained ASR models need Indian-language retraining.

Digitisation, short-time analysis and windowing

• Bandwidth and sampling: telephone speech is band-limited to 300-3400 Hz, sampled at 8 kHz (narrowband); wideband speech 50-7000 Hz at 16 kHz; CD audio 20 Hz-20 kHz at 44.1 kHz; professional audio 48/96 kHz. Nyquist requires fs > 2fmax plus an anti-aliasing filter.
• Quantisation: 8-bit non-uniform (A-law in India/Europe, μ-law in North America/Japan) for telephony, 16-bit linear PCM for high quality, 24-bit in studios. Each extra bit adds about 6 dB of SNR.
• Why short-time analysis: speech is non-stationary overall but quasi-stationary over 20-30 ms (the vocal tract cannot move faster). All analysis is therefore done frame by frame.
• Framing: frame length 20-30 ms (160-240 samples at 8 kHz), frame shift (hop) 10 ms, giving 50-66% overlap and 100 frames/second.
• Windowing: multiplying by a window reduces spectral leakage from abrupt truncation. Rectangular: narrowest main lobe (2/N) but −13 dB sidelobes; Hamming (0.54 − 0.46cos(2πn/(N−1))): main lobe 4/N, sidelobes −43 dB — the standard choice for speech. Hann has a *higher* first sidelobe (−31 dB) at the same 4/N main lobe but faster roll-off; Blackman reaches −58 dB with a 6/N main lobe.
• Pre-emphasis: a first-order filter H(z) = 1 − az⁻¹ with a ≈ 0.95-0.97 boosts high frequencies to offset the −6 dB/octave glottal and radiation roll-off and flatten the spectrum before LPC/MFCC.
• Narrowband vs wideband spectrogram: a long window (≈ 25-50 ms) resolves harmonics (narrowband, horizontal striations); a short window (≈ 5-10 ms) resolves pitch pulses in time (wideband, vertical striations and clear formant bands). Time-frequency resolution is a trade-off set by the window length.

Time-domain and frequency-domain parameters

• Short-time energy (STE): sum of squared windowed samples; high for voiced, low for unvoiced, lowest in silence. Used for endpoint detection and VAD (voice activity detection).
• Short-time average magnitude: sum of |x(n)| — cheaper than STE on fixed-point DSPs, less dominated by large samples.
• Zero-crossing rate (ZCR): number of sign changes per frame; a crude frequency measure. Roughly 10-20 crossings/10 ms for voiced, 40-60 for unvoiced at 8 kHz. STE + ZCR together give the classic voiced/unvoiced/silence decision.
• Short-time autocorrelation: R(k) peaks at multiples of the pitch period for voiced frames; the first significant non-zero-lag peak gives T0. Robust to phase and noise, but costly and prone to pitch doubling/halving.
• Centre clipping before autocorrelation suppresses formant-related secondary peaks and improves pitch estimation.
• STFT and spectrogram: X(n, ω) = Σ x(m)w(n−m)e^−jωm; magnitude plotted against time and frequency gives the spectrogram, the standard visual tool for speech.
• Other pitch detectors: cepstral peak-picking; SIFT (simplified inverse filtering); harmonic product spectrum; YIN (difference function) and RAPT (normalised cross-correlation); pitch from the glottal closure instants in the LP residual.

Linear prediction (LPC) and cepstral analysis

• LPC idea: model each sample as a weighted sum of p past samples, ŝ(n) = Σ aₖ s(n−k). Minimising the mean-squared prediction error gives the normal (Yule-Walker) equations R·a = r, solved by the Levinson-Durbin recursion in O(p²).
• All-pole model: H(z) = G / (1 − Σ aₖ z⁻ᵏ) — poles represent formants. Rule of thumb: p ≈ fs(kHz) + 4 → p = 10-12 at 8 kHz, 16-20 at 16 kHz. Each formant needs a conjugate pole pair.
• LP residual: inverse-filtering the speech by A(z) whitens it; the residual shows sharp pulses at pitch epochs for voiced speech and noise for unvoiced — used in pitch and glottal-closure detection.
• Derived representations: reflection coefficients kᵢ (|kᵢ| < 1 guarantees stability), log area ratios, LSF/LSP (line spectral frequencies) — preferred for quantisation and interpolation in codecs, LPCC (LP cepstral coefficients), PARCOR.
• Advantages of LPC: compact (10-14 parameters per frame), analytically clean, directly gives a synthesis filter, very low bit rate (LPC-10 at 2.4 kbit/s). Limitations: all-pole model cannot represent nasal zeros well, assumes stationarity within the frame, degrades in noise, sensitive to the voiced/unvoiced and pitch decisions.
• Cepstrum: c(n) = IDFT{ log |DFT{x(n)}| }. Because convolution becomes addition after the log, the slowly varying envelope maps to low-quefrency coefficients and the pitch harmonics to a peak at the pitch period (high quefrency). Liftering (filtering in the cepstral domain) separates excitation from vocal tract.
• Homomorphic deconvolution: the formal name for this log-spectrum method of undoing convolution; the complex cepstrum keeps phase, the real cepstrum discards it.

Perception, MFCC and feature extraction

• Hearing range 20 Hz-20 kHz; most speech energy below 4 kHz; the ear's frequency resolution is roughly logarithmic above 500 Hz and nearly linear below it.
• Mel scale: mel = 2595 log₁₀(1 + f/700) — a perceptual pitch scale. Bark scale and critical bands (about 24 of them) model the same effect in codec design.
• Masking: a loud tone makes a nearby weaker tone inaudible (simultaneous/frequency masking); masking also extends briefly in time (temporal masking). This is the basis of all perceptual audio coding.
• MFCC pipeline (the most-asked sequence): pre-emphasis → framing (25 ms / 10 ms hop) → Hamming window → FFT (512-point at 16 kHz) → magnitude/power spectrum → mel filter bank (20-40 triangular filters) → log of filter-bank energies → DCT (decorrelates and compacts) → keep coefficients 1-12 (+ c0 or log energy) → append Δ and ΔΔ → 39-dimensional vector. CMS/CMVN removes channel effects.
• Why the DCT: mel log energies are highly correlated; the DCT makes them nearly uncorrelated so diagonal-covariance Gaussians (in HMM-GMM systems) suffice.
• Alternative features: PLP (perceptual linear prediction), LPCC, log mel filter-bank energies (preferred for deep neural networks, which handle correlated inputs), spectrogram/raw waveform inputs for end-to-end models, and i-vectors / x-vectors / d-vectors for speaker embeddings.
• Prosody features: F0 contour, energy contour, duration, jitter and shimmer — used in emotion recognition, speaker verification and text-to-speech naturalness.

Speech coding and audio compression

• Three families:
  1. Waveform coders reproduce the waveform: PCM (G.711, 64 kbit/s), ADPCM (G.726, 16-40 kbit/s), sub-band and transform coders. Robust, speaker-independent, high bit rate.
  2. Vocoders (parametric) transmit model parameters only: channel vocoder, formant vocoder, LPC-10 (2.4 kbit/s), MELP. Very low bit rate, synthetic "buzzy" quality.
  3. Hybrid (analysis-by-synthesis): CELP family — RPE-LTP (GSM full rate, 13 kbit/s), ACELP in G.729 (8 kbit/s), AMR-NB (4.75-12.2 kbit/s, 8 kHz), AMR-WB / G.722.2 (6.6-23.85 kbit/s, 16 kHz, used for VoLTE HD voice in India), EVS (used on 4G/5G, includes super-wideband). Best quality per bit for speech.
• CELP mechanics: short-term LP filter (vocal tract) + long-term predictor/adaptive codebook (pitch) + a fixed stochastic codebook searched so that the perceptually weighted error between synthesised and original speech is minimised — hence "analysis by synthesis".
• Audio coding (music): MP3 (MPEG-1 Layer III), AAC (MPEG-2/4, better at the same rate), Opus (open, low-latency, 6-510 kbit/s). All use an MDCT filter bank plus a psychoacoustic model that allocates bits so quantisation noise stays below the masking threshold; stereo redundancy is removed by M/S or intensity stereo; Huffman coding follows.
• Quality measures: MOS (mean opinion score 1-5; toll quality ≈ 4.0), subjective ACR/DCR listening tests, and objective PESQ (ITU-T P.862) and POLQA (P.863) for narrow/wideband voice; SNR and segmental SNR only for waveform coders. Perceptual coders can have poor SNR yet excellent MOS.
• Typical rates: uncompressed CD stereo 1.411 Mbit/s; transparent AAC ≈ 96-128 kbit/s stereo; telephone PCM 64 kbit/s; VoLTE AMR-WB ≈ 12.65 kbit/s typical.

Applications: recognition, synthesis and enhancement

• ASR (automatic speech recognition) pipeline: front end (MFCC/filter-bank) → acoustic model → pronunciation lexicon → language model → decoder (Viterbi/beam search) → text. Classical acoustic model: HMM-GMM with 3-state left-to-right phone HMMs and triphone tying; then DNN-HMM; now end-to-end CTC, attention/encoder-decoder and transformer/conformer models, and self-supervised pretraining (wav2vec-style).
• DTW (dynamic time warping): dynamic-programming alignment of two feature sequences of unequal length; the classic isolated-word template-matching method and a standard numerical exam question.
• Recognition task types: isolated word vs continuous; speaker-dependent vs independent; small vs large vocabulary; keyword spotting/wake-word. Measured by WER = (S + D + I)/N.
• Speaker recognition: *identification* (who among N) vs *verification* (is it the claimed person — a binary decision with FAR/FRR and EER); text-dependent vs text-independent. Used in banking voice authentication; vulnerable to replay and voice-conversion spoofing.
• TTS (text-to-speech): text normalisation → grapheme-to-phoneme → prosody prediction → waveform generation. Methods: formant synthesis, concatenative/unit selection, HMM/statistical parametric, and neural (Tacotron-style acoustic model + WaveNet/HiFi-GAN vocoder). Indian-language TTS from IIT Madras (Donate-a-Voice/IndicTTS) and Bhashini supports screen readers and public-service IVR.
• Speech enhancement: spectral subtraction (simple, causes "musical noise"), Wiener filtering, MMSE-STSA, Kalman filtering, subspace methods, and DNN masking; beamforming with microphone arrays for spatial selectivity; AEC (acoustic echo cancellation, NLMS-based) and ANC (active noise control) in headsets.
• Other audio tasks: sound-event detection, music information retrieval (beat, chord, genre), audio fingerprinting (Shazam-type), 3D/spatial audio, dereverberation, source separation.
• Standards and Indian practice: ITU-T G-series for codecs and quality, 3GPP for AMR/AMR-WB/EVS in mobile networks, MPEG for audio coding, ITU-T P.56 for speech level measurement. TRAI regulates call-quality benchmarks for Indian operators; the Ministry of Electronics and IT's Bhashini / National Language Translation Mission drives Indian-language speech datasets and APIs, and AI4Bharat (IIT Madras) releases open ASR/TTS corpora. Specific TRAI thresholds and Bhashini language coverage change with policy revisions — check the current notification.

Key points

• Source-filter model - speech = excitation (periodic or noise) convolved with a vocal-tract filter; the basis of LPC, vocoders and TTS.
• Quasi-stationarity - speech is analysed in 20-30 ms frames with a 10 ms hop because the vocal tract moves slowly.
• Pitch F0 - 80-200 Hz male, 150-350 Hz female; the reciprocal of the pitch period T0, visible as harmonic spacing.
• Formants - vocal-tract resonances F1-F3; F1/F2 decide vowel identity and appear as LPC pole pairs.
• Voiced vs unvoiced - high energy and low ZCR versus low energy and high ZCR; the standard classification rule.
• Hamming window - −43 dB sidelobes at the cost of a main lobe twice as wide as rectangular; the default for speech.
• Pre-emphasis - 1 − 0.97z⁻¹ flattens the spectral tilt before LPC or MFCC.
• Narrowband vs wideband spectrogram - long window resolves harmonics, short window resolves pitch pulses and formants.
• Autocorrelation pitch detection - peak at lag T0; centre clipping reduces formant-caused false peaks.
• LPC order - p ≈ fs(kHz) + 4; solved by Levinson-Durbin; |reflection coefficient| < 1 ensures stability.
• LP residual - whitened error signal with sharp pulses at glottal closures; carries the excitation.
• Cepstrum - IDFT of the log magnitude spectrum; low quefrency = vocal tract, peak at T0 = pitch; converts convolution into addition.
• Mel scale - 2595 log₁₀(1 + f/700); MFCCs are log mel filter-bank energies passed through a DCT, 12-13 coefficients plus Δ and ΔΔ.
• Masking - loud components hide weaker neighbours; the psychoacoustic basis of MP3/AAC bit allocation.
• Coder classes - waveform (G.711/G.726) > hybrid CELP (G.729, AMR, EVS) > vocoder (LPC-10) in bit rate, and in that same order for robustness and non-speech handling; the reverse order in coding efficiency.
• Telephony chain - 300-3400 Hz, 8 kHz sampling, 8-bit A-law → 64 kbit/s G.711; VoLTE uses AMR-WB at 16 kHz.
• Quality metrics - MOS (subjective, toll quality ≈ 4.0), PESQ/POLQA (objective); SNR is meaningless for perceptual coders.
• ASR metrics - WER = (substitutions + deletions + insertions)/reference words; speaker verification uses FAR, FRR and EER.

Formulas and examples

Example 1 - Telephony digitisation and storage. Narrowband speech, fs = 8 kHz, 8-bit A-law, mono.

• Bit rate R = 8,000 × 8 = 64,000 bit/s = 64 kbit/s (G.711).
• One minute of speech = 64,000 × 60 = 3,840,000 bits = 480,000 bytes ≈ 469 KiB (480 kB).
• The same minute as 16-bit linear PCM at 16 kHz = 16,000 × 16 = 256 kbit/s → 1,920,000 bytes ≈ 1.83 MiB.
• Coded with AMR-NB at 12.2 kbit/s: 12,200 × 60/8 = 91,500 bytes ≈ 89 KiB; compression ratio vs G.711 = 64/12.2 ≈ 5.25:1.

Example 2 - Framing, pitch and frame count. A 2-second utterance at fs = 16 kHz, frame 25 ms, hop 10 ms.

• Total samples = 16,000 × 2 = 32,000. Frame length = 0.025 × 16,000 = 400 samples; hop = 160 samples.
• Number of frames ≈ 1 + (32,000 − 400)/160 = 1 + 197.5 → 198 frames (about 100 frames/second, as expected).
• A voiced frame shows an autocorrelation peak at lag k = 128 samples → T0 = 128/16,000 = 8 ms, F0 = 1/0.008 = 125 Hz → an adult male voice.
• Harmonics in the spectrum are therefore spaced 125 Hz apart; with a 400-sample FFT bin width 16,000/512 ≈ 31.25 Hz (512-point FFT), so harmonics are resolvable — a 5 ms window (80 samples) would not resolve them.

Example 3 - LPC order, SNR and mel conversion.

• At fs = 8 kHz, LPC order p ≈ 8 + 4 = 12 → up to 6 pole pairs, enough for 4-5 formants below 4 kHz plus spectral tilt.
• Quantising to 8 bits uniformly: SNR ≈ 6.02 × 8 + 1.76 = 49.9 dB; to 16 bits: 6.02 × 16 + 1.76 = 98.1 dB. Difference = 8 bits × 6.02 = 48.2 dB.
• Mel value of 1,000 Hz: m = 2595 log₁₀(1 + 1000/700) = 2595 × log₁₀(2.4286) = 2595 × 0.38535 = 1,000.0 mel (the scale is anchored so that 1 kHz ≈ 1000 mel).
• Mel value of 4,000 Hz: 2595 × log₁₀(1 + 5.7143) = 2595 × log₁₀(6.7143) = 2595 × 0.82702 = 2,146 mel. So the 1-4 kHz span (3 kHz wide) occupies only about 1,146 mel, while 0-1 kHz occupies 1,000 mel — confirming the compression of high frequencies.

Example 4 - Vocal-tract length and formants. Uniform tube of length L = 17 cm, c = 340 m/s.

• F1 = c/4L = 340/(4 × 0.17) = 340/0.68 = 500 Hz; F2 = 3c/4L = 1,500 Hz; F3 = 5c/4L = 2,500 Hz — the classic neutral-vowel (schwa) pattern.
• A child with L = 12 cm: F1 = 340/(4 × 0.12) = 340/0.48 = 708 Hz, F2 = 2,125 Hz — formants scale as 1/L, which is why ASR needs vocal-tract length normalisation.

Example 5 - WER and MFCC vector size.

• Reference: 100 words; recogniser output has 6 substitutions, 2 deletions, 3 insertions → WER = (6 + 2 + 3)/100 = 11%, accuracy 89%.
• MFCC front end: 13 static coefficients + 13 Δ + 13 ΔΔ = 39 dimensions per frame. At 100 frames/s and 4 bytes per float, feature stream = 39 × 100 × 4 = 15,600 bytes/s ≈ 15.2 KiB/s — more than coded speech, which is why features are computed at the device, not transmitted.

Where it is used

• Telecom and handsets: codec and voice-quality engineering at Qualcomm, MediaTek, Samsung R&D (Bengaluru/Noida), Nokia and Ericsson India — AMR-WB/EVS tuning, jitter buffers, AEC/ANC; TRAI-driven call-quality audits for Jio, Airtel and Vi.
• Voice AI and Indian languages: ASR/TTS for 22 scheduled languages under Bhashini, AI4Bharat at IIT Madras, and products from Reverie, Gnani.ai, Uniphore, Haptik and Slang Labs; call-centre speech analytics for banks and BPOs.
• Consumer and embedded audio: TWS earbuds, soundbars and smart speakers (Boat, Noise, JBL, Sony) — ANC, beamforming, wake-word detection on DSPs from Texas Instruments, Analog Devices, Cirrus Logic and Cadence Tensilica HiFi cores.
• Banking, government and healthcare: voice biometrics for telephone banking, Aadhaar-era IVR in regional languages, speech-based patient triage, screen readers and assistive TTS for visually impaired users; hearing-aid and cochlear-implant signal processing.
• Media and automotive: broadcast audio loudness and coding at Prasar Bharati and OTT platforms; in-car voice control and hands-free systems at Bosch, Continental, Harman (a large India R&D base) and Tata Elxsi.
• Job roles: DSP/audio firmware engineer, speech scientist/ML engineer, codec validation engineer, acoustics and audio QA — typical interview focus is MFCC, LPC, framing arithmetic, fixed-point implementation and codec bit rates.

Interview questions

• Q: Why is speech analysed in 20-30 ms frames? A: The vocal tract changes shape slowly, so speech is approximately stationary over that span; shorter frames give poor frequency resolution and longer ones blur phoneme transitions.
• Q: How do you decide whether a frame is voiced or unvoiced? A: Voiced frames have high short-time energy, low zero-crossing rate and a clear periodic autocorrelation peak; unvoiced frames have low energy, high ZCR and no periodicity.
• Q: What is the difference between pitch and formants? A: Pitch (F0) is the vocal-fold vibration rate and appears as harmonic spacing; formants are vocal-tract resonances that shape the spectral envelope and determine which vowel is heard.
• Q: Why is a Hamming window preferred over a rectangular one? A: Its sidelobes are about −43 dB against −13 dB, so spectral leakage between harmonics and formants is far lower; the price is a main lobe twice as wide.
• Q: What does the cepstrum do, and why take a logarithm? A: The log turns the convolution of excitation and vocal tract into a sum, so the inverse transform separates them — low quefrency gives the envelope and a peak at the pitch period gives F0.
• Q: Why is a DCT applied at the end of MFCC extraction? A: The log mel filter-bank energies are strongly correlated; the DCT decorrelates and compacts them so only 12-13 coefficients are kept and diagonal-covariance models work.
• Q: How do you choose LPC order? A: About fs in kHz plus 4 — roughly two poles per expected formant plus a few for spectral tilt and radiation; 10-12 at 8 kHz, 16-20 at 16 kHz.
• Q: Why does a 2.4 kbit/s vocoder sound worse than an 8 kbit/s CELP coder? A: The vocoder transmits only model parameters and regenerates a synthetic excitation, while CELP searches a codebook to match the actual residual waveform perceptually.
• Q: Telephone speech is limited to 300-3400 Hz — what is lost? A: Fricative energy and higher formants above 3.4 kHz, which makes /s/ and /f/ confusable, and the F0 fundamental below 300 Hz (though pitch is still perceived from the harmonics).
• Q: Why can a perceptual audio coder have terrible SNR but excellent MOS? A: It deliberately places quantisation noise under the masking threshold, so the noise is inaudible even though it is large in energy terms.
• Q: What is analysis-by-synthesis? A: The encoder synthesises candidate excitations, compares each reconstruction with the original under a perceptual weighting filter and transmits the index of the best — used in CELP-family codecs.
• Q: Why do Western-trained ASR models perform poorly on Indian speech? A: Indian phoneme inventories include retroflex and aspirated sounds, accents and code-mixing differ, and training corpora lack Indian-language data — hence Bhashini and AI4Bharat datasets for retraining.

Common mistakes

• Confusing pitch with formants, or claiming F0 is "the first formant".
• Treating frame shift as frame length, which gives the wrong frame count and the wrong frame rate.
• Forgetting pre-emphasis or windowing before LPC/MFCC, then wondering why the spectral estimate is tilted or leaky.
• Picking the zero-lag autocorrelation peak as the pitch period, or reporting double/half the true F0.
• Omitting the DCT (or applying it before the log) in MFCC, and forgetting Δ/ΔΔ when asked for the feature dimension.
• Mixing bits and bytes, or kbit/s with kB/s, when computing storage and bandwidth.
• Judging a perceptual codec by SNR instead of MOS/PESQ, or assuming a higher sampling rate alone means better speech quality when the codec is the bottleneck.

Related topics

Digital signal processing: filters and FFT · Information theory and source coding · Pattern recognition and machine learning · Mobile communication: 2G to 5G · Image and video processing basics · Embedded DSP architectures and fixed-point implementation