Speech Representation, Understanding & Synthesis
A Practitioner's Guide
From the physics of pressure waves to production voice systems. Twelve chapters that connect acoustic fundamentals, modern transformer architectures, and the engineering trade-offs that decide whether speech AI ships. Every concept ships with runnable code.
Speech AI sits at an unusual intersection: signal processing, linguistics, machine learning, and systems engineering. Academic papers assume you know Fourier analysis; DSP textbooks ignore neural networks; ML courses treat audio as just another input modality.
This is a builder's book. Whether you're a machine-learning engineer exploring audio for the first time, a software developer integrating voice into a product, a researcher pushing state of the art, or a technical leader evaluating voice strategies — you'll find the conceptual foundations and the practical implementations together.
“If you can't run it, you don't really understand it.”
Every concept ships with working code. Not toy examples, but real implementations you can run, modify, and extend. We resist the temptation to hide complexity behind library calls. When we use Whisper or wav2vec 2.0, we understand what's happening inside.
Pressure waves, sampling theory, spectrograms, and the perceptual quirks of human hearing that shape how machines represent audio.
From CTC and HMM-GMM through attention-based encoder–decoders. Whisper, wav2vec 2.0, HuBERT, Conformer — and what each gets right.
Neural TTS from WaveNet to VITS. Voice cloning, prosody, voice conversion — and the ethics that have to ride alongside the math.
Audio language models, real-time pipelines, edge deployment, deepfakes, bias, and the engineering trade-offs that decide what ships.
Hover a tile to peek at what to look for — click to read a sample.
The convergence of four disciplines. The human instrument. Applications that listen and speak.
Why audio is fundamentally different from text and images: temporal essence, the fragile dance with noise.
The Fourier transform. Spectrograms. Filtering. The mathematical foundations that don't go away.
HMM-GMM, n-grams, the neural revolution, and the path to end-to-end learning.
Wav2Vec, HuBERT, Conformer, Whisper. Streaming ASR. Self-supervised learning as the dark matter of audio AI.
The mel scale, vector quantization, contrastive learning. What embeddings encode about content, speaker, and emotion.
WaveNet, FastSpeech, diffusion, VITS, VALL-E. Voice cloning, voice conversion, and prosody.
Diarization, verification, audio generation, multimodal fusion, speech translation, deepfakes, bias, privacy.
Discrete audio tokens, SpeechGPT, AudioLM. Real-time conversational voice AI from pipelines to native models.
Robustness, bias mitigation, edge AI, brain-computer interfaces. A speech-AI timeline you can hold in one hand.
Why voice is different from text. Habit inertia, edge architectures, keyword spotting, model compression.
The capstone. Whisper, wav2vec 2.0, SpeechT5, embeddings, end-to-end pipeline, and notebook-to-production deployment.
Every chapter has a pre-executed Jupyter notebook and a CLI script you can run in minutes. The notebooks are committed already executed — you can browse them on GitHub before installing a thing. Click any card to jump straight to the code.
GPU detection, audio as numbers, sampling theory, the Nyquist theorem in code.
Loading Whisper, basic transcription, language detection, and the cost of getting it wrong.
Beam vs. greedy decoding. Word-level timestamps. Long-audio chunking and the silence problem.
Frame-by-frame CTC decoding visualized. See what self-supervised models actually predict at every 20 ms tick.
Text → mel-spectrogram → audio. Speaker embeddings as a control surface. Four distinct voices from one model.
Content vs. speaker. WavLM-SV verifier. Why "the same words in two voices" land far apart in embedding space.
The full cascade: ASR → agent → TTS. End-to-end voice loop on a single GPU, microphone in, synthesized voice out.
STFT, mel-spectrograms, MFCCs, energy envelopes. The visual language for debugging audio models.
Adapting Whisper to Hindi using FLEURS. A gentle introduction to fine-tuning, with WER you can chart against the baseline.
Zero-shot cloning, ethics-first. We build the system, then we discuss what should and shouldn't be built with it.
A dozen fine-tune runs charted side by side. Full fine-tune vs. LoRA vs. frozen-encoder. When does parameter-efficient win?
Qwen2-Audio with bitsandbytes 4-bit quantization. The next paradigm after the cascade — audio as a first-class token.
Notebooks are committed pre-executed — you can read every output on GitHub before installing a thing.
Browse all 12 notebooks →Three free one-page guides pulled from across the book. Designed to be screenshot-able, printable, and immediately useful at your desk. Scan any QR to come back to this site.
More cheat sheets coming: Voice Agent Latency Budget · Speech AI Ethics Checklist
# Bring it home
git clone https://github.com/prdeepakbabu/building-speech-ai.git
cd building-speech-aipython3 -m venv .venv
# macOS / Linux
source .venv/bin/activate
# Windows (PowerShell)
.venv\Scripts\Activate.ps1pip install -r requirements.txt
# Optional — for fine-tuning chapters 9–12
pip install -r requirements-extras.txtpython examples/01_device_setup.py
# → CUDA / MPS / CPU detected, sample rate sanity checks# Generate a 7-second test clip with SpeechT5
python examples/05_speecht5_tts.py \
--text "The quick brown fox jumps over the lazy dog." \
--output samples/test.wav
# Transcribe it with Whisper-tiny
python examples/02_whisper_basic.py --audio samples/test.wavVerified end-to-end on an NVIDIA Tesla T4 (16 GB). Apple Silicon supports chapters 1–8 via MPS; CUDA required for fine-tuning chapters.
Worldwide availability via Amazon (US, UK, EU, Canada, Australia, Japan, India*). Each Amazon marketplace converts the price to local currency at order time. *India: paperback & Kindle only; hardcover not yet supported on Amazon.in.
If this book helped your research, teaching, or production work, please cite it. Citations help self-published technical books reach the readers who'd benefit most.
Piskala, D. B. (2026). Building Speech AI: A practitioner's guide to speech recognition, synthesis, and audio language models with Python. Independently published. ISBN 979-8-249-50140-2.Piskala, Deepak Babu. Building Speech AI: A Practitioner's Guide to Speech Recognition, Synthesis, and Audio Language Models with Python. Independently published, 2026.Piskala, Deepak Babu. 2026. Building Speech AI: A Practitioner's Guide to Speech Recognition, Synthesis, and Audio Language Models with Python. Independently published.@book{piskala2026buildingspeech,
author = {Piskala, Deepak Babu},
title = {Building Speech AI: A Practitioner's Guide to Speech Recognition,
Synthesis, and Audio Language Models with Python},
year = {2026},
publisher = {Independently published},
isbn = {979-8-249-50140-2},
url = {https://www.amazon.com/dp/B0H1ZXP6YS},
note = {Companion code: \url{https://github.com/prdeepakbabu/building-speech-ai}}
}
Paperback ISBN 979-8-249-50140-2 · Hardcover ISBN 979-8-196-45507-0 · Kindle ASIN B0GX2WG2ZC.
Cite the paperback ISBN by default — it is the canonical print edition.
The future of how humans and machines communicate is being written now, by people like you.