Key takeaways
- Whisper in Python is the openai-whisper package: load a model, call transcribe, get text and timestamped segments.
- Whisper needs ffmpeg and PyTorch; if a new Python version fails, use a version PyTorch supports in a virtual environment.
- Whisper expects 16 kHz mono float32 audio; whisper.load_audio or librosa produce the right NumPy array.
- Whisper does not label speakers; combine it with a diarisation model such as pyannote, or use WhisperX.
- Self-host for privacy and volume; use a hosted API for speed to market, streaming and built-in extras.
Whisper in Python usually means openai-whisper, the open-source package for OpenAI's speech recognition model: install it, load a model and call transcribe on an audio file to get text with timestamps. This page is a developer's guide: setup and Python versions, feeding audio as NumPy arrays, word-level timestamps, adding speaker labels, choosing between self-hosted Whisper and a hosted API, apps built on Whisper, and safety. Speech-to-text is part of my OCR & Speech Workspace, so the advice comes from building with it.
What is Whisper in Python?
In Python, Whisper is the openai-whisper library, which runs OpenAI's open-source Whisper speech recognition models locally. It transcribes audio in many languages, can translate speech into English, and returns segments with start and end times.
It depends on PyTorch for the model and ffmpeg for reading audio files. Model sizes range from tiny to large, trading speed for accuracy. Faster community implementations, such as faster-whisper and whisper.cpp, run the same models more efficiently and are popular in production. The Hugging Face Transformers library also provides Whisper models through its speech recognition pipeline, which is convenient if you already use Transformers.
How do I use Whisper AI in Python?
Install ffmpeg on your system, install the package, load a model and transcribe.
Also asked as: how to use whisper ai in python
pip install -U openai-whisper
import whisper
model = whisper.load_model("small") # tiny, base, small, medium, large
result = model.transcribe("lecture.mp3", language="en")
print(result["text"])
for seg in result["segments"][:3]:
print(f'{seg["start"]:.1f}-{seg["end"]:.1f}s: {seg["text"]}')
Set the language when you know it, to avoid misdetection on short clips. Use a GPU if available; on CPU, start with small or try faster-whisper. For translation into English, pass task="translate". The result also includes the detected language and segment timings, which you can turn into SRT subtitles with a few lines of code. For very long recordings on limited memory, split the audio into parts of a few minutes, transcribe each, and keep the offsets so every line still points to the right moment in the original file.
Does Whisper AI work with Python 3.13?
It depends on whether PyTorch and Whisper's other dependencies provide packages for your Python version on your platform. If installation fails on a very new Python, create a virtual environment with a Python version that PyTorch officially supports and install there.
Check PyTorch's installation page for the supported Python versions and your operating system and GPU combination, then install the matching PyTorch build before installing Whisper. Most problems on new Python releases come from compiled dependencies not yet published as wheels, which makes pip try to build from source and fail. Using tools such as pyenv or uv to manage several Python versions side by side avoids breaking your system Python.
How do I convert any audio file to a NumPy array for Whisper?
Whisper expects mono audio sampled at 16 kHz as a float32 NumPy array with values between -1 and 1. Use whisper.load_audio, which calls ffmpeg, or a library such as librosa, then pass the array to transcribe.
Also asked as: how to convert any audio file to np.ndarray for openai whisper
import whisper, librosa
audio = whisper.load_audio("call.m4a") # float32, 16 kHz mono via ffmpeg
# or: audio, _ = librosa.load("call.m4a", sr=16000, mono=True)
result = whisper.load_model("base").transcribe(audio)
If your audio arrives as raw bytes from a microphone or stream, convert integer samples to float32 by dividing by 32768 for 16-bit audio, and resample to 16 kHz. Wrong sample rates produce garbled or sped-up transcripts, a common bug.
How do I use whisper_timestamped in Python?
whisper_timestamped is an open-source extension of Whisper that adds word-level timestamps and confidence scores. Install it, load a model through it, and call its transcribe function; each segment then includes a list of words with start and end times.
Also asked as: how can i properly use whisper_timestamped in python3
import whisper_timestamped as wt
audio = wt.load_audio("talk.mp3")
model = wt.load_model("small")
out = wt.transcribe(model, audio, language="en")
for w in out["segments"][0]["words"][:5]:
print(w["text"], w["start"], w["end"], w["confidence"])
Word timestamps help with subtitles, karaoke-style highlighting and aligning transcripts with slides. Alternatives include WhisperX, which aligns words with a separate model, and the word timestamp option in faster-whisper. Check each project's README for current usage.
Does Whisper do speaker diarisation, and how do I add it?
No. Whisper transcribes what was said but does not label who said it. To add speakers, run a diarisation model, such as pyannote.audio, to find speaker turns, then assign each transcript segment or word to the speaker active at that time. WhisperX combines these steps.
Also asked as: does whisper do speaker diarization? how to add it
The pipeline: transcribe with Whisper and get timestamps; run diarisation to get segments like "Speaker 1 from 0.0 to 12.3 seconds"; align by overlapping times; merge consecutive segments from the same speaker. Diarisation works best with clear audio and distinct voices, and struggles with overlapping speech. Some diarisation models require accepting terms and a Hugging Face token. For meetings, many cloud speech services provide diarisation built in.
AssemblyAI vs Whisper large-v3: which should you ship?
Ship self-hosted Whisper large-v3 when you need privacy, control and low cost at high volume, and can run GPUs. Ship a hosted speech API such as AssemblyAI when you need fast integration, streaming, diarisation and other features out of the box, and accept per-minute pricing and sending audio to a provider.
Also asked as: assemblyai vs whisper large-v3: which stt to ship
Decide with a test: transcribe 20 to 50 of your real recordings with both, compute word error rate against reference transcripts, and check names, numbers and speaker labels. Also compare latency, uptime commitments, data retention terms and total cost, including your engineering time for GPUs, scaling and monitoring. Accuracy differences vary by accent, domain and audio quality, so public benchmarks are only a starting point.
When should I stop self-hosting Whisper?
Consider moving to a hosted API when GPU costs and maintenance time exceed API costs, when you need real-time streaming at scale, when reliability requirements outgrow your setup, or when you need features such as diarisation and custom vocabulary without building them.
Also asked as: when to stop self-hosting whisper (and what you actually gain
Keep self-hosting when privacy or data residency rules require audio to stay on your infrastructure, when volume is high enough that GPUs are cheaper per minute, or when you need full control over models and versions. A useful middle path is a hybrid: self-host batch transcription of sensitive recordings and use an API for live features. Track cost per audio hour and engineer hours monthly to make the decision with numbers.
Does Whisper AI have an app?
OpenAI does not ship Whisper as a standalone consumer app; Whisper is a model and open-source library. Many third-party apps use Whisper for transcription on desktop and mobile, and OpenAI's own products use speech recognition in features such as voice input.
If you want an app, look for transcription apps that state they run Whisper on the device, which keeps audio private, or that use a cloud API, which is faster on weak hardware. Check reviews, privacy policies and whether recordings are uploaded. For developers, a simple personal app is easy to build: a small web page that uploads audio to your own Whisper backend and returns a transcript.
Is Whisper AI safe?
Whisper itself is safe to use: it is open-source software from OpenAI, and running it locally keeps audio on your machine. Safety concerns come from how you use it and which apps or services wrap it.
Points to watch: third-party apps may upload your recordings, so read their privacy policies; Whisper can occasionally produce text that was never spoken, especially during silence or music, which matters for medical, legal or accessibility uses; and recording people requires their consent. Download Whisper and models from official sources, keep dependencies updated, and review transcripts for important decisions.
Sources
GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU) and BYTEBATTLE (MeetKats).
Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. For campus sessions the inviting institution covers the visit end to end: travel (a First AC train ticket for journeys up to five hours, a flight for anything longer), accommodation, meals, and pickup and drop; the honorarium and workshop fees are negotiable on WhatsApp or on the call, and requests are welcome from institutions that can cover the visit this way. Email pranjulrathour41@gmail.com.
