A ten-second wait followed by a wall of text feels broken. The same answer streamed word by word feels fast, even when total time is identical. Every AI product I ship streams, and Server-Sent Events over FastAPI is the simplest way to do it.
Why SSE over WebSockets
Token streaming is one-directional: the server talks, the client listens. SSE is plain HTTP, works through most proxies, reconnects automatically in the browser, and needs no upgrade handshake. WebSockets earn their complexity when the client also streams — live speech-to-text is the case where I use them instead.
The endpoint shape
- Accept the question and any document scope as a POST, validate, and run retrieval before opening the stream — retrieval errors should return a normal HTTP error, not a broken stream.
- Return a StreamingResponse with media type text/event-stream and an async generator that yields events.
- Send a first event carrying metadata: the citation map, the retrieval confidence, and a request id.
- Yield token events as the provider streams them; end with a done event that includes usage and timing.
Handling failure mid-stream
Once headers are sent you cannot change the status code, so errors must travel inside the stream as a typed event. RAG.NextUpgrad's multi-provider fallback works at this layer: if the primary provider fails before the first token, the generator switches providers and the client never notices; if it fails mid-answer, the client receives an error event and a retry hint rather than a truncated paragraph.
Pitfalls
- Reverse proxies buffering the response — disable buffering for the stream route and send a comment line every few seconds as a heartbeat.
- Idle timeouts on free hosts — heartbeats also keep the connection alive during a slow reranking pass.
- Forgetting to flush — yield small strings; do not accumulate.
- Client-side, use fetch with a reader rather than EventSource if you need POST bodies; EventSource only supports GET.
Streaming is a small amount of code with an outsized effect on how a product feels. It is also the first thing I demo to students who think an AI app is just an API call and a text box.
GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU Kanpur) and BYTEBATTLE (MeetKats).
Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. Email pranjulrathour41@gmail.com.
