Every LLM provider occasionally times out, rate-limits, or returns a response that doesn't parse as expected. An app with no plan for this fails visibly at the worst possible moment — usually during a demo.
A practical pattern
- Set a sane request timeout, shorter than your users' patience, not the provider's maximum.
- Retry transient failures (timeouts, 5xx errors) with exponential backoff, capped at 2-3 attempts.
- On a parse failure (malformed JSON, missing field), retry once with a stricter prompt before giving up.
- Always have a final fallback message — 'something went wrong, try again' — rather than a raw stack trace reaching the user.
For a demo specifically
Test what your app does when the API is deliberately unreachable, the night before you present it. A graceful failure message reads as professional; a crash reads as unfinished.
See multi-provider LLM fallback for handling this at the provider level, not just the request level.
GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU Kanpur) and BYTEBATTLE (MeetKats).
Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. Email pranjulrathour41@gmail.com.
