'Is this a good LLM answer' doesn't fit a traditional pass/fail unit test, which is why a lot of student AI projects skip CI entirely. Most of the surrounding code still deserves automated testing.
What to actually test in CI
- Response schema validation — does the output parse into the expected structure, regardless of content quality.
- Retrieval pipeline mechanics — given a known document and query, is the expected chunk in the top results.
- API contract tests — do your endpoints return the right status codes and shapes for valid and invalid input.
- A small, fixed regression set of prompts with an LLM-as-judge or rule-based check, run on every merge to catch obvious quality drops.
What to accept can't be fully automated
Open-ended answer quality still needs periodic human review — CI catches structural breakage, not subtle quality regressions. Both matter; they're not the same job.
See testing LLM applications, a practical guide for the evaluation side this CI setup feeds into.
GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU Kanpur) and BYTEBATTLE (MeetKats).
Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. Email pranjulrathour41@gmail.com.
