Building a receipt scanner with a vision LLM: a project walkthrough

Extracting merchant, date, line items and total from a photographed receipt using a multimodal model, plus the validation layer that makes it trustworthy.

Requirements gathering, on stage
Requirements gathering, on stage

A receipt scanner is a genuinely good student project: it's visual, it has a clear success metric, and it teaches the exact pattern used in real document-AI products.

The pipeline

  1. Photograph or upload the receipt image.
  2. Send it to a vision-capable model with a prompt asking for a strict JSON schema: merchant, date, line items, total.
  3. Validate the response against the schema and re-prompt once if it fails to parse.
  4. Cross-check: do the line items sum to the stated total, within a small tolerance for rounding?

The check that matters most

Step 4 is the whole project's credibility. A model can extract plausible-looking numbers that don't actually add up; catching that automatically, and flagging the receipt for manual review when they don't, is what separates a toy demo from something you'd trust with real expense data.

See document AI for invoices: an extraction schema for the schema-design half of this problem.

VisionMultimodalProject
Pranjul Rathour
Pranjul Rathour
GenAI Engineer · Kanpur, Uttar Pradesh, India

GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU Kanpur) and BYTEBATTLE (MeetKats).

Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. Email pranjulrathour41@gmail.com.

Keep reading

More on Vision & speech

All Vision & speech posts →
Campus talks · hackathon judging · mentoring

Want this as a live session at your college?

Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges.

Pranjul Rathour, GenAI engineer in Kanpur