Multimodal LLMs: five student project ideas beyond 'chat with a PDF'

Project ideas that use vision, speech and text together — with the architecture, the hard part and the evaluation for each — chosen to be finishable in a semester and worth showing to a recruiter.

Talking through the products he has shipped
Talking through the products he has shipped

Every third portfolio I review has a PDF chatbot. It is a fine first project and a weak fifth one. Multimodal models — text, images, audio — open up problems that are harder, more interesting and far less crowded. Here are five I would be glad to see from a student, each with the part that will actually be difficult.

1. Lecture-to-notes with slide alignment

Record a lecture, transcribe it live (see real-time speech-to-text architecture), OCR the slides, and align transcript segments to the slide being shown. The hard part is alignment; the payoff is searchable notes with the slide beside every paragraph.

2. Receipt and expense auditor

Photograph receipts, extract fields into a strict schema, validate the arithmetic, and flag anomalies. The hard part is the validation layer, exactly as in document AI for invoices. Evaluate on field-level accuracy, not on "looks right".

3. Accessibility describer for campus notices

Photos of notice boards become structured announcements with dates and venues, read aloud on request. The hard part is handling handwritten and skewed text; the value is real for visually impaired students.

4. Privacy-preserving attendance

Browser-side face detection and embedding with liveness, embeddings-only storage, no photos retained — the FaceVision pattern. The hard part is the threshold and bias measurement across your actual classmates. Document the error rates; that is what makes it a serious project.

5. Voice-first assistant for a regional language

Speech in, retrieval over a small document set, speech out — the shape of Krishi Gyan, the agribot that won Changethon 2025 at IIT Roorkee. The hard part is evaluation in a language you may not have benchmarks for; build a small test set with native speakers.

How to make any of these count

  • Ship a working demo, not a notebook.
  • Publish the evaluation numbers, including failures.
  • Write a two-page README that a recruiter can read in three minutes.

Pick the one whose hard part you find interesting. That is the one you will finish.

MultimodalStudent projectsPortfolioVision
Pranjul Rathour
Pranjul Rathour
GenAI Engineer · Kanpur, Uttar Pradesh, India

GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU Kanpur) and BYTEBATTLE (MeetKats).

Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. Email pranjulrathour41@gmail.com.

Keep reading

More on Vision & speech

All Vision & speech posts →
Campus talks · hackathon judging · mentoring

Want this as a live session at your college?

Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges.

Pranjul Rathour, GenAI engineer in Kanpur