How RAG works: chatbots, rerankers, LangChain, image indexing and multimodal pipelines

Retrieval-augmented generation, RAG, makes a language model answer from your documents: it retrieves relevant passages for each question and adds them to the prompt, so the answer is grounded and can cite sources. I built RAG.NextUpgrad with hybrid dense and BM25 retrieval, reciprocal rank fusion, reranking, cited streaming answers and a confidence gate, so this page explains how RAG works, why it is used, RAG chatbots, their limits, LangChain, rerankers, and indexing images and structured data.

How does RAG work?

RAG has two phases. Indexing splits documents into chunks, embeds them and stores them in a search index. At query time, the system retrieves the most relevant chunks for a question and gives them to the language model, which answers using that context.

A full pipeline: parse documents, including OCR for scans; chunk by structure with overlap; embed chunks and build both vector and keyword indexes; for each question, rewrite it if needed, run hybrid retrieval, fuse and rerank results; build a prompt with the top passages and instructions to cite and refuse when unsure; and stream the answer with citations. Evaluation then checks whether retrieval found the right passages and whether answers stayed faithful to them.

Why is RAG used?

RAG is used to give language models access to knowledge they were not trained on, such as private company documents or recent information, while reducing hallucination and enabling citations. Updating documents is far cheaper than retraining a model.

Also asked as: why rag is used

Specific benefits: answers grounded in approved sources; citations users can verify; access control, since retrieval can respect each user's permissions; fresh information without fine-tuning; and smaller models that perform well when given the right context. That is why RAG powers support assistants, internal knowledge search, legal and policy question answering, and study tools. It does not make answers perfect; retrieval quality and prompt design still decide whether the model gets the right evidence.

How does a RAG chatbot work?

A RAG chatbot runs the retrieval pipeline on every message: it turns the user's question, plus relevant conversation history, into a search query, retrieves passages from its knowledge base, and generates a reply grounded in those passages.

Also asked as: how rag chatbot works

Follow-up questions need care. If a user asks "what about the second option?", the chatbot must rewrite it into a standalone query using the conversation, otherwise retrieval fails. Good RAG chatbots also track the sources shown, stream answers with citations, refuse or ask for clarification when evidence is weak, and log questions without good answers so the knowledge base can be improved. The chat interface is simple; the retrieval and grounding behind it do the real work.

What are RAG chatbots?

RAG chatbots are conversational assistants that answer from a specific knowledge base, such as product documentation, company policies or course material, by retrieving relevant passages before generating each answer, rather than relying only on the model's training.

They are common in customer support, HR and IT help desks, education, legal research and healthcare information services. Their strengths are accuracy on domain questions, citations and easy updates. Typical stacks combine a document parser, an embedding model, a vector or hybrid search engine, a reranker and a language model, with a web or messaging interface. The OCR & Speech Workspace I built adds document-scoped RAG with page citations, so answers point to the exact page they came from.

Are all chatbots RAG?

No. Many chatbots answer only from the model's training data, follow scripted flows, or call APIs and tools without retrieving documents. RAG is one design choice, used when answers must come from specific, changing or private knowledge.

Rule-based chatbots, common in older customer service bots, follow decision trees and predefined responses. General assistants like ChatGPT use the model's knowledge and may add web search, which is a form of retrieval. Agentic chatbots call tools such as order lookups or calendars. Many production assistants combine approaches: scripted flows for common tasks, tools for actions, and RAG for knowledge questions. Choose based on what the user needs answered and how often the knowledge changes.

Why do RAG-only chatbots disappoint?

RAG-only chatbots disappoint when retrieval fails, documents are messy, questions need reasoning across many sources, or users want actions rather than answers. A chatbot that can only search and summarise will struggle with tasks like checking an order or comparing data.

Also asked as: why rag-only chatbots suck

Common failures: poor chunking splits answers across fragments; vector-only search misses exact terms; no reranking puts weak passages first; missing query rewriting breaks follow-ups; and no refusal behaviour produces confident wrong answers. Fixes include hybrid retrieval, rerankers, structured data access through tools or SQL, agentic multi-step retrieval for complex questions, evaluation sets and confidence gates. RAG works best as one component in a system, not as the whole product.

Can LangChain be used for RAG?

Yes. LangChain provides document loaders, text splitters, embedding integrations, vector store connectors, retrievers including ensemble retrievers for hybrid search, and chains or LangGraph workflows that assemble prompts with retrieved context.

A basic LangChain RAG app loads documents, splits them with a recursive text splitter, embeds them into a vector store such as Chroma or Qdrant, creates a retriever, and composes a chain that formats retrieved passages into a prompt for a chat model. LangGraph adds control for agentic RAG, such as deciding when to retrieve again. LlamaIndex and Haystack are strong alternatives, and plain Python is viable when you want full control over every retrieval step.

Why is a reranker used in a RAG pipeline?

A reranker improves precision. First-stage retrieval is fast but approximate, so relevant passages may sit at rank 15 instead of rank 1. A reranker, usually a cross-encoder, reads the question and each candidate together and reorders them by true relevance.

Also asked as: why is reranker used in rag pipeline

Since the language model only sees the top few passages, putting the best evidence first directly improves answers and reduces hallucination. Rerankers are slower than embeddings, so they run on a short list, typically the top 20 to 100 fused results. Options include open-source cross-encoders and hosted rerank APIs. In RAG.NextUpgrad, reranking sits between fusion and the confidence gate, so weak evidence is detected before the model writes an answer.

How do you index images for RAG?

Three common approaches: convert images to text with OCR and captions, then index the text; embed images directly with a multimodal model such as CLIP so text queries find similar images; or embed page screenshots with document-retrieval models like ColPali.

Also asked as: how we index images for rag

For scanned documents, OCR with layout extraction is usually the most reliable, since text search and citations stay simple. For charts and diagrams, generate descriptive captions with a vision-language model and store them alongside the image reference. For photo collections, multimodal embeddings work well. Store metadata such as page number, document ID and image path, so answers can cite and display the original image. At answer time, pass the image itself to a vision model when details matter.

How do you build a generalised RAG pipeline for text, images and structured data?

Route each data type to the retrieval method that suits it: text through hybrid search, images through captions or multimodal embeddings, and structured data through SQL or API tools rather than embeddings. A router or agent decides which sources each question needs.

Also asked as: how to develop a generalized rag pipeline for text, images, and structured data

Use a shared ingestion layer that normalises documents into chunks with consistent metadata: source, type, page, permissions and timestamps. Tables can be stored both as text summaries for retrieval and as rows for precise queries. At query time, an agent can run text retrieval, query a database with generated SQL, and fetch image descriptions, then combine the evidence into one cited answer. Evaluate each path separately, since failures in one modality hide easily in combined results.

Pranjul Rathour

GenAI Engineer · India

GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU) and BYTEBATTLE (MeetKats).

Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. For campus sessions the inviting institution covers the visit end to end: travel (a First AC train ticket for journeys up to five hours, a flight for anything longer), accommodation, meals, and pickup and drop; the honorarium and workshop fees are negotiable on WhatsApp or on the call, and requests are welcome from institutions that can cover the visit this way. Email pranjulrathour41@gmail.com.

PortfolioInvite me to your campusHire meLinkedInGitHubXInstagramThreadsFacebookBlueskyDev.toBloggerHashnodeSCULTEmail