What is RAG in LLMs? How RAG works, building it, and fixing bad results

RAG, retrieval-augmented generation, is a way of using a large language model where the system first retrieves relevant passages from your own documents and then has the LLM answer from them. It gives LLMs knowledge they were not trained on, keeps answers current without retraining, and lets every answer cite its source. I built RAG.NextUpgrad, a production RAG platform with hybrid retrieval, re-ranking and a confidence gate, so this page explains RAG in LLMs from practice: how it works, how to build and update it, how to evaluate it on a budget, and why results go wrong.

What is RAG in LLMs?

RAG in LLMs means retrieval-augmented generation: before the model answers, a retriever searches a document collection for passages relevant to the question, and those passages are added to the prompt so the LLM answers from them.

The term comes from a 2020 research paper that combined a neural retriever with a generator. Today it describes a broad pattern used in company knowledge bots, customer support, document Q&A and research assistants. The LLM provides language understanding and fluent answers; retrieval provides the facts. Done well, users get specific answers with citations they can check, instead of the model's general and possibly outdated memory.

What is a RAG system?

A RAG system is the full application that implements retrieval-augmented generation: document ingestion and chunking, an index such as a vector store with keyword search, a retrieval and reranking step, a prompt that grounds the LLM, and evaluation and monitoring.

Also asked as: what is rag system

Components in a production system:

• Ingestion: load PDFs, web pages and databases; clean text; keep metadata.
• Chunking: split into passages that keep meaning together.
• Indexing: embeddings in a vector index plus a keyword index.
• Retrieval: hybrid search, merged with reciprocal-rank fusion.
• Reranking: a cross-encoder reorders the top candidates.
• Generation: grounded prompt, citations, refusal on weak evidence.
• Operations: updates, logging, evaluation, cost control.

How does RAG work with an LLM?

The user's question is turned into a search, the most relevant chunks are retrieved and ranked, and the top few are placed into the LLM's prompt with instructions to answer only from them and cite them. The LLM writes the answer.

Also asked as: how does rag work in llm

Step by step: embed the question; search the vector index and the keyword index; merge and rerank results; select the top passages within the context budget; build a prompt with the passages labelled by ID; generate the answer with citations; check that citations point to real passages; and decline if the evidence was weak. Streaming the answer keeps the experience fast. Logging the retrieved passages for every query is what lets you debug wrong answers later.

How does RAG enhance an LLM?

RAG gives an LLM access to private, recent and specific information, reduces hallucinations by grounding answers in sources, enables citations, and lets you update knowledge by updating documents instead of retraining the model.

Also asked as: how does rag enhance llm

It also lowers cost and risk compared with fine-tuning for knowledge: smaller models can answer well when given the right passages, and access control can be applied at retrieval time so users only get answers from documents they are allowed to see. The limits are equally important: RAG cannot fix bad source documents, and if retrieval misses the right passage, the model has nothing correct to work with. That is why retrieval quality is the main lever.

What is the difference between RAG and an LLM?

An LLM is a trained model that generates text from its input. RAG is a system design that adds a retrieval step before the LLM, so the model's input includes relevant documents. RAG uses an LLM; it is not an alternative to one.

Also asked as: difference between rag and llm

Without RAG, the LLM answers from what it learned in training, with no access to your files and no sources. With RAG, it answers from passages your system chose. That changes accuracy, freshness and trust. RAG is also different from fine-tuning, which changes the model's weights; RAG changes only what the model reads at question time, which makes it far easier to update and audit.

How do I build a RAG-based LLM application?

Start with a small, real document set and a list of real questions, build a simple pipeline end to end, measure retrieval and answer quality, then improve the weakest part. Resist adding complexity before you can measure it.

Also asked as: a comprehensive guide for building rag-based llm applications

A build order that works:

• Collect 50 real questions with known answers in your documents.
• Ingest and chunk documents by structure, such as headings, with page numbers in metadata.
• Index with embeddings and BM25; merge results with reciprocal-rank fusion.
• Add a cross-encoder reranker on the top candidates.
• Prompt the LLM to answer only from passages, with citations.
• Add a confidence gate that declines on weak evidence.
• Evaluate after every change.

RAG.NextUpgrad follows this design and runs in about 220 MB on a free tier.

How do I keep information updated in a RAG system?

Re-index incrementally: detect new, changed and deleted documents, re-chunk and re-embed only what changed, and delete outdated chunks so old versions never reach the model. Store document versions and timestamps in metadata.

Also asked as: how to keep updated information in a rag system

Practical habits:

• Give every chunk an ID derived from its document and position, so updates replace rather than duplicate.
• Track a hash of each document to detect changes cheaply.
• Schedule syncs from source systems, or trigger them on changes.
• Filter or boost by date for time-sensitive questions.
• Show the source date in citations so users see how current an answer is.

If you change the embedding model, re-embed everything; vectors from different models are not comparable.

How do I evaluate a RAG system with a local or free-tier model?

Build a test set, measure retrieval with simple metrics that need no model, and use a local model through Ollama or a free API tier as a judge for answer faithfulness, checking a sample by hand to confirm the judge agrees with you.

Also asked as: how to evaluate my rag system using local / free tier api model

Steps:

• Write 50 questions with the passage that answers each.
• Compute recall at 5 and 10 for retrieval: free, fast and deterministic.
• Generate answers and ask a judge model: is every claim supported by the passages?
• Include unanswerable questions and measure how often the system declines.
• Hand-check 20 judged answers to calibrate.

Small local judges are less reliable than large ones, so keep the rubric simple, binary where possible, and trust retrieval metrics most.

Why do RAG hallucinations often start at ingestion?

Because bad ingestion corrupts what the model reads: broken PDF extraction, tables flattened into nonsense, headers repeated in every chunk, lost page structure, or chunks that split a fact from its context. The LLM then faithfully answers from garbled or misleading text.

Also asked as: why your rag system hallucinations start at ingestion, not the llm

Fixes at ingestion:

• Use layout-aware extraction or OCR for scanned and complex PDFs.
• Keep tables as tables, or convert them to clear text rows.
• Strip repeated headers, footers and navigation.
• Chunk by headings and paragraphs, with overlap where meaning spans boundaries.
• Carry titles and section names into each chunk.

Inspect random chunks by eye before indexing; many "model" problems disappear once the text is clean.

Why does a large FAISS RAG system return irrelevant results?

With many chunks, pure semantic search tends to surface passages that are topically similar but not relevant, especially when chunks are noisy, embeddings are mismatched, or the index uses approximate settings that trade recall for speed.

Also asked as: why does my 150k chunks faiss-based rag system return irrelevant results for semantic search queries

Common causes and fixes:

• Mixed or mismatched embeddings: use one model for indexing and queries, and normalise vectors for cosine similarity.
• Noisy chunks: remove boilerplate and duplicates.
• Approximate index settings: raise nprobe for IVF or efSearch for HNSW and measure recall.
• No keyword search: add BM25 and fuse results.
• No reranking: rerank the top 50 with a cross-encoder.
• Wrong scope: filter by metadata such as document type or date.

Measure recall at 10 before and after each change.

Pranjul Rathour

GenAI Engineer · India

GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU) and BYTEBATTLE (MeetKats).

Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. For campus sessions the inviting institution covers the visit end to end: travel (a First AC train ticket for journeys up to five hours, a flight for anything longer), accommodation, meals, and pickup and drop; the honorarium and workshop fees are negotiable on WhatsApp or on the call, and requests are welcome from institutions that can cover the visit this way. Email pranjulrathour41@gmail.com.

PortfolioInvite me to your campusHire meLinkedInGitHubXInstagramThreadsFacebookBlueskyDev.toBloggerHashnodeSCULTEmail