RAG pipeline design explained: building, improving and running RAG in production

A RAG pipeline has two halves. Offline ingestion parses documents, chunks them, embeds them and builds indexes. Online query handling rewrites the question if needed, retrieves candidates with keyword and vector search, reranks them, and prompts a model to answer with citations, refusing when evidence is weak. I designed RAG.NextUpgrad this way, with hybrid retrieval, reciprocal rank fusion, reranking, cited streaming, a confidence gate and multi-provider fallback, so this page explains pipeline design, building, improving, optimising and running RAG in production.

How is a RAG pipeline designed?

Design starts from the questions users will ask and the documents that answer them. The ingestion side turns documents into searchable chunks with metadata; the query side finds the right chunks and turns them into a grounded answer. Each stage has one job and can be measured.

Also asked as: rag pipeline design explained · how to setup rag pipeline · how to optimize rag pipeline

Ingestion stages: load, parse or OCR, clean, chunk by structure, enrich with metadata and context, embed, and index for both keyword and vector search. Query stages: rewrite follow-ups, retrieve, fuse results, rerank, check confidence, build the prompt, generate with citations, and log. Keeping stages separate lets you swap components and find which one causes failures.

What is a RAG pipeline in detail?

In detail, a RAG pipeline is a chain of components connecting your data to a language model at question time. Documents become chunks with IDs, text, embeddings and metadata. Questions become queries against indexes, and the best chunks are inserted into a prompt that constrains the model to that evidence.

Also asked as: what is rag pipeline in detail

Key details often missed: storing page numbers and section titles for citations; versioning indexes when embedding models change; deduplicating boilerplate; access control filters so users only retrieve what they may see; and a refusal path when retrieval scores are low. The pipeline also needs observability: logging retrieved chunks and answers so failures can be diagnosed later.

How do I create a simple RAG pipeline?

Load a few documents, split them by paragraphs or headings, embed the chunks with a sentence embedding model, store vectors in memory or a small vector store, retrieve the top four for each question, and prompt a model to answer only from them with citations.

Also asked as: how to create a simple rag pipeline

A minimal stack: Python, a PDF or text loader, Sentence Transformers for embeddings, NumPy or Chroma for search, and any chat model API. Keep it under a hundred lines. Then write twenty questions with known answers and check how many the retriever finds. That evaluation set is what turns a demo into an engineering project, because every later change can be measured against it.

How do I set up a RAG pipeline?

Set up the environment, choose components, build ingestion, build the query service, then add evaluation. A typical setup uses Python, FastAPI for the API, PostgreSQL with pgvector or a vector database, an embedding model and a chat model with keys in environment variables.

Steps:

• Create a project with dependency pinning and a .env file excluded from git.
• Write an ingestion script that parses, chunks, embeds and upserts documents.
• Create indexes: HNSW for vectors and full-text for keywords.
• Build a /ask endpoint that retrieves, prompts and streams answers.
• Add a test set and a script that reports retrieval recall.
• Containerise with Docker for deployment.

How do I improve a RAG pipeline?

Measure first, then fix the weakest stage. Most gains come from retrieval: structure-aware chunking, hybrid keyword and vector search, reranking and query rewriting. Generation improves with clear prompt rules for citations and refusal.

Also asked as: how to improve rag pipeline

A proven improvement order:

• Fix parsing and OCR quality.
• Chunk by headings and add section context to each chunk.
• Add BM25 alongside vectors, fused with reciprocal rank fusion.
• Rerank the top candidates with a cross-encoder.
• Rewrite follow-up questions into standalone queries.
• Add a confidence threshold that refuses weak-evidence answers.

Re-run the evaluation after each change, keeping only changes that help.

How do I optimise a RAG pipeline for speed and cost?

Reduce work per query: retrieve fewer but better chunks, cache embeddings and frequent answers, use smaller models where quality allows, stream responses, and run retrieval steps in parallel. Measure latency per stage to find the bottleneck.

Specific tactics: precompute embeddings at ingestion; tune HNSW search parameters; limit reranking to the top 20 to 30; trim prompts to the best three to five chunks; use prompt caching for stable system prompts; cache answers for repeated questions; and route simple questions to cheaper models. RAG.NextUpgrad was built to run within a small free-tier memory budget, which forced careful choices about model sizes and index settings.

Is there a best way to build a RAG pipeline?

There is no single best pipeline, but there is a best process: start simple, evaluate on real questions, and add complexity only when measurements justify it. The right design depends on documents, questions, latency and cost limits.

That said, a strong default for most document question answering is structure-aware chunking, hybrid retrieval with fusion, a reranker, prompts requiring citations, a refusal threshold, and logging. Frameworks such as LangChain and LlamaIndex speed up building, but understand each component, because debugging requires knowing what happens at every stage.

What does a RAG pipeline that works in production look like?

A production RAG pipeline is reliable, measurable and safe: incremental ingestion keeps indexes current, retrieval is hybrid and reranked, answers cite sources, weak evidence triggers refusal, access control is enforced, and every query is logged for evaluation and debugging.

Also asked as: what a rag pipeline that actually works in production looks like

It also handles failure: timeouts, provider outages with fallback models, malformed documents and empty results. It is monitored with dashboards for latency, cost, refusal rate and user feedback, and evaluated regularly against a fixed test set plus new questions from logs. Changes ship through versioned indexes and prompts, so you can roll back. That is the difference between a demo and a system people trust.

Why can't my RAG pipeline answer relationship questions?

Standard RAG retrieves chunks similar to the question, but relationship questions, such as "Which suppliers are linked to delayed projects?", need facts spread across many documents connected through entities. No single chunk contains the answer.

Also asked as: why your rag pipeline can't answer relationship questions (and how we fixed it

Fixes include building a knowledge graph of entities and relationships during ingestion and querying it, as GraphRAG approaches do; multi-step or agentic retrieval that searches for one entity, then follows connections; storing structured data in SQL tables and answering with queries; and summarising document groups in advance. Choose based on how often users ask relational questions; adding a graph layer adds ingestion cost and maintenance.

Why does my on-prem OCR and RAG pipeline retrieve poorly?

Usually because the OCR text is poor: misread characters, broken words, merged columns, lost tables and repeated headers make chunks that neither keyword nor vector search can match well. Retrieval cannot find what the text does not correctly contain.

Also asked as: why does my on-prem ocr + rag pipeline design lead to poor retrieval performance

Diagnose by reading a sample of chunks: if they are garbled, fix OCR first with better scans, preprocessing, a stronger OCR model or layout-aware parsing that outputs structured text. Also check chunking across page breaks, remove headers and footers, keep tables as rows with column names, and store page numbers. In the OCR and Speech Workspace, page-batched OCR with layout awareness came before any retrieval tuning.

Pranjul Rathour

GenAI Engineer · India

GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU) and BYTEBATTLE (MeetKats).

Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. For campus sessions the inviting institution covers the visit end to end: travel (a First AC train ticket for journeys up to five hours, a flight for anything longer), accommodation, meals, and pickup and drop; the honorarium and workshop fees are negotiable on WhatsApp or on the call, and requests are welcome from institutions that can cover the visit this way. Email pranjulrathour41@gmail.com.

PortfolioInvite me to your campusHire meLinkedInGitHubXInstagramThreadsFacebookBlueskyDev.toBloggerHashnodeSCULTEmail