When we first start working with LLMs, it is easy to think that the model can answer almost anything. You ask a question, the model replies, and the answer may sound confident.

But when we try to build real applications, we quickly run into a problem.

Let's say we are building a chatbot for a company. An employee asks:

What is our company notice period?

Can the LLM answer this correctly by itself? Not always.

The company notice period is private information. Unless we provide that policy to the model, it may not know the correct answer. It might still generate a response, but that response can be a guess.

This is one of the main reasons RAG was introduced.

RAG helps us provide relevant external information to the model before it generates an answer. It improves the quality of answers, but it does not guarantee that every answer will be correct.

So let's understand how RAG works, where it helps, and where it can still fail.

What is RAG?

RAG stands for Retrieval-Augmented Generation.

In simple terms:

RAG = retrieve relevant information first, then generate an answer using that information.

Instead of expecting the model to know everything on its own, we give it useful context.

A basic RAG flow looks like this:

Example:

User asks:
What is our company notice period?

Retrieved context:
Employees must serve a 60-day notice period.

LLM answer:
The company notice period is 60 days.

Here, the model is not answering from random memory. It is answering based on the retrieved context. That is the core idea behind RAG.

Why RAG Was Introduced

LLMs have limits. They may not know:

• private company policies
• internal documents
• latest updates
• uploaded PDFs
• project-specific details
• database content

Also, we cannot always paste an entire PDF, handbook, documentation site, or large knowledge base into a prompt.

There are a few reasons for that:

• The context window has limits.
• Large prompts cost more.
• Large prompts can slow down the response.
• Too much irrelevant context can confuse the model.

RAG solves this by searching first. Instead of sending everything to the model, we retrieve only the most relevant parts.

For example, if a user asks about sick leave, the system should retrieve the sick leave section from the HR policy instead of sending the full HR handbook.

This makes the answer more focused and easier for the model to handle.

How a Basic RAG Pipeline Works

A simple RAG system has two major parts:

1. Indexing Pipeline

The indexing pipeline happens before the user asks a question. This is where we prepare documents for search.

For example:

Company policy PDF
   ↓
Small document chunks
   ↓
Vector embeddings
   ↓
Vector database

The goal is to make documents searchable by meaning, not just exact keywords.

2. Query Pipeline

The query pipeline happens when the user asks a question.

Example:

User asks:
How many sick leaves do I get?

Retrieved chunk:
Employees are eligible for 12 sick leaves per year.

LLM answer:
You get 12 sick leaves per year.

This is how RAG connects external knowledge with the LLM. The model still generates the final answer, but now it has relevant context to work with.

Where RAG Works Well

RAG works well when the answer already exists somewhere in a document or knowledge base.

It is useful for:

• company policy chatbots
• PDF question answering
• documentation assistants
• customer support knowledge bases
• internal team knowledge search
• product FAQ bots
• legal or compliance document search

For example, if a company has an HR policy document, employees can ask:

How many casual leaves do I get?
What is the notice period?
What is the work-from-home policy?
How do I apply for reimbursement?

If the correct information exists in the documents and the system retrieves the right section, RAG can give useful answers.

But this is also where the limitation starts. RAG depends heavily on what gets retrieved. If retrieval is good, the answer has a better chance of being good. If retrieval is poor, the final answer can also go wrong.

Why RAG Sometimes Gives Incorrect Answers

A RAG system has multiple steps. If one step fails, the final response can fail too. The simplest way to understand this is:

Even if the LLM is strong, it cannot reliably answer if we give it the wrong information.

Let's look at the common places where RAG can fail.

1. Poor Retrieval

Retrieval is one of the biggest failure points in RAG.

Let's say the user asks:

What is the company notice period?

The system should retrieve the notice period policy. But imagine it retrieves this instead:

Employees get 12 sick leaves per year.

Now the model receives a chunk about sick leave, not notice period.

It may answer:

The company notice period is 12 days.

This looks like the model made a mistake, but the problem started before generation. The retrieval step brought the wrong context.

This can happen because of:

• poor embeddings
• bad search query
• weak chunk titles
• missing metadata
• similar words or numbers
• bad document structure

This is why, when a RAG system gives a wrong answer, we should first check what context was retrieved. If the retrieved context is wrong, the model is already starting from the wrong place.

2. Missing Context

Sometimes the system does not retrieve the required information at all.

Example:

User asks:
What is the maternity leave policy?

Retrieved context:
This document explains office timing and dress code.

The answer may exist somewhere in the knowledge base, but the retriever failed to bring it. Or the answer may not exist in the knowledge base at all.

In that case, the model should say something like:

This information is not available in the provided context.

But if the prompt is weak, the model may still try to answer.

That is how hallucination can happen even inside a RAG system.

3. Poor Chunking

Chunking means splitting documents into smaller pieces. This sounds simple, but it has a big impact on answer quality. Bad chunking can break important meaning. For example:

Now the full sentence is split badly. If only the first chunk is retrieved, the model does not get the actual answer.

A better chunk would be:

Employees must serve a 60-day notice period before leaving the company.

Poor chunking can lead to:

• missing details
• incomplete answers
• wrong interpretation
• loss of context
• irrelevant retrieval

So chunking is not just about splitting text randomly. A good chunk should preserve meaning.

4. Context Window Limitations

Even with RAG, the LLM still has a context window limit. That means the model can only use the text that is passed into the prompt. A knowledge base may contain hundreds or thousands of pages, but the model will only see the retrieved chunks that we send to it.

If the correct information is not present in those chunks, the answer may be wrong or incomplete.

Also, sending too much context is not always better.

Too much context can:

• increase cost
• slow down the response
• confuse the model
• add unnecessary noise
• push attention away from the important part

So the goal is not to send more context. The goal is to send the right context.

5. Hallucinations Can Still Happen

A common assumption is that RAG completely removes hallucination. But that is not true. RAG can reduce hallucination, but it cannot fully remove it.

Hallucination can still happen when:

• wrong chunks are retrieved
• important context is missing
• the prompt allows guessing
• documents are outdated
• retrieved chunks conflict with each other
• the model ignores part of the context

Example:

Retrieved context:
Employees get 12 sick leaves per year.

User asks:
How many casual leaves do I get?

If casual leave is not mentioned, the model should not guess.

A stronger prompt should say:

Use only the provided context.
If the answer is not present, say "not mentioned in the provided context."
Do not guess.

RAG gives the model context, but the prompt still needs to control how the model uses that context.

6. Outdated Knowledge Base

RAG depends on the knowledge base. If the documents are outdated, the answer will also be outdated.

Example:

Old policy:
Notice period is 30 days.

New policy:
Notice period is 60 days.

If the RAG system still has the old policy, it may answer:

The notice period is 30 days.

Here, the model did not fail by itself. The system gave it outdated information. That is why production RAG systems need a proper update process.

We should ask:

• When was the document last updated?
• Are old documents removed?
• Are duplicate policies handled?
• Is there one source of truth?
• Are document versions tracked?

A RAG system is only as reliable as the knowledge base it retrieves from.

7. Conflicting Documents

Sometimes the knowledge base contains conflicting information.

Example:

Document A:
Notice period is 30 days.

Document B:
Notice period is 60 days.

If both documents are retrieved, the model may get confused. It may choose one, combine both, or give an unclear answer.

A better system should use:

• document metadata
• updated dates
• source priority
• version control
• human review for conflicts

If the provided context has conflicting information, the model should ideally mention that conflict instead of pretending there is one clear answer.

8. When RAG Is Not the Right Solution

RAG is useful when the answer exists in external knowledge. But not every problem needs RAG.

RAG may not be enough when:

• the task requires a live database query
• the system needs to perform an action
• the answer depends on user permissions
• the documents are messy or incomplete
• the data changes very frequently
• the task needs workflow automation

Example:

Cancel my subscription.

RAG alone should not cancel anything. That needs tool/API execution with proper permissions.

Another example:

How many orders did we receive today?

If the answer depends on a live database, RAG over static documents may not be the right approach. The system may need to query the database directly.

So RAG is not a replacement for backend logic, APIs, databases, or tools. It is mainly a way to provide relevant knowledge to the model.

Good Retrieval vs Poor Retrieval

The quality of a RAG answer depends heavily on retrieval.

Same model. Different context. Completely different result. That is why retrieval quality matters so much.

Simple RAG Architecture

Putting the full flow together:

Final Summary

RAG is useful because it helps LLMs answer using external knowledge. It works well for private documents, company policies, PDFs, documentation, FAQs, and knowledge bases. But RAG does not guarantee correctness.

A RAG system can fail because of:

The main lesson is:

RAG improves grounding, but it does not remove the need for good system design.

A reliable RAG system needs:

RAG is a system. The final answer depends on the quality of each step: how documents are prepared, how context is retrieved, how the prompt is written, and how the response is validated.