Large Language Models are often used through simple chat interfaces, but a lot happens internally before a response is generated.

From Prompt to Response

When a user sends a prompt, the model does not directly understand the text the way humans do. The input first goes through multiple processing steps. Each step converts the text into a form the model can work with.

A simplified flow looks like this:

Understanding this flow makes concepts like embeddings, attention, context windows, and RAG much easier to understand.

1. Text Is Not Directly Understood by the Model

Humans can read a sentence and understand its meaning directly.

For example:

What is your name?

A human understands this as a question asking for someone's name.

But a language model does not process raw human text directly. The text must first be converted into a numerical form because models work with numbers, not plain text.

That conversion starts with tokenization.

2. Tokenization: Splitting Text Into Tokens

Tokenization is the process of splitting text into smaller pieces called tokens.

A token can be:

a word
part of a word
a punctuation mark
a symbol
a space-related piece

For example:

What is your name?

may be split into tokens like:

["What", " is", " your", " name", "?"]

One common misunderstanding is thinking that one word is always one token. That is not always true.

A common word may be one token, but a long or uncommon word may be split into smaller subword tokens.

Example:

unbelievable

could be split like:

["un", "believ", "able"]

Different models can use different tokenizers, so the exact token split can vary.

The important idea is:

Tokenization converts text into smaller pieces that the model can process.

3. Token IDs: Identifying Each Token

After tokenization, each token is mapped to a token ID.

A token ID is a number that identifies a token in the tokenizer's vocabulary.

So the sentence:

What is your name?

may become:

[1500, 690, 920, 3400, 30]

These numbers come from the tokenizer's vocabulary.

The vocabulary is like a lookup table:

Token text      Token ID
------------------------
"What"          1500
"is"            690
"your"          920
"name"          3400
"?"             30

A token ID is only an identifier. It tells the model which token is being used, but it does not carry meaning by itself.

For example, the number 1500 does not naturally mean "What." It only points to the token "What" because the tokenizer vocabulary says so.

A useful analogy is a roll number.

The roll number identifies the student, but it does not describe the student's personality, interests, skills, or relationships.

Similarly:

Token ID = identifier

It identifies the token, but more is needed for meaning.

That is where embeddings come in.

4. Embeddings: Representing Meaning as Vectors

An embedding is a vector representation of a token, word, sentence, or document.

A vector is simply a list of numbers.

"What" → [0.12, -0.44, 0.81, 0.09, ...]
"name" → [0.28, 0.18, -0.39, 0.70, ...]

These numbers are learned during training. They help represent meaning-like patterns in a way the model can process mathematically.

The token ID helps the model find the correct embedding. The embedding gives the model a richer numeric representation.

A simple way to separate them:

For example:

"biryani" → token ID 5001

The ID 5001 only identifies the token.

But the embedding for "biryani" helps place it closer to related ideas like:

rice
food
spicy
pulao
meal

and farther from unrelated ideas like:

keyboard
battery
laptop
server

This does not mean the model stores words in simple human-made boxes. Instead, the model learns patterns across many dimensions.

5. Vector Space: Where Embeddings Can Be Compared

Embeddings live in a mathematical space called vector space.

A simple mental model:

Embedding = address
Vector space = map

Each embedding is like a point on a huge meaning map. Similar meanings are closer together. Different meanings are farther apart.

This is useful because it allows models and AI systems to compare meaning, not just exact words.

For example:

medical leave
sick leave
health-related absence

These phrases do not use the exact same words, but their meanings are related. Embeddings help AI systems understand that relationship.

This is the foundation of semantic search.

6. Keyword Search vs Semantic Search

Keyword search matches exact words. Semantic search matches meaning.

Suppose a user asks:

Can I take medical leave?

But the document says:

Employees get 12 sick leaves per year.

A basic keyword search may look for the exact phrase "medical leave" and miss the relevant sentence.

Semantic search can understand that:

medical leave ≈ sick leave

So it can retrieve the right information even when the wording is different.

This is one of the reasons embeddings are important in systems like:

document search
RAG applications
recommendation systems
customer support bots
resume-job matching
knowledge base search

7. Embeddings Do Not Fully Understand Context Alone

Embeddings are powerful, but they are not the full story.

An embedding can represent the general meaning of a token or text, but the meaning of a word can change depending on the sentence.

Example:

I deposited money in the bank.

Here, bank means a financial institution.

But in this sentence:

I sat near the river bank.

bank means the side of a river.

The word is the same, but the meaning changes based on context.

This is where transformers and attention become important.

8. Transformers: Processing Relationships Between Tokens

A transformer is a deep learning architecture used to process sequences like text.

The main idea behind transformers is attention.

Attention allows the model to decide which tokens are important for understanding another token.

For example:

I deposited money in the bank.

When understanding the token bank, the model should pay attention to deposited and money. These words help the model understand that bank means a financial bank.

Now compare:

I sat near the river bank.

Here, the model should pay attention to sat, near, and river. These words help the model understand that bank means the side of a river.

So a clean difference is:

Embeddings provide meaning-like vectors. Transformers process those vectors together and understand relationships based on context.

9. Self-Attention: Tokens Looking at Other Tokens

Self-attention is a mechanism where tokens in the same input sequence attend to each other.

In simple terms:

Each token looks at other tokens in the same sentence to understand its meaning.

Example:

The cat drank the milk because it was cold.

The word it refers to milk.

Why? Because cold is more likely to describe milk in this sentence.

Now compare:

The cat drank the milk because it was hungry.

Here, it refers to cat.

Why? Because hungry describes the cat.

Self-attention helps the model make these connections.

It is useful for understanding:

pronouns
word relationships
sentence meaning
instructions
longer context

10. Positional Encoding: Why Word Order Matters

Self-attention allows tokens to look at each other, but the model also needs to understand order.

The same words can mean different things when the order changes.

Example:

Dog bites man.
Man bites dog.

Both sentences contain the same words, but the meaning is different.

In the first sentence:

Dog → doing the action
Man → receiving the action

In the second sentence:

Man → doing the action
Dog → receiving the action

This is why transformers need positional information.

A simple way to understand it:

Together, they help the model understand both meaning and order.

token meaning + token position = better context understanding

Prompt order also matters for the same reason.

A messy prompt with conflicting instructions can confuse the model.

Example:

Explain in detail.
Keep it under two lines.
Include all edge cases.
Return only JSON.

The instructions conflict with each other.

A better prompt has clear structure:

Task:
Summarize the message.

Rules:
Keep it under three lines.

Output format:
Return valid JSON only.

Context:
[user message]

Clear order helps the model understand the task, rules, context, and expected output.

11. Context Window: The Model's Working Space

A context window is the maximum amount of tokens a model can process at one time.

It includes:

system instructions
previous conversation included in the request
user input
retrieved context
tool results
model output

The context window is like the model's working desk.

If too much information is placed on the desk, the model may not be able to process everything properly.

For example, sending a full 200-page document to answer one small question is usually a bad approach.

Problems:

too many tokens
higher cost
slower response
important details may get lost
irrelevant information can distract the model
context window may be exceeded

A better approach is:

This is the basic idea behind Retrieval-Augmented Generation, or RAG.

12. How This Connects to RAG

RAG helps an LLM answer using external knowledge.

Instead of expecting the model to know everything, the system retrieves relevant information and gives it to the model as context.

A simple RAG indexing flow:

A simple RAG query flow:

This is why embeddings are important in RAG.

The system is not just searching for exact words. It is searching for meaning.

Example:

User question:
Can I take medical leave?

Relevant document:
Employees get 12 sick leaves per year.

Even though the exact words are different, embeddings help retrieve the correct chunk. Then the LLM uses that retrieved context to generate the final answer.

13. Why Wrong Answers Happen in RAG Systems

If a RAG chatbot gives a wrong answer, the problem is not always the prompt.

A beginner may immediately say:

Update the prompt.

But a GenAI engineer should check the full pipeline.

For example:

User asks:
How many sick leaves do I get?

Wrong retrieved chunk:
Employees get medical insurance coverage for dependents.

Correct chunk:
Employees get 12 sick leaves per year.

This is mainly a retrieval problem.

The model received the wrong context, so changing only the prompt may not solve the real issue.

A better debugging checklist:

Was the correct chunk retrieved?
Are the chunks too large or too small?
Are the documents updated?
Are there conflicting documents?
Is the embedding search confusing similar terms?
Is the prompt asking the model to answer only from context?
Are failed questions and retrieved chunks logged?
Do we have evaluation examples?

This is an important production mindset.

Wrong output can come from different places:

So debugging GenAI systems means checking the pipeline, not just rewriting prompts.

14. Complete Flow From Prompt to Output

Putting everything together:

For a RAG application, the flow includes retrieval:

Each step has a specific role.

Final Summary

A language model does not directly understand text like a human. It processes text through a series of numerical transformations.

The core idea is:

Token IDs identify tokens, but they do not carry meaning by themselves. Embeddings provide meaning-like vector representations. Vector space allows meanings to be compared. Transformers use attention to understand how tokens relate inside a sentence. Positional encoding helps the model understand order. Context windows define how much information the model can process at once. RAG uses embeddings and retrieval to give the model relevant external context before generating an answer.

Once these pieces are clear, LLMs start to feel less like a black box and more like a pipeline that can be understood, debugged, and improved.