RAG and your own documents

RAG in four steps

1. The user's question is interpreted or rewritten as a search.
2. The system retrieves relevant passages from an index or knowledge base.
3. The passages enter the context with metadata and source labels.
4. The model answers from that material and shows which sources support the answer.

This is not the same as training the model on your documents. The documents are retrieved when needed. Failure can happen in both retrieval and generation.

A good RAG prompt separates instructions from sources

Task: Answer the question from the sources below.

Rules:
- Use only information supported by the sources.
- Put [Source ID] after every factual claim.
- Say "not documented in the sources" when evidence is missing.
- Separate quotations, summaries and your own inference.

Sources:
[S1] Title: ... | Section: ... | Date: ...
<passage>...</passage>

Question: ...

Test retrieval before wording

If the right passage is never retrieved, a better response prompt cannot rescue the system. Create questions with known answers, measure whether the correct sources appear, and test conflicting, stale and missing documents.

When should you not use RAG?

Direct file input is often enough for one bounded document. RAG is most useful when the knowledge base exceeds the context window, changes frequently or needs access controls. Continue with context engineering and citation verification.

Sources

• Microsoft Learn: RAG prompt engineering — context blocks, metadata and citation format
• OpenAI: Prompt engineering — RAG as relevant context from retrieval or file search
• Microsoft Learn: RAG guidance — where RAG fits