I work in search, and this question comes up with clients for two reasons. Some are building an internal assistant that should answer from company data. Others want to understand how ChatGPT search or Perplexity can answer with today's information, because the same mechanics decide whether their public pages get cited. The architecture is the same in both cases: the model doesn't know your data, so you fetch the right data at question time and put it in front of the model.
Why a model needs help with your data
A large language model is trained on a fixed snapshot of text. It doesn't know your price list, your policies, your tickets or yesterday's sales, and it can't learn them between requests. At answer time it only sees its training plus the text in the request. Every method below is a way of putting the right text, or a live answer from a system, into that request.
The four approaches
| Approach | How it works | Freshness | Best for |
|---|---|---|---|
| Retrieval (RAG) | Search an index of your documents, pass the top passages to the model | As fresh as your index updates (minutes to a day) | Policies, docs, knowledge bases, contracts, past tickets |
| Tool calling | The model calls an API or database query and uses the result | Live | Stock, prices, order status, account data, calendars |
| Long context | Paste whole documents into the request | As fresh as what you paste | One-off analysis of a few long documents |
| Fine-tuning | Further train the model on examples | Frozen at training time | Output format, tone, classification; not changing facts |
Most production systems combine the first two: retrieval for knowledge, tools for live records.
Retrieval-augmented generation, step by step
- Collect sources. Decide which systems count as the source of truth: the docs site, the wiki, the CRM notes, the policy folder. Leave out drafts and duplicates, or the model will quote them.
- Split and label. Break documents into passages of a few paragraphs, keeping headings with them. Store metadata with each passage: source URL, owner, last updated date, and who is allowed to see it.
- Index. Store passages in a search index. Hybrid search, which combines keyword matching with vector embeddings, usually works better than either alone, especially for product codes and names.
- Keep it fresh. Re-index when documents change, using webhooks or change feeds where the source supports them, instead of a nightly full rebuild.
- Retrieve per question. For each question, search the index, filter by the user's permissions, and rerank the results.
- Generate with citations. Pass the top passages to the model with an instruction to answer only from them, cite the source of each claim, and say so when the passages don't contain the answer.
This is also how AI search products work on the open web. ChatGPT search and Perplexity crawl and index pages, retrieve passages for each question and cite them. That's why page structure, dates and crawlability decide which public pages they quote.
Tool calling for live data
When the answer depends on a record that changes constantly, don't copy the data into an index. Give the model a tool instead: a defined function such as get_order_status(order_id) or check_stock(sku, warehouse) that your code runs against the live system. The model decides when to call it and with what arguments, your code executes it with the user's credentials, and the model writes the answer from the result.
The Model Context Protocol (MCP), an open standard Anthropic introduced in November 2024, is now a common way to expose company systems as tools to many different AI applications, without building a separate integration for each one.
Keep tools narrow and read-only where possible. A tool that can only read an order's status is much safer than one that can run any database query.
What to get right before you scale
- Permissions. Filter retrieval and tool results by what the person asking is allowed to see. An assistant that can read every document will happily quote the salary file to anyone.
- Freshness. Show the date of each source in the answer, and expire or re-index stale passages. Most wrong answers in RAG systems come from outdated documents, not from the model.
- Evaluation. Write 50 to 200 real questions with known correct answers, and test every change against them. Check retrieval (did it find the right passage?) separately from generation (did it answer correctly from it?).
- Prompt injection. Retrieved documents and web pages can contain instructions aimed at the model. Treat retrieved text as data, limit what tools can do, and require confirmation for any action that changes something.
- Logging. Record the question, retrieved sources, tool calls and answer, so you can trace a wrong answer to its cause.
Where to start
Pick one narrow use case with clear documents and a clear owner, such as answering questions from your product documentation or HR policies. Build retrieval with citations, measure it on real questions, and only then add tools for live data. If your documentation is also public, the work overlaps with getting it cited by public assistants: see optimizing documentation for ChatGPT and Perplexity.

