How to combine LLMs with real-time, organization-specific data

AI search · 10 November 2026 · Updated 18 November 2026 · 5 min read

Four ways to give a language model live company data (retrieval, tool calling, long context, fine-tuning), when to use each, and what to get right first.

Rows of white server cabinets in a modern data center

I work in search, and this question comes up with clients for two reasons. Some are building an internal assistant that should answer from company data. Others want to understand how ChatGPT search or Perplexity can answer with today's information, because the same mechanics decide whether their public pages get cited. The architecture is the same in both cases: the model doesn't know your data, so you fetch the right data at question time and put it in front of the model.

Why a model needs help with your data

A large language model is trained on a fixed snapshot of text. It doesn't know your price list, your policies, your tickets or yesterday's sales, and it can't learn them between requests. At answer time it only sees its training plus the text in the request. Every method below is a way of putting the right text, or a live answer from a system, into that request.

The four approaches

ApproachHow it worksFreshnessBest for
Retrieval (RAG)Search an index of your documents, pass the top passages to the modelAs fresh as your index updates (minutes to a day)Policies, docs, knowledge bases, contracts, past tickets
Tool callingThe model calls an API or database query and uses the resultLiveStock, prices, order status, account data, calendars
Long contextPaste whole documents into the requestAs fresh as what you pasteOne-off analysis of a few long documents
Fine-tuningFurther train the model on examplesFrozen at training timeOutput format, tone, classification; not changing facts

Most production systems combine the first two: retrieval for knowledge, tools for live records.

Retrieval-augmented generation, step by step

  1. Collect sources. Decide which systems count as the source of truth: the docs site, the wiki, the CRM notes, the policy folder. Leave out drafts and duplicates, or the model will quote them.
  2. Split and label. Break documents into passages of a few paragraphs, keeping headings with them. Store metadata with each passage: source URL, owner, last updated date, and who is allowed to see it.
  3. Index. Store passages in a search index. Hybrid search, which combines keyword matching with vector embeddings, usually works better than either alone, especially for product codes and names.
  4. Keep it fresh. Re-index when documents change, using webhooks or change feeds where the source supports them, instead of a nightly full rebuild.
  5. Retrieve per question. For each question, search the index, filter by the user's permissions, and rerank the results.
  6. Generate with citations. Pass the top passages to the model with an instruction to answer only from them, cite the source of each claim, and say so when the passages don't contain the answer.

This is also how AI search products work on the open web. ChatGPT search and Perplexity crawl and index pages, retrieve passages for each question and cite them. That's why page structure, dates and crawlability decide which public pages they quote.

Tool calling for live data

When the answer depends on a record that changes constantly, don't copy the data into an index. Give the model a tool instead: a defined function such as get_order_status(order_id) or check_stock(sku, warehouse) that your code runs against the live system. The model decides when to call it and with what arguments, your code executes it with the user's credentials, and the model writes the answer from the result.

The Model Context Protocol (MCP), an open standard Anthropic introduced in November 2024, is now a common way to expose company systems as tools to many different AI applications, without building a separate integration for each one.

Keep tools narrow and read-only where possible. A tool that can only read an order's status is much safer than one that can run any database query.

What to get right before you scale

  • Permissions. Filter retrieval and tool results by what the person asking is allowed to see. An assistant that can read every document will happily quote the salary file to anyone.
  • Freshness. Show the date of each source in the answer, and expire or re-index stale passages. Most wrong answers in RAG systems come from outdated documents, not from the model.
  • Evaluation. Write 50 to 200 real questions with known correct answers, and test every change against them. Check retrieval (did it find the right passage?) separately from generation (did it answer correctly from it?).
  • Prompt injection. Retrieved documents and web pages can contain instructions aimed at the model. Treat retrieved text as data, limit what tools can do, and require confirmation for any action that changes something.
  • Logging. Record the question, retrieved sources, tool calls and answer, so you can trace a wrong answer to its cause.

Where to start

Pick one narrow use case with clear documents and a clear owner, such as answering questions from your product documentation or HR policies. Build retrieval with citations, measure it on real questions, and only then add tools for live data. If your documentation is also public, the work overlaps with getting it cited by public assistants: see optimizing documentation for ChatGPT and Perplexity.

Key takeaways

  • Give the model your data at question time; don't expect it to learn changing facts.
  • Use retrieval with citations for documents, and tool calls for records that change by the second.
  • Store a last-updated date and access permissions with every indexed passage.
  • Test retrieval and answers against a fixed set of real questions before and after every change.
  • Treat retrieved text as untrusted, and keep tools narrow and read-only by default.

FAQ Questions

Questions people ask

What is the difference between RAG and fine-tuning?

RAG fetches relevant documents at question time and gives them to the model, so answers stay current. Fine-tuning changes the model's weights with examples, which suits style and format but freezes facts at training time.

Do I need a vector database for RAG?

You need a search index. Many teams use a vector database, but keyword search engines and databases with vector support also work, and hybrid search often gives the best results.

How do I keep LLM answers up to date with company data?

Re-index documents when they change, store and show their last-updated dates, and use tool calls to live systems for data that changes constantly.

What is MCP?

The Model Context Protocol is an open standard, introduced by Anthropic in November 2024, for connecting AI applications to tools and data sources through a common interface.

Chaitanya Kore in round sunglasses, facing the camera

Author About the writer

Chaitanya Kore

AI & SEO Strategist · Pune · Since 2017

I have worked in SEO since 2017, starting as an analyst. I have led an SEO team of 19 analysts across more than 70 projects for clients in India and Sweden. Today I run SEO strategy, including AEO and GEO, for international SaaS, eCommerce and B2B clients at WSI ALM Corp.

I also work directly with businesses in Pune on a 12-month SEO, AEO and GEO contract, planned one quarter at a time. If anything on a quarter's list isn't delivered, the extra hours are on me.

BBA, Marketing, Pune University · Certified: Google Analytics, HubSpot

Next Get in touch

Plan your first quarter

Based in Pune and working with companies across India. If anything on a quarter's list isn't delivered, the extra hours are on me.