How RAG Systems Work for Business Knowledge Bases

Retrieval-augmented generation (RAG) is the pattern behind almost every useful "ask our documents" system. Instead of hoping a language model knows your policies, products or procedures, you find the relevant passages at request time and hand them to the model along with the question. The model's job becomes reading and synthesising, not remembering.

The idea is simple; the engineering is not. Whether people trust a knowledge base or abandon it after a week comes down to the details below.

The request path

At query time a RAG system does five things:

1. Transform the question. Embed it into a vector, and often also extract keywords or rewrite it for search.
2. Retrieve candidates: the top N chunks by semantic similarity, by keyword match, or both.
3. Rerank and filter down to the handful that answer the question, dropping anything the user is not allowed to see.
4. Assemble the prompt from the selected chunks, their sources and the question.
5. Generate an answer grounded in that context, citing the chunks it used.

Each step is a place to get it wrong, but most failed RAG projects I've seen got steps 2 and 3 wrong and blamed the model.

The ingestion path

Before any of that can happen, your content has to be indexed.

Sources

Business knowledge lives in messy places: PDFs, Word documents, Confluence or Notion pages, ticket histories, CRM notes, product databases, email threads. Each source needs a connector that fetches content along with its metadata (title, author, last modified, URL, owner, permissions).

The connector also has to detect changes so the index stays current. A nightly full re-index is fine for a few hundred documents and unworkable for a few hundred thousand. And it should keep the structure: headings, tables and lists. Flatten a table into a paragraph and most of its meaning goes with it.

Chunking

Models have context limits and retrieval works on passages, so documents are split into chunks. This is the most underrated decision in the system.

Split on structure first. Headings and sections make better boundaries than fixed-size windows, so fall back to size-based splitting only inside long sections. Then prepend the document title and section path to each chunk ("Employee Handbook › Leave › Parental leave"), so a chunk read on its own still has context. A small overlap between chunks (10–20%) stops a fact that straddles a boundary from getting lost.

Tables and code need special handling. Keep them whole where you can, and consider embedding a plain-language summary alongside them.

A typical chunk is 200–500 tokens. Smaller chunks retrieve more precisely, while larger ones give the model more context. There is no universal right size; it depends on the content, so measure.

Embeddings

Each chunk goes through an embedding model, which turns text into a vector: a list of numbers where similar meanings sit close together. Choose the model once and record which one you used. Changing embedding models means re-indexing everything, because vectors from different models can't be compared.

Storage

The vectors, the chunk text and the metadata go into a store you can query by similarity and by filter. For most business systems I reach for PostgreSQL with pgvector first. It keeps the vectors next to the relational data, supports metadata filtering in the same query, and your team already knows how to back it up. Dedicated vector databases earn their place at very large scale, or when you need a feature Postgres lacks.

Hybrid search: why semantic alone is not enough

Semantic search finds passages about the meaning of a question. It is poor at exact identifiers such as product codes, error messages, people's names and policy numbers. Keyword search has the opposite strengths and weaknesses.

Production systems run both and merge the results. In Postgres that means a vector similarity query plus a full-text search query, combined with a scoring method such as reciprocal rank fusion:

WITH semantic AS (
  SELECT id, row_number() OVER (ORDER BY embedding <=> $1) AS rank
  FROM chunks WHERE tenant_id = $3
  ORDER BY embedding <=> $1 LIMIT 40
),
keyword AS (
  SELECT id, row_number() OVER (ORDER BY ts_rank_cd(tsv, query) DESC) AS rank
  FROM chunks, plainto_tsquery('english', $2) query
  WHERE tenant_id = $3 AND tsv @@ query
  ORDER BY ts_rank_cd(tsv, query) DESC LIMIT 40
)
SELECT id, SUM(1.0 / (60 + rank)) AS score
FROM (SELECT * FROM semantic UNION ALL SELECT * FROM keyword) merged
GROUP BY id ORDER BY score DESC LIMIT 12;

That one query is most of a retrieval engine. The tenant_id filter is not optional, for reasons covered below.

Reranking

Retrieval returns candidates that are probably relevant. A reranker is a smaller model that scores each (question, chunk) pair and reorders the candidates by how well they answer the question. Cutting the top 30 down to the best 5–8 this way improves answers measurably, and because the generator reads less context, it also lowers cost and latency.

Permissions: the step everyone skips

If a user can't open a document in the source system, the model must not read it on their behalf. This is the most important engineering rule in business RAG and the easiest to break. Index everything into one store, query it for everyone, and you have built an internal data leak with a friendly interface.

What works is to store the source system's permission information (owner, groups, visibility) as metadata on every chunk at ingestion, resolve the current user's groups at query time, and filter inside the retrieval query itself. Filtering after generation is too late, because the model may already have read and summarised something the user shouldn't see. Permissions also change, so re-sync them from the source, not just the content.

Multi-tenant products need the same discipline one level up: tenant isolation on every query, enforced in the data layer, and tested.

Citations and refusal

Two behaviours separate trustworthy answers from plausible ones.

The first is citations. Every answer should show which chunks it drew from, with links back to the source documents. Users do check, and the checking is what builds trust. Citations also make failures diagnosable, since a wrong answer with a citation tells you whether retrieval or generation was at fault.

The second is refusal. If the retrieved context doesn't contain an answer, the model should say so instead of improvising. Instruct it explicitly, and test it with questions you know the documents don't cover.

Evaluating a RAG system

Build an evaluation set early: 30–100 real questions from the people who will use the system, each with the correct answer and the document or documents that contain it.

Then measure retrieval separately from generation. Was the right chunk in the top results? Did the answer match? Retrieval metrics (recall at k, mean reciprocal rank) tell you whether to work on chunking, embeddings or search; generation metrics tell you whether to work on the prompt or the model. Mix the two up and you'll tune the wrong thing.

Run the evaluation on every change. It takes minutes, and it catches the slow decay that sets in when prompts are edited by feel.

Keeping it current

A knowledge base that is a month out of date is worse than none, because people stop trusting it without telling you. Re-index incrementally, on change events or at short polling intervals. Version the chunks so an updated document replaces its old vectors atomically. And show the source's last-modified date in the answer, so users can judge freshness for themselves.

A reference architecture

For a mid-sized business knowledge base, this is the shape I usually build:

• Connectors run as scheduled jobs (Laravel queues or Node workers) and write to an ingestion table.
• A chunking and embedding worker consumes that table and writes chunks, with their metadata and permissions, into Postgres + pgvector.
• A retrieval service exposes one endpoint: question plus user in, ranked and permission-filtered chunks out, using the hybrid query above and a reranker.
• A generation service assembles the prompt, calls the model, parses the citations and returns a structured answer.
• The interface shows the answer and its sources, with a "was this correct?" control that feeds the evaluation set.
• Monitoring tracks retrieval hit rate, refusal rate, latency and spend per query.

None of these parts is exotic. It is ordinary backend engineering, done carefully around a model.

When RAG is the wrong tool

RAG answers questions over content. It is the wrong pattern when the job is to do something, which needs tools and agents. It is also wrong when the answers are arithmetic over structured data (query the database directly), or when the whole corpus is small enough to fit in the prompt (just include it).

Where it does fit is the shared drive of documents that people keep asking the same questions about. There, the permissions filter and the evaluation set matter more than which model you pick.