How RAG Systems Work for Business Knowledge Bases

Retrieval-augmented generation (RAG) is the pattern behind almost every useful "ask our documents" system. The idea is simple: instead of hoping a language model knows your policies, products or procedures, you find the relevant passages at request time and hand them to the model along with the question. The model's job becomes reading and synthesising, not remembering.

The idea is simple. The engineering is not, and the difference between a knowledge base that people trust and one they abandon after a week lives entirely in the details below.

The request path#

At query time a RAG system does five things:

1. Transform the question — embed it into a vector, and often also extract keywords or rewrite it for search.
2. Retrieve candidates — pull the top N chunks by semantic similarity, by keyword match, or both.
3. Rerank and filter — narrow to the handful that actually answer the question, drop anything the user is not allowed to see.
4. Assemble context — build the prompt with the selected chunks, their sources and the question.
5. Generate — the model writes an answer grounded in that context, citing which chunks it used.

Every one of those steps is a place to get it wrong. Most failed RAG projects got steps 2 and 3 wrong and blamed the model.

The ingestion path#

Before any of that can happen, your content has to be indexed.

Sources#

Business knowledge lives in messy places: PDFs, Word documents, Confluence or Notion pages, ticket histories, CRM notes, product databases, email threads. Each source needs a connector that can:

• Fetch content and metadata (title, author, last modified, URL, owner, permissions).
• Detect changes so the index stays current — a nightly full re-index is fine for a few hundred documents and unworkable for a few hundred thousand.
• Preserve structure: headings, tables, lists. Flattening a table into a paragraph destroys most of its meaning.

Chunking#

Models have context limits and retrieval works on passages, so documents are split into chunks. This is the most underrated decision in the system.

• Structure-aware chunking beats fixed-size windows. Split on headings and sections first; only fall back to size-based splitting within long sections.
• Keep the breadcrumb. Prepend the document title and section path to each chunk ("Employee Handbook › Leave › Parental leave") so a chunk read in isolation still has context.
• Overlap slightly (10–20%) so a fact split across a boundary is not lost.
• Tables and code are special. Keep them whole where possible, and consider storing a natural-language summary alongside them for embedding.

A typical chunk is 200–500 tokens. Smaller chunks retrieve more precisely; larger chunks give the model more context. There is no universal right answer — it depends on the content, and you should measure.

Embeddings#

Each chunk is passed through an embedding model that turns text into a vector — a list of numbers where similar meanings produce nearby vectors. Choose the model once and record which one you used; changing embedding models means re-indexing everything, because vectors from different models are not comparable.

Storage#

The vectors, the chunk text and the metadata go into a store you can query by similarity and by filter. For most business systems I reach for PostgreSQL with pgvector first: it keeps the vectors next to the relational data, supports metadata filtering in the same query, and your team already knows how to back it up. Dedicated vector databases earn their place at very large scale or when you need features Postgres lacks.

Hybrid search: why semantic alone is not enough#

Semantic search finds passages about the meaning of a question. It is poor at exact identifiers — product codes, error messages, people's names, policy numbers. Keyword search is the opposite.

Production systems run both and merge the results. In Postgres that is a vector similarity query plus a full-text search query, combined with a scoring method such as reciprocal rank fusion:

WITH semantic AS (
  SELECT id, row_number() OVER (ORDER BY embedding <=> $1) AS rank
  FROM chunks WHERE tenant_id = $3
  ORDER BY embedding <=> $1 LIMIT 40
),
keyword AS (
  SELECT id, row_number() OVER (ORDER BY ts_rank_cd(tsv, query) DESC) AS rank
  FROM chunks, plainto_tsquery('english', $2) query
  WHERE tenant_id = $3 AND tsv @@ query
  ORDER BY ts_rank_cd(tsv, query) DESC LIMIT 40
)
SELECT id, SUM(1.0 / (60 + rank)) AS score
FROM (SELECT * FROM semantic UNION ALL SELECT * FROM keyword) merged
GROUP BY id ORDER BY score DESC LIMIT 12;

That single query is most of a retrieval engine. The tenant_id filter is not optional — more on that below.

Reranking#

Retrieval returns candidates that are probably relevant. A reranker — a smaller model that scores each (question, chunk) pair — reorders them by how well they actually answer the question. Reranking the top 30 down to the best 5–8 measurably improves answer quality and reduces the context the generator has to read, which lowers cost and latency at the same time.

Permissions: the step everyone skips#

If a user cannot open a document in the source system, the model must not read it on their behalf. This is the single most important engineering rule in business RAG and the easiest to violate: index everything into one store, query it for everyone, and you have built an internal data leak with a friendly interface.

The approach that works:

• Store the source system's permission information (owner, groups, visibility) as metadata on every chunk at ingestion time.
• Resolve the current user's groups at query time.
• Filter in the retrieval query, not after generation. Filtering after the fact means the model may have already seen and summarised something the user should not see.
• Re-sync permissions when they change in the source, not just content.

Multi-tenant products need the same discipline one level up: tenant isolation on every query, enforced in the data layer, tested.

Citations and refusal#

Two behaviours separate trustworthy answers from plausible ones:

Citations. Every answer shows which chunks it drew from, with links back to the source document. Users check, and the checking is what builds trust. It also makes failures diagnosable: a wrong answer with a citation tells you whether retrieval or generation was at fault.

Refusal. If the retrieved context does not contain an answer, the model should say so rather than improvise. Instruct it explicitly, and test it with questions you know are not covered.

Evaluating a RAG system#

You cannot improve what you do not measure. Build an evaluation set early:

• 30–100 real questions from the people who will use the system.
• For each: the correct answer and the document(s) that contain it.
• Measure retrieval separately from generation: was the right chunk in the top results? Did the answer match?

Retrieval metrics (recall at k, mean reciprocal rank) tell you whether to work on chunking, embeddings or search. Generation metrics tell you whether to work on prompts or model choice. Conflating them means you tune the wrong thing.

Run the evaluation on every change. It takes minutes and prevents the slow decay that happens when prompts are edited by feel.

Keeping it current#

A knowledge base that is a month stale is worse than none, because people stop trusting it silently. Design for freshness:

• Incremental re-indexing on change events or short polling intervals.
• Version chunks so updated documents replace old vectors atomically.
• Surface the source's last-modified date in the answer, so users can judge for themselves.

A reference architecture#

For a mid-sized business knowledge base, this is the shape I typically build:

• Connectors as scheduled jobs (Laravel queues or Node workers) writing to an ingestion table.
• Chunker and embedder as a worker consuming that table, writing chunks with metadata and permissions into Postgres + pgvector.
• Retrieval service exposing one endpoint: question + user → ranked, permission-filtered chunks, using the hybrid query above plus a reranker.
• Generation service that assembles the prompt, calls the model, parses citations and returns a structured answer.
• Interface that shows the answer, the sources and a "was this correct?" control feeding the evaluation set.
• Observability on retrieval hit rate, refusal rate, latency and spend per query.

None of this is exotic. It is ordinary backend engineering applied carefully around a model — which is exactly why it works.

When RAG is the wrong tool#

RAG answers questions over content. It is not the right pattern when the job is to do something (that needs tools and agents), when the answers are arithmetic over structured data (query the database directly), or when the corpus is tiny and fits in the prompt (just include it). Choosing the right pattern is part of the engineering.

If you have documents that people keep asking the same questions about, a well-built RAG system is one of the highest-return AI investments a business can make — provided it is built as a system, not a demo.