How to Build an AI-Powered Business Application

Most "AI-powered" business applications fail for a boring reason: the model was treated as the product. A demo that answers questions impressively in a notebook turns out to have no reliable way to reach the company's actual data, no way to take actions, no evaluation, and no operational story. The engineering around the model was never built.

This article is the architecture I use when a business wants an application that uses a language model to do real work — and the checklist I run before any of it goes live.

Start with the job, not the model#

Before choosing a model or a vector database, write down the job in one sentence with a verb in it:

• Triage inbound support emails and draft a first response.
• Answer staff questions about internal policies with citations.
• Extract line items from supplier invoices into the finance system.
• Turn a sales call transcript into structured CRM updates.

Each of these is a different system. The first needs classification plus generation with a human approval step. The second is retrieval-heavy and needs source attribution. The third is document intelligence with strict validation. The fourth is structured output feeding an API.

If the job can't be written down this way, the project isn't ready to be built. "Add AI to the app" is not a job.

The seven layers of an AI application#

Once the job is clear, the application decomposes into layers. Each is engineering work, and each is where projects succeed or fail.

1. The model#

Pick models per task, not per project. A cheap fast model for classification and routing; a stronger model for drafting or reasoning over long context; an embedding model for retrieval. Put a thin provider abstraction between your code and the vendor SDKs so a model can be swapped without touching business logic — you will want to do this within the first year.

2. The application layer#

This is where most of the work lives:

• Prompt assembly — system instructions, retrieved context, tool definitions and the user's input, built deterministically from typed inputs rather than string concatenation scattered through the codebase.
• Output validation — if the result feeds code, it must be schema-constrained (JSON with a defined shape) and validated before use. Free text is for humans.
• Evaluation — a set of real examples with expected outcomes that you run whenever the prompt, model or retrieval changes. Without this you are guessing whether a change made things better.
• Cost and latency controls — token budgets, caching of repeated context, timeouts and fallbacks.

3. Business data#

The model knows nothing about your business. Retrieval is how it learns at request time. That means:

• An ingestion pipeline from your sources (database records, documents, tickets, wikis) into a searchable index.
• Chunking that respects document structure, embeddings, and a hybrid of semantic and keyword search.
• Access control on retrieval. If a user cannot see a document in the source system, the model must not see it on their behalf.

I've written about this in more depth in How RAG Systems Work for Business Knowledge Bases.

4. Tools#

Tools are typed functions the model can call: look up an order, create a ticket, fetch a customer's history, run a calculation. Good tools are:

• Narrow — one clear purpose, a small typed input, a predictable output.
• Safe by default — reads are free, writes require confirmation or run in a sandboxed flow.
• Logged — every call recorded with inputs, outputs and the user it ran for.

The quality of a tool-using system is mostly the quality of its tool definitions.

5. Workflows#

An AI step almost never stands alone. It sits inside a process: a queue picks up the email, the classifier runs, a draft is generated, a human approves it, the reply is sent, the CRM is updated. That is ordinary workflow engineering — background jobs, retries, idempotency, state — with an AI component inside it.

Put the human checkpoint where the risk is. Reading and summarising: no checkpoint. Sending, spending or deleting: checkpoint.

6. User interface#

The interface should make the system's behaviour legible:

• Show sources for retrieved answers.
• Show the structured data that was extracted, and let people correct it.
• Make the approve/reject step fast, because that step is what keeps the automation trustworthy.

A chat box is one possible interface. It is rarely the best one for a defined business task.

7. Deployment#

Containers, CI/CD, secrets management, environment separation — the same as any application — plus monitoring specific to AI systems: latency per step, token spend per feature, retrieval hit rates, validation failure rates, and a sample of outputs reviewed by a person each week.

A worked example: support triage#

Take the first job from the list — triage inbound support emails and draft a response. Here is how the layers map onto a concrete design.

Inbound email
  → queue job: classify(intent, urgency, product)      [fast model, JSON schema]
  → if urgency = critical: page on-call, stop
  → retrieve: similar resolved tickets + product docs   [hybrid search, tenant-scoped]
  → draft reply with citations                          [stronger model]
  → create ticket in helpdesk with draft attached       [tool call]
  → agent reviews, edits, sends                         [human checkpoint]
  → record: classification, draft, final reply          [evaluation data]

Notice that the model runs twice, for different purposes, and both outputs are constrained. Notice also that the last step is not an afterthought — the corrected replies become the evaluation set that tells you whether the drafts are improving.

The classification step in code#

The shape matters more than the exact SDK. In a Laravel or Node.js application it looks like:

const TriageSchema = z.object({
  intent: z.enum(["billing", "bug", "how-to", "account", "other"]),
  urgency: z.enum(["low", "normal", "high", "critical"]),
  product: z.string().max(60),
  summary: z.string().max(200),
});
 
export async function triage(email: InboundEmail) {
  const result = await llm.generateObject({
    model: models.fast,
    schema: TriageSchema,
    system: TRIAGE_INSTRUCTIONS,
    prompt: renderEmail(email),
  });
  return TriageSchema.parse(result); // never trust the model's JSON without parsing
}

Everything downstream depends on TriageSchema being enforced. If parsing fails, the job retries once with the error message included, then falls back to a "needs human" bucket. The system degrades to what a support team already does — it never invents a category.

What has to be true before production#

I use this list as a gate. If any item is false, the system is a prototype.

1. The job is written down in one sentence, with a definition of a good result.
2. There is an evaluation set of at least a few dozen real examples, and a script that runs it.
3. Every output that feeds code is schema-validated, with a defined behaviour on failure.
4. Retrieval respects permissions from the source systems.
5. Writes and sends have a human checkpoint or a documented reason they don't.
6. Every model call and tool call is logged with the user it ran for.
7. Token spend is metered per feature and someone will see the number.
8. A person reviews a sample of outputs weekly, and that review feeds back into the evaluation set.
9. The model can be swapped without changing business logic.
10. Failure is graceful — when the model is down or slow, the process falls back to the manual path rather than blocking it.

Common mistakes#

Starting with a chatbot. A general chat interface is the hardest thing to make reliable and the least likely to fit an existing workflow. Start with the narrow job.

Skipping retrieval permissions. Indexing every document into one shared vector store and querying it for every user is the fastest way to leak data internally.

No evaluation set. Without one, every prompt change is a coin flip and nobody can say whether the system got better.

Treating the model's JSON as trustworthy. Models produce plausible-looking structures that are subtly wrong. Parse, validate, and handle failure.

Ignoring cost until the invoice. A feature that costs a few pence per run is fine until it runs on every page view.

Where to begin#

If you are building this for the first time, sequence it like this: define the job, build the workflow with a stubbed AI step (a human does it), then replace the stub with the model behind validation, then add retrieval, then add tools. At each stage the system is useful and shippable. That order also means the evaluation data starts accumulating from day one, because humans were doing the job in the loop before the model was.

That is the difference between an AI feature and an AI-powered application: the application was engineered around the job, and the model earned its place inside it.