How to Build an AI-Powered Business Application

Most "AI-powered" business applications fail for a boring reason: the model was treated as the product. A demo answers questions impressively in a notebook, but it has no reliable way to reach the company's actual data, no way to take actions, no evaluation, and no plan for running it. The engineering around the model was never built.

What follows is the architecture I use when a business wants an application that uses a language model to do real work, and the checklist I run before any of it goes live.

Start with the job, not the model

Before choosing a model or a vector database, write the job down in one sentence with a verb in it:

• Triage inbound support emails and draft a first response.
• Answer staff questions about internal policies with citations.
• Extract line items from supplier invoices into the finance system.
• Turn a sales call transcript into structured CRM updates.

Each of these is a different system. The first needs classification plus generation, with a person approving the draft. The second is retrieval-heavy and needs source attribution. The third is document intelligence with strict validation. The fourth is structured output feeding an API.

If the job can't be written down this way, the project isn't ready to build. "Add AI to the app" is not a job.

The seven layers of an AI application

Once the job is clear, the application breaks down into layers. Each one is engineering work, and a project can fail at any of them.

1. The model

Pick models per task, not per project: a cheap, fast model for classification and routing, a stronger one for drafting or reasoning over long context, and an embedding model for retrieval. Put a thin provider abstraction between your code and the vendor SDKs, so you can swap a model without touching business logic. You will want to within the first year.

2. The application layer

Most of the work lives here.

Prompt assembly should be deterministic. System instructions, retrieved context, tool definitions and the user's input get built from typed inputs in one place, not concatenated as strings across the codebase. If a result feeds code, it has to be schema-constrained (JSON with a defined shape) and validated before use; free text is for humans.

Then there is evaluation: a set of real examples with expected outcomes, run whenever the prompt, the model or the retrieval changes. Without it, you're guessing whether a change made things better. And the layer needs cost and latency controls, meaning token budgets, caching of repeated context, timeouts and fallbacks.

3. Business data

The model knows nothing about your business, and retrieval is how it learns at request time. You need an ingestion pipeline that moves your sources (database records, documents, tickets, wikis) into a searchable index, with chunking that respects document structure, embeddings, and a mix of semantic and keyword search.

Retrieval also needs access control. If a user can't see a document in the source system, the model must not see it on their behalf.

There's more on this in How RAG Systems Work for Business Knowledge Bases.

4. Tools

Tools are typed functions the model can call: look up an order, create a ticket, fetch a customer's history, run a calculation. A good tool is narrow, with one clear purpose, a small typed input and a predictable output. It is safe by default, so reads run freely while writes need confirmation or run in a sandboxed flow. And every call is logged with its inputs, its outputs and the user it ran for.

A tool-using system is mostly as good as its tool definitions.

5. Workflows

An AI step almost never stands alone. It sits inside a process: a queue picks up the email, the classifier runs, a draft is generated, a person approves it, the reply goes out, the CRM is updated. That is ordinary workflow engineering (background jobs, retries, idempotency, state) with an AI component inside it.

Put the human checkpoint where the risk is. Reading and summarising need none. Sending, spending and deleting do.

6. User interface

The interface should make the system's behaviour visible. Show the sources behind retrieved answers. Show the structured data that was extracted, and let people correct it. Make approving or rejecting fast, because that step is what keeps the automation trustworthy.

A chat box is one possible interface. For a defined business task it is rarely the best one.

7. Deployment

Deployment is the same as for any application (containers, CI/CD, secrets management, separate environments) plus monitoring specific to AI. Track latency per step, token spend per feature, retrieval hit rates and validation failure rates, and have a person review a sample of outputs each week.

A worked example: support triage

Take the first job from the list, triaging inbound support emails and drafting a response. Here is how the layers map onto a concrete design:

Inbound email
  → queue job: classify(intent, urgency, product)      [fast model, JSON schema]
  → if urgency = critical: page on-call, stop
  → retrieve: similar resolved tickets + product docs   [hybrid search, tenant-scoped]
  → draft reply with citations                          [stronger model]
  → create ticket in helpdesk with draft attached       [tool call]
  → agent reviews, edits, sends                         [human checkpoint]
  → record: classification, draft, final reply          [evaluation data]

The model runs twice, for different purposes, and both outputs are constrained. The last step matters as much as the others: the corrected replies become the evaluation set that tells you whether the drafts are improving.

The classification step in code

The shape matters more than the exact SDK. In a Laravel or Node.js application it looks like this:

const TriageSchema = z.object({
  intent: z.enum(["billing", "bug", "how-to", "account", "other"]),
  urgency: z.enum(["low", "normal", "high", "critical"]),
  product: z.string().max(60),
  summary: z.string().max(200),
});
 
export async function triage(email: InboundEmail) {
  const result = await llm.generateObject({
    model: models.fast,
    schema: TriageSchema,
    system: TRIAGE_INSTRUCTIONS,
    prompt: renderEmail(email),
  });
  return TriageSchema.parse(result); // never trust the model's JSON without parsing
}

Everything downstream depends on TriageSchema being enforced. If parsing fails, the job retries once with the error message included, then drops the email into a "needs human" bucket. At worst the system falls back to what a support team already does, and it never invents a category.

What has to be true before production

I use this list as a gate. If any item is false, the system is still a prototype.

1. The job is written down in one sentence, with a definition of a good result.
2. There is an evaluation set of at least a few dozen real examples, and a script that runs it.
3. Every output that feeds code is schema-validated, with defined behaviour on failure.
4. Retrieval respects the permissions of the source systems.
5. Writes and sends have a human checkpoint, or a documented reason they don't.
6. Every model call and tool call is logged with the user it ran for.
7. Token spend is metered per feature, and someone will see the number.
8. A person reviews a sample of outputs each week, and the review feeds the evaluation set.
9. The model can be swapped without changing business logic.
10. When the model is down or slow, the process falls back to the manual path instead of blocking.

Common mistakes

The most common is starting with a chatbot. A general chat interface is the hardest thing to make reliable and the least likely to fit an existing workflow, so start with the narrow job.

Close behind is skipping retrieval permissions. Indexing every document into one shared vector store and querying it for every user is the fastest way to leak data internally.

Without an evaluation set, every prompt change is a coin flip and nobody can say whether the system got better. Trusting the model's JSON is another: models produce plausible-looking structures that are subtly wrong, so parse, validate and handle the failure. And cost gets ignored until the invoice arrives. A feature that costs a few pence per run is fine until it runs on every page view.

Where to begin

If you're building this for the first time, sequence it like this. Define the job. Build the workflow with a stubbed AI step that a person performs. Replace the stub with the model, behind validation. Then add retrieval, then tools.

The system is useful and shippable at every stage. And because people were doing the job inside the workflow before the model arrived, the evaluation data starts building up from the first day.