I enjoy the first working demo of an AI feature. A clean prompt, a handful of examples and a good model can make an idea feel possible in an afternoon.

That moment matters, but it can also give a team false confidence. A demo answers one question: can the idea work? A production AI system has to answer harder ones. Does it work repeatedly? Can we understand why it failed? What happens when the model is uncertain, a tool is unavailable or a user ignores the happy path?

That gap is where most of the engineering lives.

Start with the decision, not the model

Before choosing a model or agent framework, define the decision the system is helping someone make. This changes the architecture immediately.

A useful specification includes:

  • the input context the system is allowed to use;
  • the action or recommendation it may produce;
  • the cost of a false positive and a false negative;
  • the point at which a human must take over;
  • the evidence needed to evaluate the outcome.

When these are explicit, model selection becomes one design choice inside a larger product system—not the product itself.

The right unit of AI product quality is not “good output.” It is a useful outcome produced within known boundaries.

Build evaluation into the workflow

Teams often treat evaluation as a final QA step. I have found it more useful as part of the development loop from the beginning.

Create a small, representative dataset before polishing the interface. Include straightforward requests, ambiguous cases, incomplete context, adversarial inputs and examples where the correct behaviour is to decline or escalate.

Then evaluate at three layers:

  1. Task quality: Did the system produce the correct or useful result?
  2. System behaviour: Did it choose the right tools, sources and recovery path?
  3. Product outcome: Did the result help the user complete the real task?

Automatic scores are useful for fast iteration, but they should be calibrated against human review. An LLM-as-judge can detect trends; it should not quietly become the only definition of quality.

Design the failure path first

Reliable AI products do not pretend uncertainty does not exist. They make uncertainty visible and useful.

For an agentic workflow, that means explicitly modelling states such as needs_context, tool_failed, requires_review and safe_to_continue. It also means preserving enough state to resume a conversation without asking the user to start again. I explore that design in more detail in Stateful agents for real conversations.

This is less theatrical than a fully autonomous agent—and much more valuable in production.

Observe the whole system

Token usage and latency matter, but they are only infrastructure signals. Product observability should also tell us:

  • which requests are being escalated;
  • where users edit or reject an answer;
  • which knowledge sources are repeatedly missing;
  • which tool calls fail and whether recovery succeeds;
  • how quality changes after a prompt, model or retrieval update.

This is the feedback I want from production. It shows the team where people are struggling and gives the next engineering decision something firmer than instinct to work from.

What I keep coming back to

The model will change. Frameworks will change faster. A durable AI system is built around clear decisions, measurable behaviour, recoverable workflows and informed human control.

That is the work after the demo. It is where an AI feature finally becomes a product.