AI Agent Development: From Prototype to Production

In November 2024 the owner of a freight brokerage in Gdansk asked me to move his dispatch helper from demo to daily use. The demo answered twenty sample questions about loads, tariffs, and docks. It ran against forty clean documents. It impressed the room. I connected the same helper to his live document store with eleven hundred files, gave forty dispatchers access, and watched it fail four times before lunch. It quoted a tariff from a contract that had expired in March. It invented a dock number for a warehouse with six docks. It took twenty-two seconds to answer one question about a delayed load. It burned nine cents per call at his volume. The owner paused the rollout that afternoon. Every production agent I ship now walks the road below. Six stages carry it from that failed morning to steady daily use. I run them in order. Each stage lists the numbers I measured on client builds.

Why demos die in production

Demos live in a greenhouse. Twenty sample questions, forty fresh documents, one user, zero cost meter. Production opens the door. Two thousand real questions arrive with typos, half sentences, and photos of paperwork. Eleven hundred documents include expired contracts, duplicate tariffs, and three versions of the same price list. Forty people ask at once on Monday morning. Every call lands on the invoice.

Three killers end most demos in the first week. Stale retrieval feeds the model old pages, and the model quotes them with full confidence. Missing evals hide the damage, because nobody reruns a fixed question set after each change, so drift spreads and nobody sees it. Unmeasured cost and latency shock the owner at month end, when ten thousand calls at nine cents each total nine hundred dollars for answers nobody trusts.

Order of repair matters. I stabilize retrieval first, because no wording fix survives wrong source pages. I add the eval set second, because I need a scoreboard before I change anything else. I cap cost and latency third, because budgets shape model choice. I cover the compliance layer fourth, because customer names and phone numbers deserve their own review. Then I ship. Skip the order and you tune sentences against broken search.

RAG pipeline that holds up

I store documents in Postgres with pgvector on Neon. One table holds chunks: chunk_id uuid, tenant_id text, source_url text, updated_at timestamptz, content text, embedding vector(1536). An HNSW index on the embedding column serves nearest-neighbor search with m 16 and ef_construction 64. A btree index on tenant_id plus updated_at filters each query to the right customer and fresh files first.

I cut text into chunks of four hundred to six hundred tokens with eighty tokens of overlap. Each chunk keeps its source page and date. Shorter chunks lost context in my tests: recall on a fixed question set sat at sixty-one percent. Longer chunks diluted the match: the model quoted whole pages instead of single lines. The four-to-six-hundred window lifted recall to eighty-eight percent on the same set.

Each query filters by tenant_id, searches the five nearest chunks, and drops anything the owner marked expired. The answer prints its sources underneath: document name, page, date. Dispatchers click through and trust what they verify. I rerun the recall check weekly. One writer owns each document, and stale copies leave the store the same day the owner replaces them.

Evals: the 200-sample habit

I keep two hundred fixed questions with written answers and source pages. Eighty cover routine work: load numbers, tariffs, dock hours. Sixty carry typos, missing streets, and half-remembered names. Forty demand a refusal: the bot states it cannot find the document and hands the chat to a dispatcher. Twenty plant traps: expired tariffs, conflicting price lists, two warehouses sharing one street name.

I rerun the full set before every release and log date, model version, and score. A drop of more than three points blocks the release, and that rule holds. The Gdansk build taught me why: one March tariff file expired, the set caught eleven wrong quotes in a single run, and the release waited one day while I replaced the file. Without the set those eleven quotes would have reached forty carriers.

Every production miss joins the set the same day it surfaces. A driver phones about a wrong window, I add the exact question with the right answer and source. The set grows past two hundred over time. The name stays. Two hundred marks the minimum, and the minimum holds the line.

Cost and latency budgets

I price every call before I choose a model. One Gdansk answer pulled five chunks of about three hundred fifty tokens each, added three hundred tokens of instructions and history, and wrote three hundred fifty tokens in reply. Input totaled near two thousand one hundred tokens. Output totaled three hundred fifty.

At small-model rates of fifteen cents per million input tokens and sixty cents per million output tokens, that call costs about five ten-thousandths of a dollar. Two thousand one hundred input tokens at $0.15 per million come to $0.00032. Three hundred fifty output tokens at $0.60 per million come to $0.00021. Embeddings add $0.00002. Total near $0.0005. Ten thousand calls per month cost five dollars. The same shape on a flagship model at $5 per million input and $15 per million output costs near $0.016 per call, or one hundred sixty dollars per month for the same ten thousand calls.

So I route routine reads to the small model and reserve the flagship for damage-claim letters where one wrong clause costs more than a month of calls. I set two latency lines: first visible reply inside two seconds through streaming status, full answer inside nine seconds. I read p50 and p95 weekly from the call log. When p95 crosses the line two weeks in a row, I trim chunks before I touch the model.

The compliance layer

Agents that read customer names, phone numbers, and contracts need a small data layer from day one: I log every tool call with an approval reference, scrub identifiers before they reach logs, and stamp each row with a deletion date so old transcripts leave the store on schedule. I described the exact tables, the scrub setup, and the nightly delete job in Why Your AI Agent Needs a Compliance Layer, and I add that layer to every production build before it serves real traffic.

Shipping checklist

I run this list the morning before any launch.

1. Retrieval filters by tenant_id and drops expired files.
2. Every answer prints source name, page, and date.
3. The 200-question set reruns green with no drop over three points.
4. Per-call cost sits on paper with monthly volume math attached.
5. p95 answers inside nine seconds two weeks running.
6. Outbound messages wait for an approval reference.
7. Every stored row carries a deletion date with a nightly delete job behind it.
8. One flag rolls the release back in minutes.

If your helper still lives in demo form, request an AI audit and I will score your build against each stage above.