Ten Million Events, One Verdict: Building the AI UX Research Platform
Four sources, no verdict
Clicks, replays, heatmaps and support tickets each lived in their own tool, and each one produced a report that rarely got read to the end. An analyst opened four dashboards before the weekly meeting and still missed the pattern that had sat in the chat log since Monday. I built a platform that pulls those four sources into one searchable store. The team reads the conclusion first and reaches the evidence by link when it disagrees.
Three agents over one timeline
The spine is the session timeline. Three agents work along it in order: the first transcribes calls and chat, the second pulls intents and friction points out of that text, the third groups the intents into Jobs-to-Be-Done statements. Each stage writes its result back against the timestamp it read from, so the last stage quotes words that were actually said. I keep the stage order and the retry rules here the same as the ones I set out in AI Agent Development: From Prototype to Production.
Trust ships as a link
An insight card holds three things: the replay timestamp, the raw quote and the model's confidence score. Clicking the card moves the player to that moment. A confidence number nobody can check gets the tab closed, so the check sits one click away. The habit of showing the work behind every answer is the one I describe in Glass-Box AI: Showing Model Confidence So People Trust It.
Transcripts and replays carry customer text, so retention windows, export rules and the deletion path had to be part of the build from the start. Those rules come from Why Your AI Agent Needs a Compliance Layer.
What moved
| Measure | Before | After |
|---|---|---|
| Events in the search index | 0 | 10,000,000 |
| Insight precision, scored by a judge model | never measured | 0.89 |
| One research cycle | 21 days | 3 days |
The HNSW index lives in the same Neon Postgres that stores the sessions, so the search stayed inside one database and no second vector service came with it. The 0.89 comes from an LLM-as-judge run over sampled insights: a second model re-reads the quote and marks whether the claim holds. A batch the judge marks as unsupported goes back through the extraction stage with a stricter prompt. I cut the interface around acting rather than reading, and the screens that follow from that choice are in Designing Interfaces That Act: The Agentic UI Guide.
What I check before building something like this
Start from the decision the report changes. If a team cannot name one, the platform becomes a fifth dashboard and the four tools stay open anyway. Next, decide what a model may conclude on its own. Here every claim links straight back to the replay that produced it, and that boundary sets how strict the prompts have to be. The last check costs money: ten million events in a single index make vectors inside Postgres the cheaper answer, and that answer shaped the whole schema.
The description and the tag list sit on the project page: AI-Powered UX Research Platform.