Prioris

The problem

Feedback shows up everywhere a product team looks — support tickets, sales calls, reviews, a survey someone ran last quarter. It's useful, but it's scattered, and pulling it together into “here's what we should actually build” is mostly manual. Usually it ends up in one person's head, or a spreadsheet nobody opens twice.

Prioris handles that part. I built it end to end — the pipeline, the evals, and the MCP server. It reads through the raw feedback, groups it into themes, and drafts a first-pass PRD, so you're editing something instead of starting from a blank page.

What it does

• Clusters feedback into insights. You put in the raw items; it groups them into themes and keeps the underlying quotes attached to each one.
• Answers questions about the feedback. Ask something like “what are enterprise users asking for?” and you get an answer grounded in the actual source items.
• Drafts PRDs. Pick a cluster and it writes a structured first draft you can edit down.
• Works inside your editor. An MCP server exposes the clusters to coding agents, so you can pull up customer voice while writing a spec without switching tabs.

Stack

It's a Next.js 16 app (App Router, React 19, Tailwind 4) on Vercel, with API routes talking to Postgres through Prisma. Supabase handles auth and file storage. OpenAI does the extraction, embeddings, and labels; Claude runs as the judge in my offline evals.

The pipeline

Here's the path a single piece of feedback takes:

1. CSV Uploadraw tickets, reviews, notes, survey rows
2. Parsenormalize each row into a feedback item
3. Atomizegpt-4o-mini splits each item into atomic claims
4. Embedtext-embedding-3-small
5. Clustergreedy cosine, threshold 0.60; incremental on later runs
6. Label + Mergegpt-4o-mini labels each cluster; near-duplicates merge at 0.88
7. PersistPostgres: Record, Complaint, Cluster, Evidence
8. Serveclusters returned as JSON, with evidence attached

The agent

On top of the clusters there's a workspace chat with two modes.

• Ask mode is read-only and always runs on gpt-4o-mini — quick questions grounded in the feedback.
• Agent mode gets the full toolset and escalates to o3-mini when the work is heavier: long messages, several references, or certain keywords.

Either way the model decides which tool to call — search, Q&A, or PRD generation — through function calling. Search itself is in-memory cosine over the cluster labels, with a keyword filter for exact matches.

The MCP server

Three tools make the clusters available to any coding agent, so you can stay in your editor and still reference what customers actually said.

• prioris_get_clusters — labels and mention counts
• prioris_get_evidence — source quotes behind a cluster
• prioris_list_runs — past clustering runs

Evals

The outputs here are fuzzy, so “it looks right” isn't a real answer. I built evals for the two steps most likely to go wrong — extraction and clustering — and both run offline against gold sets I labeled by hand.

Extraction exact-match

4 of 17

Judge fail rate

~10%

Clustering F1

0.39 → 0.56

Extraction. The atomizer breaks each piece of feedback into separate claims. I check it two ways. First, exact-match against the gold set, which comes out low (4 of 17) and is meant to — good extractions reword things, so a string comparison misses them. It's just a floor. The metric I actually rely on is an LLM judge (Claude) scoring against a rubric; it returns pass/fail with a short critique, and right now about 10% of candidate claims fail. Before trusting the judge at all, I checked it against my own labels — on the calibration set it agreed with me on every case (TPR and TNR both 1.0, though that's only 17 cases). There's no point leaning on a judge you haven't checked against real labels.

Clustering. For clustering I use pairwise F1: related claims should land in the same group, unrelated ones shouldn't. The gold set is 45 atoms I sorted into 24 clusters by hand, taken from real Notion reviews. The test re-embeds those atoms, runs the actual clustering code, and scores every pair. The baseline came in at 0.39. Adding cluster descriptors pushed it to 0.56 — that was one specific change, and the eval is the only reason I know it helped instead of hurt.

No gate yet. For now I just compare each run's F1 to the last one by hand, following a short playbook. There's no CI check that fails the build when a score drops. That one's still on the list.

What I'm not claiming

• The gold sets are small — 17 extraction cases, 45 atoms across 24 clusters. Enough to make decisions with, not enough to claim much past that.
• 0.56 F1 on clustering is okay, not great. Plenty of room left.
• The judge matched my labels perfectly, but on a small set, so I read that as a good sign rather than proof.
• Everything runs offline and by hand. There's no production traffic behind these numbers, and no automated gate.