Work in progress · Research tooling
UT Analysis Pipeline
A 26-agent orchestration system for research analysis, with the quality gates built in first.
An open-source multi-agent system, actively in progress. Sequential and parallel agents, typed data contracts, and machine-checkable quality gates, wired together with a CLI and an MCP server. The interesting part isn’t that AI does the analysis. It’s that every claim it makes has to survive verification before it reaches a human.
What this build demonstrates
The problem
Why a single model call doesn’t scale to this
A single prompt, “analyze these transcripts and give me a report,” produces plausible output. But you can’t inspect it, challenge one specific claim inside it, tell where it’s uncertain, or rerun just the part that failed. One opaque call in, one opaque blob out.
For research analysis that’s disqualifying. A finding that can’t be traced to a participant, a task and a verbatim quote isn’t a finding, it’s a claim. And a system that can’t tell you which of its claims are weak will quietly hand you the weak ones alongside the strong ones.
Decompose everything, hide nothing. Every step is its own agent, with a typed input, a typed output, and a defined place for a human or an evaluator to intervene.
26 agents is a lot of code. The payoff is that every claim the system makes can be traced back through the exact sequence of agents that produced it, and every claim has passed a check that can fail.
In plain terms
What the pipeline actually does
Transcripts and observer notes go in. Evidence-grounded findings, themes, contradictions, research gaps, a plan for the next study, and a stakeholder report come out. In between, twelve stages, each one inspectable.
- Ingest. Transcripts and notes are parsed into structured turns.
- Extract. Findings are pulled out, each carrying a verbatim quote, a participant, a task and a confidence score.
- Evaluate. Five checks per finding. Failures loop back for correction, up to three times.
- Rate severity. Verified findings only. Four weighted criteria on Nielsen’s 0 to 4 scale.
- Map to heuristics. Classify each issue against usability heuristics.
- Fan out. Six specialists read the same findings concurrently through six lenses.
- Consolidate. Merge overlapping themes without flattening what each lens saw.
- Find contradictions, then reconcile. Disagreement is segmented and explained, never averaged away.
- Critique. An independent critic agent challenges the evidence before anything is reported.
- Find gaps, plan the next study. Unanswered questions become a concrete follow-up.
- Score and flag. Nine-dimension rubric. Failures auto-create a human checkpoint.
- Report. Audience-tailored output, plus a run summary recording every stage and eval result.
The vocabulary, briefly
Agent is one worker doing one job in the line, with a defined input and output. Typed model is a data shape with required fields, so a finding without a quote and a participant isn’t low quality, it’s an error the system refuses to create. MCP is a standard interface that lets an AI assistant call these steps directly rather than someone running commands by hand.
Architecture
Five layers, CLI down to persisted state
Each layer only talks to the one directly below it, which keeps orchestration logic, agent implementations, and quality checks independently testable.
Why 26 agents
Not for the count. They fall into four groups, and the shape is the argument for the decomposition.
Quality gates · 01
How a finding gets verified
Two evaluation paths run over the pipeline’s output. The first checks whether a finding is true to the transcript. The second checks whether an artifact is good enough to act on. They fail for different reasons and they run independently.
Path one: five-check verification
The ut-evaluator agent takes each extracted finding and runs five checks against the source material. A finding has to pass all five.
CHECK 01
Verbatim match Does the quote actually appear in the transcript, word for word?CHECK 02
Meaning accuracy Does the finding represent what the participant meant, not a plausible-sounding drift from it?CHECK 03
Task attribution Is the finding tied to the task it actually occurred in?CHECK 04
No researcher interpretation Is this what happened, or is it already a conclusion about what it means? Interpretation belongs downstream, not in the extraction.CHECK 05
Justified severity Is the claimed severity supported by the evidence attached to it?The revelation loop
A failed check doesn’t discard the finding and doesn’t hard-fail the run. The evaluator generates specific correction instructions and sends the finding back for re-extraction, up to three iterations. Anything still failing after that is surfaced rather than silently dropped.
Path two: the nine-dimension quality rubric
Separately, ut-eval-rubric scores each artifact across nine dimensions, among them evidence, specificity, actionability and provenance. Each dimension resolves to pass, warning, or fail.
A failure doesn’t wait for someone to notice it. It automatically creates a human-review checkpoint, which is the mechanism that stops a weak artifact moving downstream while looking exactly like a strong one.
Why two paths instead of one score
Verification and quality are different questions. A finding can be perfectly faithful to the transcript and still be useless, because it’s vague, or nobody could act on it. And a beautifully actionable recommendation can be built on a quote that was never said. Collapsing both into one number would hide which failure you had.
Results persist to project_data/data/evals as <run_id>_eval_results.json, and run state updates so downstream steps know what passed.
uv run ut-analysis run-evals --project-dir ./project_data --run-id <id>Interfaces
Two ways to drive it
Every step is separately runnable, which matters more than it sounds. When evaluation flags a problem you rerun that stage, not the whole analysis.
14 CLI commands, in workflow order
init-project set up a new analysis project
ingest-transcript load a session transcript
ingest-notes load observer notes
status where is this project in the pipeline
extract pull findings out of the raw material
evaluate run the 5-check verification
orchestrate-themes route findings to the right specialist lanes
run-theme-agents run the six specialists in parallel
consolidate-themes merge their overlapping output
find-research-gaps identify what we still don't know
plan-next-study turn those gaps into a follow-up plan
run-evals score artifacts on the 9-dimension rubric
list-review-checkpoints show everything flagged for a human
run-summary full record of what ran and what happened24 MCP tools
MCP is a standard interface that lets an AI assistant call the pipeline directly, using the same typed contracts a person would. So an assistant can pull a transcript, run extraction, and report what passed verification, without anyone hand-running commands.
The tools cover loading and saving artifacts, inspecting pipeline status, running analysis steps, and pulling from an external research repository. The one that matters most for research integrity is validate_provenance, which checks that a claim traces back to real source material.
Quality gates · 02
How severity gets scored
Only findings that cleared evaluation reach the severity rater. Everything below runs on verified input, which is the whole reason the two stages are separate.
The scoring model is the one I’ve used in practice for years, written down as code so it’s inspectable and arguable rather than a matter of my judgment on the day. Four criteria, each scored 0 to 1, combined as a weighted sum.
| Criterion | What it asks | Default weight |
|---|---|---|
| Impact | How badly does this affect the person when it happens? | 0.4 |
| Frequency | How often does it happen? | 0.3 |
| Persistence | Does it keep happening, or do people learn around it? | 0.2 |
| Scope | How much of the product or population does it touch? | 0.1 |
Impact is weighted above frequency on purpose
A rare catastrophe outranks a common annoyance. Frequency-led prioritization systematically buries the problems that only show up on the worst day, and those are usually the ones that decide whether someone keeps using the product at all. Weights are configurable per project, but that ordering is the default for a reason.
Mapping to Nielsen’s 0 to 4 scale
The weighted score maps onto Nielsen’s standard severity levels, so ratings are comparable against a scale people already know rather than a private one.
Every rating carries its argument
A number on its own invites people to either accept it or ignore it. Each SeverityRating object carries the material needed to contest it:
- Textual rationale for why this score, in plain language.
- Criteria descriptions showing how each of the four was assessed.
- Sample evidence: the quote, the task, the participant it came from.
Confidence is scored separately
Severity says how bad it is. Confidence says how sure the system is that it read the situation correctly. Those are different, and conflating them is how a shaky finding gets treated like a solid one.
Confidence starts at a base of 0.5 and rises with quote length, the presence of a task ID, the presence of a participant ID, and the confidence of the original extracted finding, capped at 1.0. Ratings are written to data/severity and feed heuristic mapping, the recommender and the final report.
The honest limitation
The weights and the confidence formula are hand-tuned. They encode my judgment about what matters, which is defensible but not validated. Calibrating them against expert-labeled data is the next real piece of work on this, and until that’s done the right way to read a severity score is as a structured argument rather than a measurement.
Design patterns
Ten patterns, one pipeline
Each chosen to solve a concrete problem rather than for novelty.
- Sequential pipeline decomposition. The controller advances through phases only when each phase’s prerequisites are met.
- Parallel specialist fan-out. Six thematic agents analyze the same findings concurrently instead of one agent doing it serially.
- Controller / orchestrator. A theme orchestrator reads findings once and routes subsets to the specialists that should see them.
- Structured JSON handoffs. Every agent produces typed JSON plus a Markdown rendering. Nothing passes between steps as free-form prose.
- Evaluation and rubric gates. A 9-dimension quality rubric runs inline and can pass, warn, or fail each artifact.
- Human-in-the-loop checkpoints. Low confidence, weak evidence, or unresolved contradictions auto-create a review checkpoint instead of shipping silently.
- Contradiction reconciliation. Disagreement between sources is segmented and explained, never averaged away.
- Revelation loops. When the evaluator fails a finding, it triggers re-extraction rather than discarding or hard-failing.
- Gap-aware planning. Unanswered questions are extracted as a first-class artifact and converted into concrete next-study plans.
- Full observability. A run-summary artifact records every stage, every agent invoked, and every eval result for a given run.
Parallel fan-out
Six agents, one orchestrator, one merge
The theme orchestrator routes every verified finding to whichever of six lanes are relevant. The specialists run concurrently over the same data, and a consolidator merges overlapping themes without collapsing the nuance each lens surfaced.
Key decisions
Choices that shaped the system
- Keyword-based lane routing, not an LLM call. Deterministic, fast, and unit-testable. A routing decision you can assert on in a test suite instead of eyeballing. Swapping in a classifier later is a defined upgrade, not a rewrite.
- Findings can live in multiple lanes. A single frustration can be both a pain point and a workflow breakdown, so routing allows overlap rather than forcing one category.
- Dual JSON and Markdown output on every agent. JSON for downstream agents, Markdown for a human to open and read without tooling.
- Confidence bounds on every artifact. Which is what makes the rubric’s overclaiming check possible in the first place.
- Failures create checkpoints automatically. Quality problems don’t wait for a human to notice them.
Why build this
What it changed about how I work
Two things, and neither is speed.
It forced me to write down my own judgment. The severity weights, the five checks, the rubric dimensions are all things I’d been doing implicitly for years. Encoding them meant making them explicit enough to be wrong in public, which is a useful discipline and occasionally an uncomfortable one.
It clarified where I don’t want AI anywhere near the work. First-pass coding of a large transcript volume, verification against source, structural consistency: those are genuinely better automated. Deciding what matters, reading the thing a participant almost said, judging whether a finding is worth a roadmap fight: those aren’t. The pipeline is deliberately built so a human is required at exactly those points, not optional.
What’s next
Where the architecture goes from here
- LLM-based lane routing to replace keyword matching with semantic classification.
- Embedding-based theme clustering in place of word-overlap similarity, which currently misses semantically identical findings phrased differently.
- Full MCP tool coverage so all 26 agents are independently callable, not just the original ten.
- Confidence calibration trained against expert-labeled data rather than the current hand-tuned formula.
About this project. UT Analysis Pipeline is open source, built in Python 3.11 with Pydantic, Click, and the Model Context Protocol. Browse the source on GitHub.