Work in progress · Multi-Agent Systems
I'm currently building a 26-agent orchestration pipeline for research analysis.
UT Analysis Pipeline is an open-source multi-agent system, still actively in progress — sequential and parallel agents, typed data contracts, and machine-checkable quality gates, wired together with a CLI and an MCP server. Here's how it's put together so far.
What this build demonstrates
The engineering problem
Why a single model call doesn't scale to this
A single prompt — "analyze these transcripts and give me a report" — can produce plausible output. But you can't inspect it, challenge one specific claim inside it, tell where it's uncertain, or rerun just the part that failed. It's one opaque call in, one opaque blob out.
I wanted to see what it takes to build the alternative: a system where every step in a complex analysis workflow is its own agent, with a typed input, a typed output, and a defined place for a human or an evaluator to intervene. That constraint — decompose everything, hide nothing — is what shaped the whole architecture.
26 agents is a lot of code. The payoff is that every claim the system makes can be traced back through the exact sequence of agents that produced it.
Architecture
Five layers, CLI down to persisted state
The system is organized as five layers stacked on top of a JSON/Markdown persistence store. Each layer only talks to the one directly below it, which keeps the orchestration logic, the agent implementations, and the quality checks independently testable.
Design patterns
Ten patterns, one pipeline
The architecture leans on a specific set of agentic design patterns, each chosen to solve a concrete problem rather than for novelty.
- Sequential pipeline decomposition — the controller advances through phases only when each phase's prerequisites are met.
- Parallel specialist fan-out — six thematic agents analyze the same findings concurrently instead of one agent doing it serially.
- Controller / orchestrator pattern — a theme orchestrator reads findings once and routes subsets to the specialists that should see them.
- Structured JSON handoffs — every agent produces typed JSON plus a Markdown rendering; nothing is passed as free-form prose between steps.
- Evaluation / rubric gates — a 9-dimension quality rubric runs inline and can pass, warn, or fail each artifact.
- Human-in-the-loop checkpoints — low confidence, weak evidence, or unresolved contradictions auto-create a review checkpoint instead of shipping silently.
- Contradiction reconciliation — disagreement between sources is segmented and explained, never averaged away.
- Revelation loops — when the evaluator fails a finding, it triggers re-extraction rather than discarding or hard-failing.
- Gap-aware planning — unanswered questions are extracted as a first-class artifact and converted into concrete next-study plans.
- Full observability — a run-summary artifact records every stage, every agent invoked, and every eval result for a given run.
Ten agents, one linear pass
Ingestor, extractor, evaluator, severity rater, heuristic mapper, synthesizer, contradiction detector, reconciler, critic, and reporter form the original linear backbone — each one a typed step with a single responsibility.
Structured handoffs prevent drift
Because each agent consumes typed JSON instead of prose, finding IDs, participant IDs, and quotes travel intact through all ten steps — there's no summarization loss between hops.
Six agents, one orchestrator, one merge
The theme orchestrator routes every verified finding to whichever of six lanes are relevant, the specialists run concurrently over the same data, and a consolidator merges overlapping themes across lanes without collapsing the nuance each lens surfaced.
Concurrency, not just speed
The point of running six agents in parallel isn't throughput — it's that a "trust issue" reads differently to the trust analyst than to the workflow analyst. Fan-out surfaces interpretations a single pass would flatten into one.
Key decisions
Choices that shaped the system
- Keyword-based lane routing, not an LLM call. Deterministic, fast, and unit-testable — a routing decision you can assert on in a test suite instead of eyeballing. Swapping in an LLM classifier is a defined future upgrade, not a rewrite.
- Findings can live in multiple lanes. A single frustration can be both a pain point and a workflow breakdown, so the routing logic allows overlap rather than forcing a single category.
- Dual JSON + Markdown output on every agent. JSON for machine consumption and downstream agents; Markdown for a human to open and read without tooling.
- Confidence bounds attached to every artifact. Every output carries an explicit confidence score, which is what makes the eval rubric's overclaiming check possible in the first place.
- Eval rubric auto-creates review checkpoints. Quality problems don't wait for a human to notice them — a failed dimension creates a checkpoint automatically.
What I'd build next
Where the architecture goes from here
- LLM-based lane routing to replace keyword matching with semantic classification.
- Embedding-based theme clustering in place of word-overlap similarity, which currently misses semantically identical findings phrased differently.
- Full MCP tool coverage so every one of the 26 agents is independently callable as an MCP tool, not just the original ten.
- Confidence calibration trained against expert-labeled data rather than the current hand-tuned formula.
About this project. UT Analysis Pipeline is open source, built in Python 3.11 with Pydantic, Click, and the Model Context Protocol (MCP). Browse the full source on GitHub →