LLM Council: the pattern, Karpathy's repo, and the hosted version

An LLM council is a setup where multiple frontier language models answer the same prompt in parallel, review each other's answers — anonymized, so no model knows who wrote what — and a chairman (or moderator) model synthesizes the reviewed answers into one final response. The pattern went mainstream in November 2025 when Andrej Karpathy open-sourced llm-council (github.com/karpathy/llm-council, ~23.3k GitHub stars as of July 2026): a reference implementation where models answer, rank each other's anonymized responses, and a Chairman model writes the final synthesis, all routed through OpenRouter. The repo is deliberately minimal — you clone it, supply your own API keys in a .env file, and get no persistence or memory between sessions. Council AI is an independent productization of that same architecture: parallel answers, anonymized cross-review, and moderator synthesis, plus a consensus score on every answer, persistent memory, and a hosted app (web + iOS) with zero setup, from $19.99/month. To be clear about the relationship: Karpathy has no affiliation with Council AI — the architecture he open-sourced is the architecture Council runs as a product. Below: how each works, an honest repo-vs-hosted comparison (the repo wins on free and hackable), and the benchmark evidence on whether councils actually beat single models.

What is an LLM council?

An LLM council replaces the single-chatbot loop with a three-stage pipeline:

1. Parallel answers. The same prompt goes to several frontier models at once — different labs, different training data, different failure modes.
2. Anonymized peer review. Each model reads the other models' answers with identities stripped, so no model knows which lab wrote what — including its own answer. This prevents brand deference (models rating GPT or Claude highly because of the name) and self-preference.
3. Chairman / moderator synthesis. A final model reads the answers plus the peer reviews and writes one synthesized response.

The payoff is error de-correlation: a hallucination from one model rarely survives review by four models trained on different data. Where the council agrees, confidence is earned; where it splits, you've found the genuinely hard part of your question.

Karpathy's llm-council repo, accurately described

In November 2025, Andrej Karpathy published llm-council — an open-source web app implementing exactly this pipeline. It went viral (~23.3k GitHub stars as of July 2026) and effectively named the category. What it does:

• Sends your query to a configurable set of models, which answer independently.
• Shows each model the others' anonymized answers and has them rank the responses.
• A designated Chairman model reads everything and produces the final synthesis.
• Routes all model calls through OpenRouter, so one API key covers every lab.

What running it requires: cloning the repo, installing dependencies, creating a .env file, and supplying your own OpenRouter API key — you pay per token on your own account. What it deliberately leaves out: persistence and memory (each session starts from zero), mobile apps, and integrations. It's a reference implementation, not a product — and per its commit history it hasn't been developed since November 2025.

A hosted LLM council: the same architecture with zero setup

Council AI is an independent productization of the same architecture. The pipeline is the one described above — parallel answers from frontier models, anonymized cross-review so no model knows who wrote what, and moderator synthesis — with the additions you'd expect from a product:

• Consensus score on every answer — a numeric measure of how strongly the models agreed, with disagreements surfaced, not averaged away.
• 25+ curated models across 9 labs — OpenAI, Anthropic, Google, xAI, DeepSeek, Alibaba, Mistral, Moonshot, Z.ai — kept current as lineups change.
• Hosted, web + iOS — sign up and ask; no repo, no .env, no API keys.
• Persistent memory, including ChatGPT/Claude memory import, plus a personal RAG document library and an MCP server on the Ultra tier.

One thing this page won't claim: endorsement. Karpathy is not affiliated with Council AI and has never promoted it. The honest description is that the architecture he open-sourced is the architecture Council runs as a product.

karpathy/llm-council vs Council AI: the honest comparison

And where the repo genuinely wins:

• Free — beyond your own token spend, there's no subscription.
• Fully local control — your keys, your machine, your data path.
• Hackability — it's a small, readable codebase; change the council composition, the ranking prompt, or the Chairman model in an afternoon.

If you're a developer who wants to own the pipeline, clone the repo. If you want the pattern as a daily tool — with memory, current models, and nothing to maintain — that's what the hosted version is for.

Do LLM councils actually beat a single model?

The honest answer: yes on deep research and verification-heavy work, measurably — and no, not everywhere.

On OpenRouter's DRACO deep-research benchmark, fused multi-model panels scored 69.0 versus 65.3 for the best solo model (Claude Fable 5) — and roughly 75% of that lift came from the synthesis step, not merely from having more answers to choose from. That result matches the council thesis: the value isn't ten answers, it's the structured reconciliation of ten answers.

The limits are just as real. Single strong models still win at pure coding speed — a council adds latency a tight edit loop doesn't want — and at creative writing, where synthesis tends to sand off the distinctive voice a single model produces. Councils earn their overhead when a wrong answer is expensive: research synthesis, code review, contracts, decisions. For casual chat, one good model is enough.

Frequently asked questions