llm-council

Facilitate council-style deliberation among specialized models with structured prompts, evidence gates, and explicit confidence ceilings.

⭐8stars🔀2forks📁View on GitHub🕐Updated Jan 4, 2026⬇️1downloads

When & Why to Use This Skill

The llm-council skill implements a robust multi-model consensus mechanism based on the Karpathy LLM Council pattern to facilitate high-stakes decision-making. By orchestrating a three-stage deliberation process—Collect, Rank, and Synthesize—across multiple specialized AI models (such as Claude, Gemini, and Codex), it effectively mitigates hallucination risks and ensures that critical technical choices are backed by structured evidence and explicit confidence scores.

Use Cases

  • Complex Architecture Decisions: Evaluate competing system designs (e.g., Microservices vs. Monolith) by aggregating perspectives from multiple models to reach a validated consensus.
  • Security & Risk Assessment: Conduct rigorous audits of authentication flows or infrastructure changes where multi-model validation is required to minimize security vulnerabilities.
  • Critical Bug Triage: Analyze high-impact production issues through a 'council' to determine root causes and prioritize fixes with objective, cross-model rationale.
  • Strategic Technology Selection: Compare and rank frameworks, databases, or programming languages based on specific project requirements with a synthesized final recommendation.
namellm-council
descriptionMulti-model consensus using Karpathy LLM Council pattern for critical decisions
allowed-toolsBash, Read, Write, TodoWrite

LLM Council Skill


LIBRARY-FIRST PROTOCOL (MANDATORY)

Before writing ANY code, you MUST check:

Step 1: Library Catalog

  • Location: .claude/library/catalog.json
  • If match >70%: REUSE or ADAPT

Step 2: Patterns Guide

  • Location: .claude/docs/inventories/LIBRARY-PATTERNS-GUIDE.md
  • If pattern exists: FOLLOW documented approach

Step 3: Existing Projects

  • Location: D:\Projects\*
  • If found: EXTRACT and adapt

Decision Matrix

Match Action
Library >90% REUSE directly
Library 70-90% ADAPT minimally
Pattern exists FOLLOW pattern
In project EXTRACT
No match BUILD (add to library after)

Purpose

Run 3-stage multi-model consensus for critical decisions where:

  • Single-model hallucination risk is unacceptable
  • Multiple perspectives improve decision quality
  • High-stakes choices need validation

Architecture (Karpathy Pattern)

STAGE 1: COLLECT
  +---> Claude ---> Response A
  |
Query --+---> Gemini ---> Response B
  |
  +---> Codex ----> Response C

STAGE 2: RANK
  Each model reviews others (anonymized)
  Produces rankings with rationale

STAGE 3: SYNTHESIZE
  Chairman aggregates rankings
  Produces final answer with consensus score

When to Use

Perfect For:

  • Architecture decisions
  • Technology selection
  • Critical bug triage
  • Security assessment
  • High-risk deployments
  • Contentious design choices

Don't Use When:

  • Simple, low-risk decisions
  • Time-critical responses
  • Single correct answer exists
  • Cost is a concern (3x API usage)

Usage

Basic Council

/llm-council "Should we use microservices or monolith for this system?"

With Threshold

/llm-council "Which auth approach is best?" --threshold 0.75

With Chairman Override

/llm-council "Architecture decision" --chairman gemini

Command Pattern

bash scripts/multi-model/llm-council.sh "<query>" "<threshold>" "<chairman>"

Configuration

Parameter Default Description
threshold 0.67 Minimum consensus score
chairman claude Model that synthesizes final answer
models [claude, gemini, codex] Participating models

Consensus Scoring

  • >0.80: Strong consensus - proceed with confidence
  • 0.67-0.80: Moderate consensus - consider minority views
  • <0.67: Weak consensus - escalate to human review

Memory Integration

Results stored to Memory-MCP:

  • Key: multi-model/council/decisions/{query_id}
  • Tags: WHO=llm-council, WHY=consensus-decision

Output Format

{
  "query": "Original question",
  "final_answer": {
    "synthesis": "Combined answer...",
    "chairman": "claude"
  },
  "consensus_score": 0.85,
  "responses": {
    "claude": "...",
    "gemini": "...",
    "codex": "..."
  },
  "rankings": [
    {"model": "A", "rank": 1, "rationale": "..."}
  ]
}

Failure Modes

Deadlock (No Consensus)

  • All models disagree
  • Consensus < threshold
  • Action: Store for human review

Model Unavailable

  • One model times out
  • Action: Continue with 2 models (2/3 quorum)

Chairman Failure

  • Synthesis fails
  • Action: Fallback to highest-ranked response

Integration Examples

Architecture Decision

const decision = await runCouncil(
  "Microservices vs Monolith for our scale?",
  { threshold: 0.75 }
);

if (decision.consensus_score >= 0.75) {
  proceed(decision.final_answer);
} else {
  escalateToHuman(decision);
}

Security Assessment

const assessment = await runCouncil(
  "Is this authentication approach secure?",
  { threshold: 0.80 }
);
// Higher threshold for security decisions

Sources