consolidate-transcripts

dparedesi's avatarfrom dparedesi

Consolidate transcripts from a channel into a single file, sorted by date (newest first), up to 800K tokens. Use when preparing transcripts for LLM context or bulk analysis.

⭐6stars🔀0forks📁View on GitHub🕐Updated Dec 26, 2025
[Memory and Context Management]

When & Why to Use This Skill

This Claude skill is a specialized utility designed to optimize LLM context management by consolidating multiple YouTube transcripts into a single, token-aware Markdown file. It solves the challenge of context window limits by providing precise token counting (up to 800K) and chronological sorting, making it an essential tool for users performing bulk analysis or building comprehensive knowledge bases for high-context models like Claude and GPT-4.

Use Cases

  • LLM Context Optimization: Merge an entire channel's worth of transcripts into a single file while strictly adhering to model-specific token limits (e.g., 128K for GPT-4 or 200K for Claude) to prevent data truncation.
  • Bulk Research and Synthesis: Prepare large datasets of video content for deep-dive analysis, allowing an AI to identify recurring themes or historical trends across hundreds of transcripts simultaneously.
  • Structured Knowledge Base Creation: Automatically generate a unified archive with a Table of Contents and metadata, transforming raw transcript folders into a searchable and organized research document.
  • Token-Efficient Prompt Engineering: Use the integrated tiktoken counting to maximize the amount of relevant information included in a prompt without exceeding the 1-million-token ceiling of advanced AI agents.
nameconsolidate-transcripts
descriptionConsolidate transcripts from a channel into a single file, sorted by date (newest first), up to 800K tokens. Use when preparing transcripts for LLM context or bulk analysis.

Consolidate Transcripts

Why? LLMs have context limits. This skill merges multiple transcripts into a single file with accurate token counting, so you can feed an entire channel's content to Claude or GPT without exceeding limits.

Quick Start

python scripts/consolidate_transcripts.py <channel_name>

Output: ~/Documents/YTScriber/<channel_name>/<channel_name>-consolidated.md

[!NOTE] This feature is currently a standalone script. A ytscriber consolidate CLI command is planned for a future release.

Workflow

1. Identify the Channel

List available channels:

ls ~/Documents/YTScriber/

2. Choose Token Limit

Use Case Recommended Limit Flag
Claude (200K context) 150000 --limit 150000
GPT-4 Turbo (128K) 100000 --limit 100000
Full archive (Claude Pro) 800000 (default)
Quick sample 50000 --limit 50000

[!TIP] The default 800K limit leaves ~200K tokens for prompts and responses when using Claude's 1M context.

3. Run Consolidation

python scripts/consolidate_transcripts.py <channel_name> [--limit TOKENS] [--verbose]

Examples:

# Default (800K tokens)
python scripts/consolidate_transcripts.py library-of-minds

# Custom limit for GPT-4
python scripts/consolidate_transcripts.py aws-reinvent-2025 --limit 100000

# Verbose output showing all included files
python scripts/consolidate_transcripts.py dwarkesh-patel --verbose

4. Verify Output

Check the consolidated file was created:

ls -la ~/Documents/YTScriber/<channel_name>/*-consolidated.md

Parameters

Option Description Default
channel_name Folder name in data directory Required
--limit, -l Maximum tokens to include 800000
--verbose, -v Show detailed file list False

Output Format

The consolidated file includes:

  1. Header — Generation metadata, total transcripts, token/word counts
  2. Table of Contents — Dates, titles, tokens, words per transcript
  3. Transcripts — Full text with title, date, author, source URL

Troubleshooting

Problem Cause Solution
ModuleNotFoundError: tiktoken tiktoken not installed pip install tiktoken
No transcripts found Empty transcripts folder Run ytscriber download first
FileNotFoundError Channel doesn't exist Check ls ~/Documents/YTScriber/ for valid names
Output file is small Few transcripts available Use --verbose to see what was included
Token count seems wrong Old tiktoken version pip install --upgrade tiktoken

Common Mistakes

  1. Wrong channel name — Use the folder name exactly as shown in ls ~/Documents/YTScriber/, not the YouTube channel name.
  2. Forgetting to download transcripts first — Consolidation requires transcripts to exist. Run ytscriber download first.
  3. Using too high a limit — If you exceed your LLM's context, you'll get truncation errors. Use the limit guide above.
  4. Expecting real-time updates — Re-run consolidation after downloading new transcripts.

Reference

  • Transcripts sorted newest first (descending by date)
  • Files without dates in filename are placed last
  • Token counting uses cl100k_base encoding (GPT-4/Claude compatible)
  • Consolidated files are gitignored (not committed)
  • Re-running overwrites the previous consolidated file