data-designer

majiayu000 的头像作者 majiayu000

使用统计采样器和 Claude 的本地法学硕士功能生成高质量的综合数据集。当用户要求创建合成数据、生成数据集、创建假/模拟数据、生成测试数据、训练数据或任何数据生成任务时使用。支持 CSV、JSON、JSONL、Parquet 输出。改编自 NVIDIA NeMo DataDesigner (Apache 2.0)。

📖 中文导读:使用统计采样器和 Claude 的本地法学硕士功能生成高质量的综合数据集。当用户要求创建合成数据、生成数据集、创建假/模拟数据、生成测试数据、训练数据或任何数据生成任务时使用。支持 CSV、JSON、JSONL、Parquet 输出。改编自 NVIDIA NeMo DataDesigner (Apache 2.0)。 以下正文为该技能的原始 SKILL.md 内容(英文),上方安装命令可直接使用。

name: data-designer description: Generate high-quality synthetic datasets using statistical samplers and Claude's native LLM capabilities. Use when users ask to create synthetic data, generate datasets, create fake/mock data, generate test data, training data, or any data generation task. Supports CSV, JSON, JSONL, Parquet output. Adapted from NVIDIA NeMo DataDesigner (Apache 2.0).

Data Designer

Generate synthetic datasets combining statistical samplers with Claude's LLM capabilities. No external API keys required.

Workflow

  1. Clarify requirements - Ask about purpose, columns, size, format
  2. Create schema - Write dataset_schema.json defining columns
  3. Generate preview - Run batch_generator.py for 3-5 rows
  4. Iterate - Refine based on feedback
  5. Generate full dataset - Batch generate, then merge
  6. Deliver - Export to requested format

Column Types

Statistical Samplers (No LLM)

Type Description Key Params
category Weighted random choice values, weights
subcategory Hierarchical (parent-based) mapping, category
uniform Uniform distribution low, high, dtype
gaussian Normal distribution mean, std, min_val, max_val
bernoulli Binary probability p, true_value, false_value
poisson Poisson distribution mean
datetime Random dates start, end, format
person Synthetic personas fields, age_range, locale
uuid Unique IDs prefix, format

LLM Columns (Claude generates)

Type Description
llm_text Free-form text
llm_code Code with syntax validation
llm_structured JSON matching schema
llm_judge Quality scoring

Schema Format

Create dataset_schema.json:

{
  "name": "dataset_name",
  "seed": 42,
  "columns": [
    {"name": "category", "type": "category", "params": {"values": ["A","B"], "weights": [0.6,0.4]}},
    {"name": "text", "type": "llm_text", "prompt": "Write about {{ category }}.", "depends_on": ["category"]}
  ],
  "output": {"format": "csv", "filename": "output"}
}

For full schema reference: references/schema.md

Jinja2 Templating

Reference columns in prompts:

Write a {{ rating }}-star review for {{ product_name }} by {{ customer.first_name }}.

Supports: {{ var }}, {{ obj.field }}, {% if %}, filters

Scripts

Generate Data

# Preview
python scripts/batch_generator.py --schema schema.json --rows 5 --output preview.json --preview

# Full generation
python scripts/batch_generator.py --schema schema.json --rows 100 --batch-size 20 --output batches/

Merge & Export

python scripts/merger.py --input batches/ --output dataset.csv --flatten

Formats: csv, json, jsonl, parquet

Generation Strategy

  1. Sampler columns first - Python scripts, fast
  2. LLM columns in dependency order - Topological sort by depends_on
  3. Batch processing - Generate in batches of 20-50 for large datasets

For LLM columns, Claude generates directly:

  • Render Jinja2 prompt with row data
  • Generate content
  • Validate if configured
  • Retry on failure (max 3)

Examples

Simple:

"Generate 50 product reviews with ratings 1-5"

Complex:

"Create 200 support tickets with: ticket_id (UUID), customer (name, email), category (billing/technical/general), priority (1-5 gaussian), description (LLM)"

Code:

"Generate 100 Python functions with description, code (validated), tests"

Tips

  • Use seed for reproducibility
  • Preview first, then scale
  • Keep LLM prompts specific
  • Use subcategory for correlated data

Attribution

Adapted from NVIDIA NeMo DataDesigner (Apache 2.0).