experiment-advisor
Given a research goal, use existing skills, training reports, and references to propose a small, concrete experiment plan.
When & Why to Use This Skill
The Experiment Advisor is a specialized Claude skill designed to streamline the research and development lifecycle. By synthesizing historical training reports, experiment logs, and technical references, it generates concrete, data-driven experiment plans. This skill bridges the gap between high-level research objectives and actionable technical execution, ensuring that new experiments are optimized based on past insights and resource constraints.
Use Cases
- Machine Learning Optimization: Automatically propose the next set of hyperparameters (learning rates, batch sizes, architecture tweaks) based on previous training run results and loss curves.
- Resource-Constrained Planning: Design experiment batches that maximize research insights while strictly adhering to specific hardware availability or compute budgets.
- Failure Prevention: Cross-reference current research goals with troubleshooting logs to suggest experiment designs that explicitly avoid previously documented errors or bottlenecks.
- Research Continuity: Quickly resume projects or pivot strategies by synthesizing days of experiment logs into a structured 'what to try next' action plan.
| name | experiment-advisor |
|---|---|
| description | > |
| short-description | "Plan next experiments using past knowledge" |
Skill: experiment-advisor
When to use
Use this skill when:
- The user message starts with
<advise>, or - The user clearly asks "What should we try next?" in an experiment context.
Inputs
- User goal (including constraints like run budget, hardware).
- Optional references to files (configs, reports, logs) mentioned in the goal.
Behavior
Understand the goal
- Parse which models, datasets, tasks, and constraints are involved.
- If unclear, ask a concise clarifying question.
Gather context
- Scan
.codex/skills/for relevant result skills:- Matching models, datasets, or task types.
- Read relevant
training_reports/*.md:- Start with the most recent ones that mention these models/datasets.
- Skim the last few days of
references/experiment-log.md. - If the goal mentions an error, look up matching patterns in
references/troubleshooting.md.
- Scan
Propose a plan
- Design 2–5 experiments, each with:
- Input data selection (possibly mixtures).
- Model / architecture details (at a high level).
- Key hyperparameters (LR, batch size, epochs, etc.).
- Any relevant variation (e.g. different mixture ratios).
- Respect run/hardware constraints.
- When applicable, reuse defaults from existing skills instead of inventing new ones.
- Design 2–5 experiments, each with:
Output format
Start with a short natural-language summary.
Then provide a markdown table with the experiments:
id description key_differences notes Optionally propose file paths for configs or reports to create.
If errors were mentioned, explicitly call out how this plan avoids known failure patterns from
troubleshooting.md.
Logging
- When useful, append a short entry to
references/experiment-log.mdsummarizing the newly proposed plan (only if the user agrees). - Include a short "General description" line in the entry for non-technical context.
- When useful, append a short entry to