prepare-dataset
Process and validate datasets for training. Use when setting up data pipelines.
When & Why to Use This Skill
The 'prepare-dataset' skill is a specialized tool designed to automate the critical stages of machine learning data preprocessing. It streamlines the creation of robust data pipelines by providing automated functions for data loading, validation, normalization, and augmentation. By ensuring data integrity and consistency, this skill helps developers and data scientists accelerate the transition from raw data to model-ready training sets, significantly improving the efficiency of ML workflows.
Use Cases
- ML Pipeline Automation: Setting up automated workflows to ingest, clean, and transform raw data into structured formats for model training.
- Data Quality Assurance: Validating dataset statistics, checking for missing values, and identifying outliers to ensure high-quality input for algorithms.
- Feature Engineering and Scaling: Applying normalization, standardization, and categorical encoding to prepare features for optimal model performance.
- Dataset Balancing and Augmentation: Enhancing training sets through transformations like rotation or flipping to improve model generalization and robustness.
- Experimental Data Splitting: Automatically partitioning datasets into training, validation, and test sets to ensure unbiased model evaluation.
| name | prepare-dataset |
|---|---|
| description | "Process and validate datasets for training. Use when setting up data pipelines." |
| mcp_fallback | none |
| category | ml |
| tier | 2 |
| user-invocable | false |
Prepare Dataset
Load, preprocess, and validate datasets for machine learning model training including normalization and augmentation.
When to Use
- Setting up data pipelines for training
- Normalizing and cleaning raw data
- Splitting into train/validation/test sets
- Applying data augmentation
Quick Reference
# Dataset preparation pipeline
class DatasetLoader:
def load(self, path: str) -> Tuple[ndarray, ndarray]:
# Load raw data
pass
def normalize(self, data: ndarray) -> ndarray:
# Normalize to [0, 1] or standardize
pass
def split(self, data: ndarray, ratios: Tuple[float, float, float]):
# Split into train/val/test
pass
def augment(self, data: ndarray) -> ndarray:
# Apply transformations if needed
pass
Workflow
- Load raw data: Read dataset from file (CSV, HDF5, NumPy)
- Validate data: Check shape, dtype, missing values
- Preprocess: Normalize, standardize, encode categorical features
- Split sets: Create train/validation/test splits
- Augment data: Apply transformations if needed (rotation, flip, etc.)
Output Format
Dataset preparation report:
- Raw data shape and statistics
- Data validation results (missing values, outliers)
- Preprocessing applied (normalization, encoding)
- Train/val/test split sizes
- Final dataset shape and statistics
- Augmentation transformations applied
References
- See
extract-hyperparametersskill for data preprocessing config - See
evaluate-modelskill for test set evaluation - See
/notes/review/mojo-ml-patterns.mdfor Mojo data loading