data-cleaning

数据清洗、预处理和质量保证技术

📖 中文导读:数据清洗、预处理和质量保证技术 以下正文为该技能的原始 SKILL.md 内容(英文),上方安装命令可直接使用。

name: data-cleaning description: Data cleaning, preprocessing, and quality assurance techniques version: "2.0.0" sasmp_version: "2.0.0" bonded_agent: 05-programming-expert bond_type: SECONDARY_BOND

Skill Configuration

config: atomic: true retry_enabled: true max_retries: 3 backoff_strategy: exponential

Parameter Validation

parameters: tool_preference: type: string required: true enum: [python, r, excel, sql] default: python data_size: type: string required: false enum: [small, medium, large] default: medium

Observability

observability: logging_level: info metrics: [rows_cleaned, missing_handled, duplicates_removed]

Data Cleaning Skill

Overview

Master data cleaning and preprocessing techniques essential for reliable analytics.

Topics Covered

  • Missing value handling (imputation, deletion)
  • Outlier detection and treatment
  • Data type conversion and validation
  • Duplicate identification and removal
  • String cleaning and normalization

Learning Outcomes

  • Clean messy datasets
  • Handle missing data appropriately
  • Detect and treat outliers
  • Ensure data quality

Error Handling

Error Type Cause Recovery
Memory error Dataset too large Use chunking or sampling
Type conversion failed Invalid data format Apply preprocessing first
Encoding issues Wrong character encoding Detect and specify encoding
Validation failure Data doesn't meet schema Review and adjust validation rules
  • programming (for automation)
  • foundations (for data quality concepts)
  • databases-sql (for SQL-based cleaning)