correlation-explorer
Find and visualize correlations between variables in datasets. Use for data exploration, feature selection, or identifying relationships between columns.
When & Why to Use This Skill
The Correlation Explorer skill is a comprehensive data analytics tool designed to uncover and visualize complex relationships within datasets. By supporting multiple statistical methods—including Pearson, Spearman, and Kendall—and providing automated heatmap visualizations, it enables users to perform deep data exploration, streamline feature selection for machine learning, and validate statistical significance with p-value testing.
Use Cases
- Feature Engineering: Identify and remove redundant, highly correlated features to improve machine learning model performance and reduce multicollinearity.
- Business Intelligence: Analyze sales datasets to determine which factors, such as marketing spend or customer demographics, correlate most strongly with revenue growth.
- Scientific Research: Use significance testing and p-values to validate relationships between variables in experimental research data and academic studies.
- Financial Data Exploration: Visualize correlations between different asset classes, stock prices, or economic indicators to assist in risk management and portfolio diversification.
| name | correlation-explorer |
|---|---|
| description | Find and visualize correlations between variables in datasets. Use for data exploration, feature selection, or identifying relationships between columns. |
Correlation Explorer
Analyze correlations between variables in CSV/Excel datasets.
Features
- Correlation Matrix: Compute all pairwise correlations
- Heatmap Visualization: Color-coded correlation display
- Significance Testing: P-values for correlations
- Multiple Methods: Pearson, Spearman, Kendall
- Strong Correlations: Find highly correlated pairs
- Target Analysis: Correlations with specific variable
Quick Start
from correlation_explorer import CorrelationExplorer
explorer = CorrelationExplorer()
# Load and analyze
explorer.load_csv("sales_data.csv")
matrix = explorer.correlation_matrix()
# Find strong correlations
strong = explorer.find_strong_correlations(threshold=0.7)
print(strong)
# Generate heatmap
explorer.plot_heatmap("correlation_heatmap.png")
CLI Usage
# Compute correlation matrix
python correlation_explorer.py --input data.csv --output correlations.csv
# Generate heatmap
python correlation_explorer.py --input data.csv --heatmap heatmap.png
# Find strong correlations
python correlation_explorer.py --input data.csv --strong --threshold 0.7
# Correlations with target variable
python correlation_explorer.py --input data.csv --target sales
# Use Spearman correlation
python correlation_explorer.py --input data.csv --method spearman
# Include p-values
python correlation_explorer.py --input data.csv --pvalues
API Reference
CorrelationExplorer Class
class CorrelationExplorer:
def __init__(self)
# Data loading
def load_csv(self, filepath: str, **kwargs) -> 'CorrelationExplorer'
def load_dataframe(self, df: pd.DataFrame) -> 'CorrelationExplorer'
# Analysis
def correlation_matrix(self, method: str = "pearson") -> pd.DataFrame
def correlation_with_pvalues(self, method: str = "pearson") -> tuple
def correlate_with_target(self, target: str, method: str = "pearson") -> pd.Series
# Discovery
def find_strong_correlations(self, threshold: float = 0.7) -> list
def find_weak_correlations(self, threshold: float = 0.3) -> list
# Visualization
def plot_heatmap(self, output: str, **kwargs) -> str
def plot_scatter(self, var1: str, var2: str, output: str) -> str
# Export
def to_csv(self, output: str) -> str
def to_json(self, output: str) -> str
Correlation Methods
| Method | Best For |
|---|---|
pearson |
Linear relationships, normal data |
spearman |
Non-linear, ordinal data |
kendall |
Small samples, ordinal data |
# Pearson (default) - parametric
matrix = explorer.correlation_matrix(method="pearson")
# Spearman - rank-based, non-parametric
matrix = explorer.correlation_matrix(method="spearman")
# Kendall - robust to outliers
matrix = explorer.correlation_matrix(method="kendall")
Output Format
Correlation Matrix
sales marketing customers
sales 1.000 0.854 0.723
marketing 0.854 1.000 0.612
customers 0.723 0.612 1.000
Strong Correlations
[
{"var1": "sales", "var2": "marketing", "correlation": 0.854, "abs_corr": 0.854},
{"var1": "sales", "var2": "customers", "correlation": 0.723, "abs_corr": 0.723}
]
With P-Values
{
"correlations": DataFrame,
"pvalues": DataFrame,
"significant": [...], # p < 0.05
}
Example Workflows
Feature Selection
explorer = CorrelationExplorer()
explorer.load_csv("features.csv")
# Find features correlated with target
target_corr = explorer.correlate_with_target("target")
important_features = target_corr[abs(target_corr) > 0.3].index.tolist()
print(f"Important features: {important_features}")
# Find multicollinear features (to potentially drop)
strong = explorer.find_strong_correlations(threshold=0.9)
print("Highly correlated pairs (consider dropping one):")
for pair in strong:
print(f" {pair['var1']} <-> {pair['var2']}: {pair['correlation']:.3f}")
Sales Analysis
explorer = CorrelationExplorer()
explorer.load_csv("sales_data.csv")
# What drives sales?
sales_corr = explorer.correlate_with_target("revenue")
print("Factors correlated with revenue:")
for var, corr in sales_corr.sort_values(ascending=False).items():
if var != "revenue":
print(f" {var}: {corr:.3f}")
# Visualize
explorer.plot_heatmap("sales_correlations.png")
Data Exploration
explorer = CorrelationExplorer()
explorer.load_csv("dataset.csv")
# Get full picture
corr, pvals = explorer.correlation_with_pvalues()
# Find all significant correlations
significant = []
for i in range(len(corr.columns)):
for j in range(i+1, len(corr.columns)):
if pvals.iloc[i, j] < 0.05:
significant.append({
'var1': corr.columns[i],
'var2': corr.columns[j],
'r': corr.iloc[i, j],
'p': pvals.iloc[i, j]
})
Heatmap Options
explorer.plot_heatmap(
output="heatmap.png",
cmap="coolwarm", # Color scheme
annot=True, # Show values
figsize=(12, 10), # Figure size
vmin=-1, vmax=1, # Color scale
title="Correlation Matrix"
)
Dependencies
- pandas>=2.0.0
- numpy>=1.24.0
- scipy>=1.10.0
- matplotlib>=3.7.0
- seaborn>=0.12.0