numpy-set-ops

majiayu000's avatarfrom majiayu000

Set-theoretic operations for finding unique elements, membership testing, and array intersections. Triggers: unique, isin, intersect1d, setdiff1d, union1d.

⭐0stars🔀0forks📁View on GitHub🕐Updated Jan 5, 2026
[Data Cleaning]

When & Why to Use This Skill

This Claude skill leverages NumPy's vectorized set operations to perform efficient data deduplication, membership testing, and array intersections. It streamlines the process of identifying unique elements and filtering datasets, making it an essential tool for high-performance data preprocessing, synchronization, and analysis.

Use Cases

  • Deduplicating large feature matrices or datasets to ensure data integrity and remove redundant records.
  • Filtering datasets by creating boolean masks to exclude forbidden values or specific outliers based on membership testing.
  • Synchronizing and finding commonalities between two distinct datasets using array intersections and differences.
  • Optimizing data storage through unique value extraction and inverse index mapping for efficient data compression.
namenumpy-set-ops
descriptionSet-theoretic operations for finding unique elements, membership testing, and array intersections. Triggers: unique, isin, intersect1d, setdiff1d, union1d.

Overview

NumPy provides vectorized set operations for 1D arrays and multidimensional subarrays. These tools allow for deduplication, membership testing, and finding differences/intersections between datasets.

When to Use

  • Deduplicating rows in a large feature matrix.
  • Filtering a dataset to exclude a list of forbidden values.
  • Synchronizing two datasets by finding their intersection.
  • Compressing data by storing unique values and their index mappings.

Decision Tree

  1. Need to find non-duplicate elements?
    • Use np.unique.
  2. Need to reconstruct the original array from unique values?
    • Set return_inverse=True in np.unique.
  3. Checking if elements exist in another list?
    • Use np.isin(data, target_list).

Workflows

  1. Finding Unique Rows in a Dataset

    • Create a 2D array.
    • Call np.unique(arr, axis=0).
    • Inspect the result to see the deduplicated records.

  2. Reconstructing an Array from Sets

    • Call u, inv = np.unique(arr, return_inverse=True).
    • Store 'u' and 'inv' separately (useful for data compression).
    • Rebuild the original array using u[inv].

  3. Filtering by Membership

    • Define a 'forbidden' set of values.
    • Generate a boolean mask using ~np.isin(data, forbidden).
    • Filter the data: clean_data = data[mask].

Non-Obvious Insights

  • Flattening by Default: Set operations work on flattened 1D versions of input arrays unless an axis is explicitly specified.
  • NaN Handling: Like sorting, unique treats NaN as a value and sorts it to the end of the unique output.
  • Lexicographic Row Sort: When axis=0 is used in unique, the resulting unique rows are sorted lexicographically.

Evidence

  • "Returns the sorted unique elements of an array." Source
  • "isin(element, test_elements...)... broadcasting over element only." Source

Scripts

  • scripts/numpy-set-ops_tool.py: Routines for unique row detection and inverse reconstruction.
  • scripts/numpy-set-ops_tool.js: Simulated set intersection logic.

Dependencies

  • numpy (Python)

References