data-quality

timequity's avatarfrom timequity

Data quality testing with dbt tests, Great Expectations, and monitoring.

⭐0stars🔀0forks📁View on GitHub🕐Updated Dec 18, 2025
[Testing and QA]

When & Why to Use This Skill

This Claude skill provides a robust framework for data quality testing and observability using industry-standard tools like dbt and Great Expectations. It helps data engineers and analysts ensure data integrity across five key dimensions—completeness, uniqueness, validity, consistency, and timeliness—by automating schema validation, custom business logic assertions, and proactive monitoring of data pipelines.

Use Cases

  • Automated Schema Validation: Implement dbt schema tests to enforce 'not null', 'unique', and 'accepted values' constraints on critical data warehouse columns.
  • Data Pipeline Guardrails: Use Great Expectations to validate incoming datasets against complex business rules (e.g., range checks, regex patterns) before they reach production environments.
  • Proactive Anomaly Detection: Monitor data freshness, row count trends, and null percentage shifts to identify data drift or upstream pipeline failures before they impact downstream reports.
  • Relational Integrity Checks: Define relationship tests in dbt to ensure referential integrity between fact and dimension tables, preventing orphaned records and inconsistent analytics.
namedata-quality
descriptionData quality testing with dbt tests, Great Expectations, and monitoring.

Data Quality

Quality Dimensions

Dimension Description Test
Completeness No missing values NOT NULL, count checks
Uniqueness No duplicates UNIQUE, distinct counts
Validity Values in range Range checks, regex
Consistency Matches across sources Cross-table checks
Timeliness Data is fresh Freshness checks

dbt Tests

Schema Tests

models:
  - name: fct_orders
    columns:
      - name: order_id
        tests:
          - unique
          - not_null
      - name: status
        tests:
          - accepted_values:
              values: ['pending', 'completed', 'cancelled']
      - name: amount
        tests:
          - not_null
          - dbt_utils.accepted_range:
              min_value: 0
              max_value: 1000000

Custom Tests

-- tests/assert_positive_revenue.sql
select *
from {{ ref('fct_orders') }}
where amount < 0

Relationship Tests

- name: customer_id
  tests:
    - relationships:
        to: ref('dim_customer')
        field: customer_id

Great Expectations

import great_expectations as gx

context = gx.get_context()

validator = context.sources.pandas_default.read_csv("data.csv")

validator.expect_column_values_to_not_be_null("order_id")
validator.expect_column_values_to_be_unique("order_id")
validator.expect_column_values_to_be_between("amount", 0, 1000000)

results = validator.validate()

Monitoring

  • Row count trends
  • Null percentage trends
  • Schema drift detection
  • Freshness SLAs
  • Anomaly detection