name: data-explore
description: Explore Hive tables and datasets in the Wellcome Collection data warehouse. Use this skill to query tables, understand schemas, or analyze data distributions. Invoke with /data-explore.
Data Exploration
This skill helps explore the Wellcome Collection data warehouse stored in Hive tables.
Available Tables
Core Tables
Table
Records
Description
works
738,073
Wellcome Collection works catalog
images
128,755
Image metadata from IIIF manifests
iiif_manifests
~340,000
IIIF manifest metadata
iiif_canvases
~42M
Page-level canvas data
Text Tables
Table
Records
Description
alto_text
~226,000
OCR text from works (9.8B words total)
alto_sentence
~550M
Sentence-level text
alto_text_chunks
~74M
Text chunks for embedding
Embedding Tables
Table
Description
vlm_embeddings
VLM image embeddings (1536-dim)
text_embeddings
Qwen3 text embeddings
Label Tables
Table
Records
Description
dn_labels
646,786
Combined genres/subjects/contributors
Quick Start
Connect to Spark
from wc_simd.utility import get_spark_session
spark = get_spark_session()
List Tables
spark.sql("SHOW TABLES").show()
Explore Schema
spark.table("works").printSchema()
Sample Data
spark.table("works").show(10, truncate=False)
Count Records
spark.table("works").count()
Common Queries
Works with OCR Text
SELECT COUNT(*) FROM works w
JOIN alto_text t ON w.id = t.id
Genre Distribution
SELECT genre.label, COUNT(*) as cnt
FROM works
LATERAL VIEW explode(genres) AS genre
GROUP BY genre.label
ORDER BY cnt DESC
LIMIT 20
Subject Distribution
SELECT subject.label, COUNT(*) as cnt
FROM works
LATERAL VIEW explode(subjects) AS subject
GROUP BY subject.label
ORDER BY cnt DESC
LIMIT 20
Concept Types
SELECT concept.type, COUNT(*) as cnt
FROM works
LATERAL VIEW explode(subjects) AS subject
LATERAL VIEW explode(subject.concepts) AS concept
GROUP BY concept.type
Key Statistics
Total words: 9.8 billion (~12,500 King James Bibles)
Text chunks: 74.3 million (1000 chars, 200 overlap)