data-explore

在Wellcome Collection数据仓库中探索Hive表和数据集。使用此技能查询表、理解模式或分析数据分布。通过/data-explore命令调用。

📖 中文导读:在Wellcome Collection数据仓库中探索Hive表和数据集。使用此技能查询表、理解模式或分析数据分布。通过/data-explore命令调用。 以下正文为该技能的原始 SKILL.md 内容(英文),上方安装命令可直接使用。

name: data-explore description: Explore Hive tables and datasets in the Wellcome Collection data warehouse. Use this skill to query tables, understand schemas, or analyze data distributions. Invoke with /data-explore.

Data Exploration

This skill helps explore the Wellcome Collection data warehouse stored in Hive tables.

Available Tables

Core Tables

Table Records Description
works 738,073 Wellcome Collection works catalog
images 128,755 Image metadata from IIIF manifests
iiif_manifests ~340,000 IIIF manifest metadata
iiif_canvases ~42M Page-level canvas data

Text Tables

Table Records Description
alto_text ~226,000 OCR text from works (9.8B words total)
alto_sentence ~550M Sentence-level text
alto_text_chunks ~74M Text chunks for embedding

Embedding Tables

Table Description
vlm_embeddings VLM image embeddings (1536-dim)
text_embeddings Qwen3 text embeddings

Label Tables

Table Records Description
dn_labels 646,786 Combined genres/subjects/contributors

Quick Start

Connect to Spark

from wc_simd.utility import get_spark_session
spark = get_spark_session()

List Tables

spark.sql("SHOW TABLES").show()

Explore Schema

spark.table("works").printSchema()

Sample Data

spark.table("works").show(10, truncate=False)

Count Records

spark.table("works").count()

Common Queries

Works with OCR Text

SELECT COUNT(*) FROM works w
JOIN alto_text t ON w.id = t.id

Genre Distribution

SELECT genre.label, COUNT(*) as cnt
FROM works
LATERAL VIEW explode(genres) AS genre
GROUP BY genre.label
ORDER BY cnt DESC
LIMIT 20

Subject Distribution

SELECT subject.label, COUNT(*) as cnt
FROM works
LATERAL VIEW explode(subjects) AS subject
GROUP BY subject.label
ORDER BY cnt DESC
LIMIT 20

Concept Types

SELECT concept.type, COUNT(*) as cnt
FROM works
LATERAL VIEW explode(subjects) AS subject
LATERAL VIEW explode(subject.concepts) AS concept
GROUP BY concept.type

Key Statistics

  • Total words: 9.8 billion (~12,500 King James Bibles)
  • Text chunks: 74.3 million (1000 chars, 200 overlap)
  • Languages: 78.9% English
  • Genre labels: 1,523 unique
  • Subject labels: 204,322 unique
  • Concept labels: 126,714 unique