workflow-management
TD workflow debugging and operations. Covers tdx wf commands for monitoring (sessions, attempt, logs), retry/backfill patterns, alerting (_error with Slack/email), and data quality checks.
When & Why to Use This Skill
This Claude skill provides a comprehensive toolkit for TD (Treasure Data) workflow management, debugging, and operational monitoring. It enables users to efficiently track session statuses, inspect detailed task logs, and manage complex data pipelines using tdx CLI commands. By incorporating robust retry patterns, backfill logic, and automated alerting via Slack or email, the skill ensures high reliability and data integrity. It is designed to streamline the troubleshooting process for DevOps and Data Engineers, offering practical solutions for incident response and routine maintenance in data orchestration environments.
Use Cases
- Monitoring and filtering workflow sessions in real-time to identify and isolate failed tasks or performance bottlenecks.
- Debugging pipeline failures by inspecting specific attempt logs, verifying query syntax, and validating session parameters.
- Automating incident notifications through Slack or Email alerts to ensure immediate response to critical workflow errors.
- Implementing data quality checks and validation gates within workflows to prevent the processing of incomplete or corrupted data.
- Managing complex data recovery operations, including idempotent backfills and task-specific retries with parameter overrides.
- Securing sensitive credentials and API keys using integrated secrets management for safe workflow execution.
| name | workflow-management |
|---|---|
| description | TD workflow debugging and operations. Covers tdx wf commands for monitoring (sessions, attempt, logs), retry/backfill patterns, alerting (_error with Slack/email), and data quality checks. |
TD Workflow Management
Setup & Context
tdx wf use my_project # Set default project for session
tdx wf pull my_project # Pull project locally for editing
tdx wf push # Push changes with diff preview
Monitoring Commands
tdx wf sessions # List runs (uses session context)
tdx wf sessions --status error # Filter by status
tdx wf attempt <id> tasks # Show task status
tdx wf attempt <id> logs +task_name # View logs
Debugging Steps
- Check error in
tdx wf attempt <id> logs +failed_task - Verify query syntax if td> failed
- Check time ranges - does data exist for session_date?
- Validate parameter values
- Check resource limits (memory, timeout)
Retry Operations
tdx wf attempt <id> retry # Retry from start
tdx wf attempt <id> retry --resume-from +step # Retry from task
tdx wf attempt <id> retry --params '{"key":"val"}' # Override params
tdx wf attempt <id> kill # Stop running
Alerting
+critical_task:
td>: queries/important.sql
_error:
+slack_alert:
sh>: |
curl -X POST ${secret:slack.webhook_url} \
-H 'Content-Type: application/json' \
-d '{"text": "Workflow failed: ${session_id}"}'
Data Quality Checks
+process:
td>: queries/process.sql
create_table: results
+validate:
td>:
query: |
SELECT COUNT(*) as cnt,
SUM(CASE WHEN id IS NULL THEN 1 ELSE 0 END) as nulls
FROM results
store_last_results: true
+check:
if>: ${td.last_results.cnt == 0}
_do:
+fail:
sh>: exit 1
Wait for Data
+wait_for_data:
sh>: |
for i in {1..30}; do
COUNT=$(tdx query -d analytics "SELECT COUNT(*) FROM src WHERE date='${session_date}'" --format csv | tail -1)
if [ "$COUNT" -gt 0 ]; then exit 0; fi
sleep 60
done
exit 1
Idempotent Operations
+safe_insert:
td>:
query: |
DELETE FROM target WHERE date = '${session_date}';
INSERT INTO target SELECT * FROM source WHERE date = '${session_date}'
Backfill Pattern
+backfill:
loop>:
dates: ["2024-01-01", "2024-01-02", "2024-01-03"]
_do:
+process:
call>: main_workflow.dig
params:
session_date: ${dates}
Secrets Management
tdx wf secrets list # List secret keys (values hidden)
tdx wf secrets set API_KEY=xxx # Set a secret
tdx wf secrets delete API_KEY # Delete a secret
Usage in .dig files:
+task:
sh>: curl -H "Authorization: ${secret:API_KEY}" https://api.example.com
Common Issues
| Issue | Solution |
|---|---|
| Timeout | Add timeout: 3600s, _retry: 2 |
| Intermittent failures | Add _retry: 5 with exponential backoff |
| Out of memory | Reduce data volume, use approx functions |
| Duplicate runs | Use idempotent DELETE+INSERT pattern |