Files
claw-ideas/ideas/data-engineering/pipeline-orchestrator.md
Space-Banane 9956660ed8
All checks were successful
Check Links / lycheee (push) Successful in 5s
added more bs
2026-04-03 18:21:19 +02:00

469 B

Data Pipeline Orchestrator

Automate ETL/ELT workflows: schedule jobs, handle retries, detect schema drift, spin up jobs on demand, and auto-heal failed runs.

Agent capabilities to add:

  • Connect to data sources, run transformations, and validate outputs
  • Auto-scale compute for heavy jobs and failover to cheaper instances
  • Send alerts, open issues, or create remediation PRs when data quality breaks

Integrations: Airflow, Prefect, dbt, Snowflake, BigQuery, S3.