Fleet 1.13:Teams are now shipping 5x more PRs with autonomous pipelines.See what's new →
FleetFleet
Agent templateData

Data Engineer AI Agent (Template)

A data engineer agent builds and maintains data pipelines: ingestion, transformation, and delivery of data from source systems to analytical targets. It writes pipeline code, manages schema evolution, and ensures that data quality checks run on each pipeline stage.

Data engineering work is tightly coupled to your specific stack: which orchestrator you use, how schemas are registered, and what your data quality framework looks like. A role-specific prompt captures these so the agent writes Airflow DAGs rather than Prefect flows, and uses your existing base operators rather than inventing new ones.

What this agent owns

  • Write and maintain ETL and ELT pipeline code for batch and streaming sources
  • Manage schema evolution and migration in your data warehouse or lake
  • Implement data quality checks at ingestion and transformation stages
  • Monitor pipeline run failures and diagnose data freshness issues
  • Optimize slow transformation steps to meet SLA windows

Recommended model: Claude Sonnet

Pipeline implementation follows predictable patterns; Sonnet handles SQL, Python transforms, and DAG definitions accurately at lower cost.

Example tasks

  • Write an Airflow DAG to ingest a new SaaS API source daily
  • Add a row-count and null-rate quality check to an existing transformation
  • Migrate a pipeline from a deprecated source table to its replacement
  • Optimize a slow dbt model that is missing an appropriate partition filter
# create an agent from this template, then start it
$ fleet agent create --name data-engineer--vendor claude-code --template <template-name>
$ fleet agent start data-engineer

Find the exact template name with fleet template list.

Run this agent in your fleet

One binary. Five minutes. See every agent, coordinate every handoff, and keep a full audit trail of what your fleet did.