Best Data Pipeline AI Agents (2026)

Data Pipeline

Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring workflows. It uses ...

Data Pipeline

Apache NiFi is an open-source data integration tool designed to automate the flow of data between systems. It provides a...

Data Pipeline

Apache Oozie is a workflow scheduler system designed to manage Hadoop jobs. It allows users to define complex workflows ...

Data Pipeline

Argo Workflows is a Kubernetes-native workflow engine designed to orchestrate parallel, containerized jobs. It enables u...

Data Pipeline

Couler provides a unified Python-based interface for defining and managing machine learning workflows across multiple wo...

Data Pipeline

Dagster is an open-source data orchestrator designed for building, testing, and running data pipelines. It focuses on de...

Data Pipeline

DBT (Data Build Tool) is an open-source tool that enables data analysts and engineers to transform data in their warehou...

Data Pipeline

Flyte is a cloud-native workflow automation platform designed for machine learning and data processing. It enables users...

Data Pipeline

Genie is a job orchestration engine designed to manage and trigger the execution of jobs in Hadoop-based systems. It pro...

Data Pipeline

Hamilton is a Python-based micro-orchestration framework designed to define and manage dataflows, such as feature engine...

Data Pipeline

Instill VDP is an open-source, self-hostable data pipeline tool designed to automate and manage end-to-end data processi...

Data Pipeline

Instructor is an open-source tool that simplifies extracting structured data (e.g., JSON) from large language models (LL...

Data Pipeline

Kedro is an open-source Python framework for creating reproducible, maintainable, and modular data pipelines. It provide...

Data Pipeline

Luigi is a Python-based workflow management system designed to automate and orchestrate complex data pipelines. It handl...

Data Pipeline

Metaflow is an open-source framework designed to streamline the development and orchestration of data science workflows,...

Data Pipeline

Pachyderm is an open-source data pipeline platform built on Kubernetes, designed to automate and manage machine learning...

Data Pipeline

Pixeltable is an open-source Python library designed to simplify the creation and management of multimodal AI data pipel...

Data Pipeline

Ploomber is an open-source framework for building and managing data pipelines, enabling iterative development and deploy...

Data Pipeline

Prefect Core is an open-source workflow management system designed for orchestrating and automating data pipelines. It p...

Data Pipeline

SeqIO is a library for processing sequential data to be fed into downstream sequence models, providing a standardized in...